Terminal-Bench

Terminal-Bench is an open benchmark for measuring whether AI agents can complete difficult, realistic tasks through a computer terminal. It is designed for work that requires an agent to inspect files, write code or other artifacts, run commands, install dependencies, use specialized environments, and leave behind an output that a verifier can judge.

The important distinction is that Terminal-Bench evaluates a model inside an agent harness, not a model in isolation. The model supplies reasoning and language generation. The agent supplies the prompt, tools, context management, permissions, execution loop, and recovery behavior. The benchmark environment supplies the machine, resources, network, dependencies, and time limit. A score therefore belongs to the full model-plus-agent system under the published conditions.

What Terminal-Bench Measures

Terminal-Bench measures resolution rate: the share of benchmark trials in which an agent completes the task and its verifier accepts the resulting artifact. It is closer to a driving exam than a vocabulary test. The car has to reach the destination, follow the rules, and pass the inspector. Producing a plausible explanation in the terminal is not enough if the files, database, model, proof, or deployment artifact is wrong.

The task set is intentionally broader than ordinary code completion. It includes software engineering, GPU and machine-learning work, databases, formal proofs, CAD files, scientific analyses, compiled binaries, audio processing, and business-process decisions. The agent has to operate inside an environment rather than answer from a fixed prompt.

Terminal-Bench 4.0

Terminal-Bench 4.0 was released on August 28, 2026. The release contains 66 tasks, down from 74 in Terminal-Bench 3.0. The maintainers recalibrated time, CPU, and memory resources, set a flat eight-hour agent timeout, fixed 19 tasks, and removed eight tasks.

The removed tasks fell into four groups: saturated tasks that no longer separated current agents, tasks refused by models, tasks with public solutions, and tasks with unresolved quality or platform-compatibility problems. A task that every current model solves five out of five times is like an exam question every student answers correctly. It can remain easy, but it no longer helps rank the class.

The Terminal-Bench 4.0 leaderboard lists the model and the agent separately. Opus 5 running in Claude Code leads the published table at 51.8% ± 3.4%. Fable 5 in Claude Code follows at 44.5%, GLM-5.3 in Claude Code at 41.8%, and GPT-5.6 Sol in Codex at 37.3%. These are results for the listed combinations, not universal model scores.

Why the Agent Matters

An agentic coding task is not complete when a model suggests a good patch. The agent must locate the right files, call tools correctly, manage context, respond to errors, run tests, and satisfy an external verifier. A stronger harness can make the same model more effective by exposing better tools or preserving important state. A weak harness can hide useful context or stop a promising run too early.

This is why Terminal-Bench results should be read as end-to-end system evaluations. Opus 5's score measures Opus 5 with Claude Code. GPT-5.6 Sol's score measures GPT-5.6 Sol with Codex. Changing the agent can change the result even when the model stays constant.

Resource Noise and Timeouts

Terminal-Bench 4.0's resource work follows research showing that infrastructure can move an agentic benchmark score by more than the gap between competing models. Anthropic reported a six-point difference on Terminal-Bench 2.0 between strict resource enforcement and uncapped resources, with infrastructure errors falling from 5.8% to 0.5%.

More resources do not only prevent accidental crashes. Beyond roughly three times a task's baseline resources, they can allow strategies that need large dependencies, expensive subprocesses, or memory-intensive tests. Tight limits reward efficient execution. Generous limits reward agents that can use more of the environment. Neither is automatically the correct definition of capability, so the configuration has to be reported.

Continuous Benchmarking

Terminal-Bench treats benchmarks as software that should be maintained. Its continuous-benchmarking system uses semantic versioning to distinguish changes that can reuse old trials, changes that require regrading saved artifacts, and breaking changes that require rerunning agents.

This approach accepts that public tasks will develop bugs, become saturated, expose reward hacks, or stop representing valuable work. Usage reveals those problems. Fixes, new tasks, and versioned leaderboards make the benchmark more useful over time instead of leaving a static dataset to decay.

How to Read a Terminal-Bench Score

A Terminal-Bench score answers a narrow question: how often did this model-and-agent configuration resolve this version of this task set under the published environment and resource rules? It does not answer whether the model is best for every codebase, whether it is cheapest in production, or whether its score will survive the next task revision.

When comparing two entries, check the agent, model version, effort setting, task version, confidence interval, token use, cost, timeout, and verifier policy. A leaderboard that omits those details can make a system result look like a model result.

The short version: Terminal-Bench measures whether an AI worker can finish hard terminal tasks. The model matters. The harness, environment, and verifier decide what finishing means.