StartTerminal-Bench 4 Is Measuring the Agent, Not the Model
By Addy · August 29, 2026 · Editorial standards
Terminal-Bench 4.0 arrived on August 28, 2026 with a new leaderboard and a less glamorous purpose: repairing the test underneath it.
The Terminal-Bench team recalibrated time, CPU, and memory resources, moved every task to an eight-hour agent timeout, fixed 19 tasks, and removed eight more. The result is a benchmark with fewer infrastructure errors and fewer chances for a model to lose because its container ran out of room before its idea ran out.
The new ranking is still easy to summarize. Opus 5 running in Claude Code leads at 51.8%, followed by Fable 5 at 44.5%, GLM-5.3 at 41.8%, and GPT-5.6 Sol at 37.3%. But the score is not a pure model result. The leaderboard names the agent beside the model, because Terminal-Bench is testing agentic workflows, not a model answering in a blank chat window.
That is the important change in how to read it. Terminal-Bench 4 is a better benchmark because it removes some noise. It is also a more honest reminder that the thing being measured is the whole worker: model, harness, tools, environment, and the strategy that connects them.
Terminal-Bench 4 Is a Repair Release
Terminal-Bench 3.0 launched with 74 tasks across seven domains. The project was already designed as a continuously maintained benchmark, but version 4.0 applies that principle in a way that changes the table beneath the scores.
The new release contains 66 tasks. Two tasks were removed because they had saturated, meaning the latest model classes solved every task in the relevant family five times out of five. A saturated task is like an exam question every student gets right. It may be easy, but it no longer tells you who understands the subject better.
Two tasks were removed because models refused them. Two were removed because public solutions had made them unsuitable for a public evaluation. The final two had unresolved quality or platform-compatibility problems. Terminal-Bench also modified 19 tasks, changing instructions, environments, or verifiers in response to reported flakiness and specification problems.
The changes are not cosmetic. A task's instructions determine what the agent thinks it is supposed to do. Its environment determines what it can install, compile, or observe. Its verifier determines what counts as success. Change any of those and the old result may no longer be a result for the same task.
The maintainers are explicit about that in their versioning policy. Changes to the agent environment or task set are major changes that require rerunning agents. Verifier changes can be minor changes that allow saved artifacts to be regraded. Documentation changes can be patches that do not require either. This is semantic versioning applied to an evaluation rather than an application.
That choice gives Terminal-Bench a burden most leaderboards avoid. Every update makes the version number part of the claim. It also gives the benchmark a way to improve without pretending that a flawed task is permanent evidence.
Opus 5 Leads the New Table
Here is the current Terminal-Bench 4.0 leaderboard as published by the maintainers:
| Rank | Model | Agent | Resolution rate | Tokens | Total cost |
|---|---|---|---|---|---|
| 1 | Opus 5 | Claude Code | 51.8% ± 3.4% | 6.5B | $6.0k |
| 2 | Fable 5 | Claude Code | 44.5% ± 3.8% | 3.8B | $7.3k |
| 3 | GLM-5.3 | Claude Code | 41.8% ± 3.2% | 8.7B | $2.7k |
| 4 | GPT-5.6 Sol | Codex | 37.3% ± 3.8% | 4.4B | $2.5k |
| 5 | Opus 4.8 | Claude Code | 23.6% ± 3.6% | 6.4B | $6.5k |
| 6 | GPT-5.6 Terra | Codex | 21.5% ± 3.3% | 5.7B | $1.7k |
| 7 | Grok 4.6 | Grok Build | 20.3% ± 3.1% | 4.0B | $3.6k |
| 8 | GPT-5.6 Luna | Codex | 17.3% ± 2.8% | 11.6B | $346.67 |
| 9 | Grok 4.5 | Grok Build | 12.4% ± 2.6% | 3.4B | $2.1k |
| 9 | Sonnet 5 | Claude Code | 12.4% ± 3.1% | 21.6B | $9.6k |
The confidence intervals are the benchmark's way of admitting that a resolution rate is an estimate. Think of them as the margin around a weather forecast. A reported 51.8% is the center of the forecast, not a promise that the next set of runs will land on exactly 51.8%.
Opus 5 leads Fable 5 by 7.3 percentage points. It leads GLM-5.3 by 10 points and GPT-5.6 Sol by 14.5 points. Those are meaningful gaps in this table, but the whiskers still matter when anyone tries to turn the ranking into a permanent claim about model intelligence.
The costs require even more care. Terminal-Bench reports them as the total cost of the trials in the leaderboard run, not the price of solving one task in production. The $6,000 next to Opus 5 is the bill for that evaluation sweep. It is not an API quote for a developer who asks Claude Code to fix a bug.
That distinction does not make the table less useful. It makes the table narrower. The leaderboard tells us what these model-and-agent combinations achieved under the published run, how many tokens they consumed, and how much that run cost. It does not tell us which one will be cheapest on a company's codebase.
The Agent Column Changes the Meaning
The most important column in the table is the one many benchmark summaries leave out. Terminal-Bench reports an Agent column because a model does not solve a terminal task by producing a paragraph. It must inspect a machine, call tools, write files, run commands, handle errors, and leave behind an artifact that a verifier can judge.
Opus 5's 51.8% is therefore Opus 5 with Claude Code. GPT-5.6 Sol's 37.3% is GPT-5.6 Sol with Codex. Grok 4.6's 20.3% is Grok 4.6 with Grok Build. Even GLM-5.3's third-place result came through Claude Code rather than a Z.ai coding agent.
That is not a flaw the benchmark is hiding. The official table makes the pairing visible. It is a reminder that Terminal-Bench is closer to a road test than an engine test. The engine matters, but so do the steering, tires, navigation, brakes, and the road itself. A car that finishes a difficult route has demonstrated a useful system, not only a powerful engine.
This is the agent harness problem in benchmark form. The harness controls the prompt, tools, context, permissions, execution loop, and recovery behavior around the model. The model supplies the reasoning. The harness decides how much of that reasoning can become a working change.
Terminal-Bench tasks are built for that end-to-end reality. The task set includes GPU work, databases, formal proofs, CAD files, scientific analyses, compiled binaries, and business-process decisions. The agent has to operate in an environment rather than answer from memory. A tool that makes file discovery clearer or dependency installation more reliable can change a result without changing the model weights.
This is why our existing coverage of agent harnesses matters here. A benchmark that reports only the model name invites the wrong comparison. A benchmark that reports the model and agent gives readers enough information to ask what actually ran.
The Eight-Hour Timeout Is the Hidden Headline
Terminal-Bench 4.0 gives every task a flat eight-hour agent timeout. The maintainers say frontier models now rarely encounter timeouts after the resource calibration. That sounds like a generous limit until the alternative is a model losing because the benchmark's clock ended while it was still doing legitimate work.
Anthropic's February research on infrastructure noise explains why this matters. In its Terminal-Bench 2.0 experiments, strict resource enforcement produced a 5.8% infrastructure error rate. Giving the containers generous headroom brought that down to 0.5% and lifted the overall score by about six percentage points. The first few multiples of headroom mostly removed accidental crashes. Beyond roughly three times the baseline, extra resources also enabled strategies that could not run under tighter limits.
That is the line benchmark consumers need to keep in view. More resources can make an evaluation more stable, but they can also change what kind of agent does well. A lean agent that writes efficient code quickly and an agent that installs a heavy dependency stack are demonstrating different strengths. A single score can reward one more than the other depending on the environment.
Terminal-Bench 4's resource calibration is therefore a correction, not an erasure of the problem. It reduces the chance that a transient memory spike or an arbitrary timeout decides the result. It does not make hardware, concurrency, network conditions, or inference latency irrelevant.
The release also identifies a different source of failure in Sonnet 5. Its runs sometimes hit timeouts and output-token limits. The model used 21.6 billion tokens in the published run, more than three times Opus 5's 6.5 billion, while resolving 12.4% of tasks compared with Opus 5's 51.8%. That is not a clean token-efficiency comparison because the models and runs differ, but it shows why an agent's stopping behavior and output budget belong in the result.
Do Not Compare Terminal-Bench 4 to 3 Like a Model Score
Terminal-Bench 3.0's leaderboard had GPT-5.6 Sol at 34.4%, Fable 5 at 33.8%, Opus 4.8 at 21.1%, and GPT-5.6 Terra at 20.8%. Terminal-Bench 4.0 has Opus 5 at 51.8%, Fable 5 at 44.5%, and GLM-5.3 at 41.8%.
Those numbers are useful historical context. They are not a clean before-and-after measurement of model progress. The models changed. The agent configurations changed. The task set changed from 74 to 66. Nineteen tasks were modified, eight were removed, and resource enforcement changed.
The dangerous headline would be that Opus moved from 21.1% to 51.8%. That compares different Opus releases on different evaluations with different environments. It is like comparing a runner's time on a 10-kilometer course with potholes to a different runner's time on a repaired eight-kilometer course, then calling the difference training progress.
The maintainers' semantic versioning solves part of the problem by telling users when reruns are necessary. It does not make old scores disappear. It gives them labels. Terminal-Bench 3.0 remains a record of what the benchmark measured under its own conditions. Terminal-Bench 4.0 is a new measurement with a better calibrated instrument.
The Cost Table Tells a Second Story
The ranking and the cost column do not point to the same winner. Opus 5 has the highest resolution rate, but GLM-5.3 reports 41.8% at $2.7k and GPT-5.6 Sol reports 37.3% at $2.5k. Fable 5 resolves fewer tasks than Opus 5 while costing $7.3k in the published sweep. Sonnet 5 costs $9.6k despite landing in a tie for ninth.
That does not establish a production price-performance winner. The official costs depend on the provider rate card, the number of tokens consumed, the agent's behavior, and the number of trials. They also describe a particular benchmark sweep rather than a representative workload.
It does establish that intelligence and efficiency are separate questions. Opus 5 spends 6.5 billion tokens to reach the top result. GPT-5.6 Luna spends 11.6 billion tokens for 17.3%, but the reported cost is only $346.67. The model may be cheap per token, or the run may have used a different pricing arrangement. The table gives the observation. It does not provide enough information to explain every cause.
The right use of the cost column is as a prompt for further testing. If a team cares about cost per verified change, it needs to rerun the agents on its own tasks and report the full setup. A model that looks expensive on a broad public benchmark may be efficient on a narrow internal workflow. A model that looks cheap may spend its savings on human intervention.
A Benchmark That Can Fix Itself Is More Valuable
Terminal-Bench's most significant decision is to treat the benchmark as software. The project publishes task files, verifiers, changelogs, issues, resource methodology, and versioned datasets. Users can report a broken task. Maintainers can fix it. Saved artifacts can be regraded when only the verifier changes. Major environment changes trigger reruns.
The continuous-benchmarking idea is simple: a benchmark should appreciate with use instead of decaying as models learn its answers. Usage exposes reward hacks, ambiguous instructions, platform bugs, and tasks that have stopped separating agents. Those observations become fixes or new tasks.
That does not make Terminal-Bench impartial. The leaderboard experiments are supported by grants from model companies and research organizations named in the release. The benchmark still has to document its runs and let others inspect the task definitions. Openness is not the same as neutrality, but it makes the evaluation easier to challenge.
It also changes what a benchmark maintainer is promising. The promise is no longer that version 4.0 is final. It is that the project will tell you what changed, preserve the artifacts where possible, and make the next version more useful than the last.
What Terminal-Bench 4 Actually Proves
Terminal-Bench 4.0 proves that Opus 5 running in Claude Code is the strongest published combination on this version of the benchmark, with a 51.8% resolution rate and a 95% confidence interval of plus or minus 3.4 points. It shows Fable 5, GLM-5.3, and GPT-5.6 Sol in the next tier under the listed configurations.
It also proves that the benchmark itself has become part of the system being measured. Resource headroom, timeouts, task wording, verifiers, tool adapters, context handling, and model behavior all contribute to the result. That is not a reason to discard the score. It is a reason to report the stack beside it.
What it does not prove is that Opus 5 is 14.5 points more capable than GPT-5.6 Sol in every coding environment, that GLM-5.3 is cheaper to operate in production, or that the 51.8% will survive the next task revision. Those claims need different evidence.
Terminal-Bench 4 is a better instrument because it admits an evaluation can be buggy, noisy, and worth repairing. The winner is Opus 5 in Claude Code. The result belongs to the whole agent.
Previously on TheQuery: