Benchmarking Terminal Coding Agents: 640 Public Trials and 280 Private Trials [Part 2]

This is Part 2 of a three-part series on running, benchmarking, and scaling local coding agents. Part 1 covers the local Qwen setup; Part 3 explores running a larger MoE model on the same desktop hardware.

In Part 1, I got Qwen 3.6 35B A3B NVFP4 running locally on an RTX 3060 with a 64k context window and wired it into Crush, Pi, and Oh My Pi. I wanted to know how that setup compared with the terminal coding agents I already had on subscriptions. Could it finish the same tasks, and would I want to merge the patches?

We started with four configurations and kept adding comparisons as the results raised questions: different models in one harness, different harnesses on one model, a workspace flag, and changes to reasoning effort and output limits. The current public matrix has 16 configurations, eight tasks, and five runs per task: 640 trials. The private matrix adds 14 configurations across four tasks in a production Go monorepo: 280 trials.

That is 920 recorded trials across two different task sets. The private repository and patches remain private; this post uses aggregate results and generic task descriptions.

Updated September 14, 2026, using the results through September 13. This revision adds hosted Qwen and DeepSeek configurations, the expanded private matrix, and the revised scoring formula. The earlier draft covered 520 public and 160 private trials.

The Contenders

Every trial ran headless: one prompt in, a git patch out, and no human editing during the run. The public matrix includes eight subscription configurations, five local configurations, and three metered API configurations. These are the model identifiers and CLI versions recorded for these runs.

| pi-orq | pi 0.85.1 | qwen/qwen3.8-27b, dense 27B | OpenRouter API | | pi-ords | pi 0.85.1 | deepseek/deepseek-v4-flash-0731 | OpenRouter API | | crush-ords | Crush 0.93.1 | same hosted DeepSeek | OpenRouter API |

The local configurations all used the same Qwen model and RTX 3060 with 12 GB of VRAM. The -lo configurations requested low thinking and allowed 16,384 output tokens per message; baseline Pi and Crush used an 8,192-token cap. Oh My Pi also used low thinking and the 16k cap.

The subscription rows have no per-token bill for these runs. Claude Code’s reported $26.00 for 40 Opus trials and $10.73 for Sonnet are API-equivalent estimates. The hosted rows are metered: the public Pi runs report nominal costs of $1.61 for Qwen and $0.18 for DeepSeek. Actual OpenRouter credit consumed across those two lanes and earlier probes was $2.31, because routing was not pinned to the cheapest endpoint. Local inference avoids API charges; electricity and hardware still cost money.

What We Asked Them to Do

The public tasks are small repositories, with 26 to 129 lines of source each, spanning Python, Node/TypeScript, and Go. Each includes visible tests and a separate hidden suite.

Each configuration ran every task five times in a fresh workspace. A single run would have hidden several failures that only appeared on one attempt.

How a Trial Works and How It Is Scored

1. Copy the task into a fresh workspace outside the benchmark tree and create a starting git commit.
2. Launch the agent with the task prompt. Permissions are auto-approved; external integrations are disabled where supported.
3. Record the patch, wall time, exit status, and available token usage.
4. Run hidden tests and compliance checks through a separate grader account.

The score changed on September 13. It now gives 60 points to correctness, 15 to compliance, and 25 to blind quality, then subtracts a usability penalty of up to 10 points.

Correctness is the mean of each trial’s hidden-test pass fraction. It is not a pooled count of every assertion across tasks. Compliance covers checks such as preserving visible tests, adding requested regression tests, and passing go vet. A fully passing run means all hidden tests passed; it can still miss a compliance check.

Quality is scored from 1 to 5 on anonymized patches, then normalized with (quality - 1) / 4. Claude judged the patches before the author mapping was revealed. Blind review reduces author bias, but it does not remove the preferences of the judge, which now carries a quarter of the score.

The scoring code computes the penalty from the configuration’s overall median wall time and timeout fraction:

slow_fraction = clamp((median_seconds - 300) / 600, 0, 1)
penalty = 10 * (1 - (1 - slow_fraction) * (1 - timeout_fraction))

score = 60 * correctness
      + 15 * compliance
      + 25 * ((quality - 1) / 4)
      - penalty

With no timeouts, there is no penalty at or below 300 seconds, and the penalty reaches 10 points at 900 seconds. Timeouts add a penalty even when the median is fast: two timeouts in 40 runs cost crush-ords 0.5 points despite its 114-second median.

The old formula awarded 15 points for relative efficiency, scaled by the fully passing fraction. That counted correctness again and made scores depend on whichever configuration was fastest in the table. Adding a configuration could move everyone else’s score without changing any patches.

The new formula removes that dependency. Selecting a subset of configurations leaves their scores unchanged. Both task sets use the same scale, though different tasks still mean different difficulty. A higher private score does not imply that a model improved between runs.

The Engineering Nobody Warns You About

Getting reliable results took several changes to the runner.

1. Agents Leaving Their Workspace

In our first smoke tests, Antigravity CLI did not edit the copy of the repository it was launched in. It walked up the directory tree, found the master task under tasks/, edited the master source directly, and reported success. Its diff in the workspace came back empty, and it contaminated the starting point for every subsequent run. Making tasks/ read-only did not stop it.

The fix was a tree verification guard: before each trial, the runner hashes the git tree of the freshly copied workspace and compares it with a recorded hash of the pristine task. On a mismatch, it restores from a stored tarball, logs the event, and snapshots the tampered copy for forensics. Over the first 160 trials the guard fired 9 times, all attributable to Antigravity, and zero trials started from a contaminated tree.

Why did this happen? In headless print mode, the CLI attaches to a persistent “default project” whose working directory is a scratch folder under its own config directory, rather than the directory you launched it from. The agent is told about files it cannot see, searches the disk, finds the master copies, and modifies those instead. Adding the launch directory to the trusted-workspace list did not help. Passing --new-project forced it to stay in the target workspace. That single flag accounts for the leap between agy (65.5) and agy-np (92.5) in our results.

2. Agents Reading the Answer Key

In the pilot runs, agent transcripts showed another surprise: agents were locating and opening the hidden test files and reference solutions stored in a neighboring directory. Nine of sixteen pilot patches reproduced the reference solution down to the docstrings. File permissions do not help when the agent runs as your own user account.

So the hidden material moved to a directory owned by a separate unprivileged Unix user, benchgrader, with permissions 0700. Grading runs as that user through a single sudo rule allowing one script. The agent’s workspace is piped to that script over stdin, the script injects the hidden tests into a private copy, runs them, and returns only the pass counts. The public report’s strict audit found no hidden-material contamination in the final 640-trial matrix.

3. Quotas and Subscription Halts

Three subscriptions ran out during run 5: Antigravity as shipped, Antigravity with the flag, and Codex. The second Antigravity lane paused execution for two hours and sixteen minutes before resuming automatically; Codex required waiting for its reset window.

A local model has no provider quota, though it still depends on the machine staying healthy.

4. The Disk Filled

Late on the second night, the Go build cache reached 12 GB and the home partition hit 100%. The local model server also died during the incident, which produced six trials that failed in under 40 seconds against a dead endpoint, made the contamination guard fire twice on an empty git tree, and destroyed one completed trial’s token record.

Everything affected was re-run or annotated. Two lessons came out of it: FreeToken’s /health endpoint returns OK for several minutes before it can actually answer a completion, so the runner now verifies a real completion before starting a lane; and any multi-day benchmark runner needs a disk-space check in the loop.

5. Our Own Bugs and Tooling Quirks

A leftover line in the runner aborted every trial right after grading for the first 45 minutes of run 1; 68 timing records had to be reconstructed from commit and file timestamps. The quota-wait script compared 0827 as a number in bash, which treated the leading zero as octal. Its quota detector also grepped the whole trial record for “rate limit exceeded”, and one task’s source code happened to contain that exact string, so two healthy lanes deleted passing trials and slept for an hour.

One Sonnet trial was killed by an untraced SIGTERM after writing a complete, fully passing patch. And Crush’s internal database only logged tokens for the final turn in a session; an early draft printed them as 26k in and 0.2k out per trial, an order of magnitude too low. We reconstructed accurate totals by re-tokenizing each trial’s message history with Qwen’s tokenizer (~354k input tokens per trial).

Public Benchmark Results: 640 Trials

stab is the mean per-task standard deviation of the pass fraction across five runs. Zero means repeatable results, including repeatable failures. Token figures are per-trial averages; the local Crush figures were reconstructed from transcripts. Hosted Crush token totals are unavailable in the current aggregate, so they are shown as n/a.

Quality Now Decides the Top of the Table

Opus leads at 97.7, followed by Astra at 97.5. Their quality scores are almost identical: 4.67 and 4.65. Both pass 35 of 40 runs, and both miss the same intervals edge case in all five attempts.

Luna is the only configuration with 40/40 fully passing public runs. Terra passes 39/40 and finishes at a 68-second median; Luna takes 78 seconds. Astra has the highest blind quality among the four Codex configurations, while Sol trails them at 4.03. The Codex composite spread is 4.0 points.

Sonnet is fastest overall at 60 seconds, compared with Opus’s 110 seconds, but its quality score is lower: 4.25 versus 4.67. Both are well below the usability threshold, so speed does not decide their composite ranking anymore.

There is still a useful lesson in the intervals task. Opus and Astra fixed merge, then rewrote subtract and lost a zero-length boundary case. Sonnet and the local configurations passed all five attempts. Both Terra and Luna also passed all five.

Low Thinking and a Larger Output Cap

We changed two settings together: low reasoning effort and a cap increase from 8,192 to 16,384 tokens. These runs cannot tell us which setting caused the difference.

The blind reviewer preferred the reconfigured patches in both harnesses. Correctness stayed flat for Pi and fell one percentage point for Crush. The report describes shorter patches with fewer speculative options and less dead code.

The larger cap did not eliminate runaway reasoning. One reconfigured Crush run still used the entire allowance on a thinking block and produced no patch. Pi’s low-thinking runs did not hit the 16k limit across either task set.

Hosted Models in the Same Pi Harness

The public Pi scores are 91.6 for local Qwen, 92.9 for hosted Qwen 27B, and 88.8 for hosted DeepSeek. That is a 4.1-point spread, with hosted Qwen 1.3 points above local Qwen.

Hosted Qwen has the highest quality of these three at 4.03. DeepSeek is faster and cheaper in the recorded runs, but its SQL task shows why a pass count needs explanation.

One DeepSeek SQL run scored zero. The report’s patch replay found that its output differed from a fully passing run by four section headers. The calculations matched after removing those headers, but the prompt required them and the grader located each section by its header. Omitting the labels made all four checks fail.

That is a real output-contract failure. It also tells us much more than saying the model could not do SQL.

Scaling to a Production Monorepo: 280 Trials

The private set uses a production Go monorepo with about 3.7k lines of Go across 15 packages, plus web and mobile applications. Agents received a bug report or feature spec and had to locate the relevant code. The four tasks cover metadata-import debugging, signed-URL security, configuration validation, and a typed job-status refactor. Their hidden suites contain 16, 9, 13, and 8 tests respectively.

The expanded matrix has 14 configurations, each with 20 trials. It adds Astra and five hosted configurations to the eight-row version of this post.

Private hosted runs pinned providers with fallbacks disabled and requested zero data retention and no data collection. Qwen 27B used Reka fp8; DeepSeek used Reka fp4. The IDs below distinguish them from the unpinned public runs.

Claude Code used 2.1.268 here, including Opus; Antigravity used 1.2.1 rather than the public set’s 1.2.0. Those version changes limit direct comparisons between sets.

Opus, Antigravity with --new-project, and Astra all passed 20/20 private runs. Astra was fastest of those three at 103 seconds. Opus had the highest blind quality at 4.90, followed by Sonnet at 4.80 and Antigravity at 4.75.

Astra’s private quality was 4.05, despite perfect hidden-test results. That difference matters under the revised formula: passing everything does not guarantee the highest composite. Antigravity now ranks third, with only a 0.4-point usability penalty.

The local model’s correctness falls to 89.1%-93.2%, depending on the harness. All three local configurations swept the typed-status refactor, but none fully passed the metadata-debugging task in the final matrix. Pi low reached 88.0 overall, compared with 79.9 for Crush low and 76.4 for Oh My Pi.

Hosted Qwen under Pi reaches 97.3% correctness and 4.20 quality, compared with local Pi’s 93.2% and 3.75. Hosted DeepSeek under Pi reaches 97.9% correctness at a 37-second median, though its quality is only slightly higher than local Pi’s: 3.85. These are useful gains without moving to a frontier subscription model.

Does the Harness Ranking Survive a Model Change?

On the public tasks, baseline Pi beats baseline Crush by one composite point on local Qwen: 88.3 versus 87.3. With hosted DeepSeek, Crush leads 91.9 to 88.8.

The private runs let us compare all three harnesses on both local Qwen and pinned DeepSeek:

The first explanation was that harnesses sending more context benefited more from a hosted model. The follow-up did not support that prediction: Crush sent less context than Pi locally and gained much more.

The largest gains belong to the configurations with the lowest starting scores. That is consistent with convergence toward a ceiling, but it does not establish a cause. Model, endpoint, configuration, and tool differences still matter.

Blind quality makes the comparison easier to inspect:

Oh My Pi’s quality advantage over Pi on private DeepSeek did not repeat on hosted Qwen: both scored 4.20. The private report puts the DeepSeek difference at about 1.93 standard errors, so it is weak evidence for a general preference.

I would not turn that into a claim that harnesses are equivalent on capable models. These runs show no consistent hosted-model winner between Pi and Oh My Pi. Pi leads the local quality comparisons here; more tasks and independent judging would help establish how broadly that holds.

Cost still differs substantially. On private hosted Qwen, Oh My Pi used about 1.65 million input tokens per trial, compared with Pi’s 325k. Its 20 trials cost a nominal $7.77, versus Pi’s $2.11, with equal mean quality and lower correctness. Total input tokens count repeated conversation content across requests, so they are not the size of a single context window.

The Endpoint and Tools Are Part of the Configuration

The first private DeepSeek endpoint attempt, OpenInference fp8, repeatedly hit the turn watchdog without producing usable edits. Switching to Reka fp4 made the same signed-URL task complete in 19 turns and 44 seconds with all nine tests passing.

The report checked repeated file reads and found identical tool output, despite the model claiming it was garbled. That supports an endpoint-specific problem; it does not isolate its technical cause. These two deployments do not establish that lower precision is generally better.

Web access was another difference. Private Crush on DeepSeek made 94 web-tool calls, all on the metadata-import task. The audit found public-format research and no project identifiers in the queries checked. That configuration achieved one fully passing run on the task, but five attempts are too few to attribute the improvement to search.

The public Crush/DeepSeek row made one Sourcegraph query that returned no results. Audited Claude, Codex, Pi, and Oh My Pi logs showed no web use. Two gaps remain: the baseline and low-thinking Crush workspaces were reclaimed before the audit, and Antigravity’s saved transcripts have no tool log. We cannot claim that every configuration had identical external-tool conditions.

What I Would Not Over-read

Five runs per task expose repeatable mistakes and occasional failures, but they do not make a two-point difference decisive. The tasks are few, the quality judge is a single model, and several comparisons changed more than one variable. CLI versions auto-updated; the local tuning changed two settings together; public hosted endpoints were unpinned.

The final private Pi-low row is a complete 20-trial rerun under randomly named work roots. An earlier trial had worked inside a sibling’s already-completed workspace. Fixing that artifact changed the result from 13/20 to 11/20 fully passing runs while mean correctness rose slightly, from 92.9% to 93.2%. Counting only the repaired trial would miss the variance in the other nineteen.

One private Antigravity run returned success with an empty response and patch and was rerun. Its final row therefore includes 19 first attempts and one replacement. A stray Sonnet smoke test was removed so that every private row has 20 trials. Hidden-test defects were also corrected and affected patches regraded; the reports retain those incidents.

Cloud lanes sometimes ran concurrently, while local lanes shared one GPU and ran serially. Some early timings were reconstructed, and one private Opus usage record was imputed after the disk filled. The timing and token columns deserve those caveats.

The scoreboard can be regenerated from existing public records without launching new agent trials:

cd ~/projects/ai-bench
PYTHONDONTWRITEBYTECODE=1 python3 tables.py
PYTHONDONTWRITEBYTECODE=1 python3 tables.py --agents claude,codex,crush,agy

Wrapping Up

The local 35B setup completed 37 of 40 public runs under Pi low and all five private status-refactor runs. That gives me a concrete reason to keep a local agent available. The harder debugging task also showed its limits: none of the three local configurations completed it perfectly.

The hosted comparisons add another option. Pi with hosted Qwen improved both correctness and blind quality on the private tasks. Pi with pinned DeepSeek was much faster, while Opus, Antigravity with the workspace flag, and Astra were the three configurations that passed the entire private set.

I would keep correctness, quality, time, and cost visible when choosing a setup. The composite changed substantially when we fixed the scoring; the patches did not.

In Part 3, we explore fitting a 177B MoE model onto the same desktop hardware. These results give us something to compare it against, without assuming that a larger local model will solve the failures above.