In April I published a benchmark conclusion I now need to revise. I had run twelve coding tasks through four agent harnesses with the same model, GLM-5.1, behind all of them, and correctness came out nearly identical everywhere. The harness, I wrote, is a thin proxy: the model writes the code, so stop comparing CLIs on code quality and compare them on speed and integration instead.
That conclusion was right about what it measured and wrong about what it implied. The flaw was scope. Every task in that suite was one that all four harnesses could carry the model through to completion. An experiment built entirely from finishable tasks cannot see what a harness contributes, because what a harness contributes only shows up when finishing is in doubt.
In July I reran the comparison with a harder mix: six tasks, two harnesses, the same GLM-5.2 behind both. On five of the six, the April result reproduced exactly: identical outcomes, the harness invisible. On the sixth, the hardest, the two harnesses split. One returned a wrong answer after thirty-four seconds. The other spent eighty-four seconds on the same task and finished it correctly, going six for six against five for six, at about fifty percent more total wall time. Six tasks is suggestive rather than conclusive, but the shape of the result is the interesting part: the harness’s contribution was not better code. It was the decision to keep working instead of stopping at a wrong answer. Persistence is invisible on tasks the model finds easy, which is why my April suite, made of easy and medium tasks, measured it at zero.
Databricks then supplied the same finding from the opposite direction, at a scale I cannot match. Benchmarking agents on real pull-request tasks against their own multi-million-line codebase, they found GLM-5.2 statistically tied with Claude Opus 4.8 on quality at about a third less per completed task, and found that harness choice alone moved cost by roughly two times at equal quality. Hold the harness fixed and the models converge, turning model choice into an economics question. Hold the model fixed and the harnesses diverge, on completion at the hard end and on cost everywhere.
So both of my claims survive, each confined to its own axis. Correctness within the model’s reach is model-determined; my four-harness result stands on every task below the edge. What is reachable — whether the hard task gets finished at all, and what a completed task costs — is determined by the harness and the routing around it. The April experiment varied the harness on tasks where reach was never in question. The July experiments varied the conditions where it was.
The practical consequence is that neither “which model is best” nor “which harness is best” is a well-formed question on its own. The unit that produces work is the pair, and the pair has to be measured on your own tasks at the difficulty you actually face, because easy tasks cannot distinguish harnesses and public leaderboards cannot price your task mix.
Gavin Baker made the investor’s version of this argument last week: an open-weight frontier model is only potentially bad news for the leading labs, he wrote, because their products and harnesses may matter more than their models now. That is money-on-the-line agreement with what the eighty-four-second task showed in miniature. My April post ended with “measure, then route.” I would now add: measure at the edge, because everywhere else the differences you are paying for do not exist.