skip to content

The Harness Is Part of the Token Bill


I got the same hidden-test repair score from two Grok 4.6 coding routes, but one reported sixteen times as many total tokens. The answers were nearly the same length. Most of the gap was context the coding harness spent before the model answered.

I measured it with Grok 4.6. I took six one-defect Python programs from the public QuixBugs benchmark and ran each twice through three routes: a thin no-tool Pi invocation, Grok Build with its normal configured context, and Grok Build with a minimal system-prompt override. The model saw the source and its docstring but not the tests. Every response was applied to a clean checkout and graded against the hidden target tests.

All thirty-six runs passed. Thin Pi accepted twelve of twelve. Native Grok Build accepted twelve of twelve. Minimal-prompt Grok Build accepted twelve of twelve. On this narrow task, there was no quality difference to explain away.

The token difference was not narrow. Pi reported 19,981 total tokens across its twelve runs. Native Grok Build reported 325,319. That is just over sixteen times as much, or 93.9 per cent fewer through Pi depending on which direction you prefer to read it. Median elapsed time was 7.1 seconds through Pi and 14.7 through native Build.

The answers were not sixteen times longer. Pi produced 4,735 output tokens. Native Build produced 5,189. The accounting formula matched on every run: input plus cache read plus output equalled the reported total. Almost the entire gap was context arriving before the answer.

The obvious suspicion was a bloated system prompt, so I ran the third condition. Passing Grok Build a minimal system-prompt override reduced its aggregate total by only 4.5 per cent, from 325,319 to 310,624. That control does not tell me where every remaining token came from. The balance could sit in the agent scaffold, hooks, global rules, or another layer the override does not replace. This is therefore a measurement of my configured Build route, not a claim about a clean installation. It does show that one editable system prompt was not the explanation.

There are two important limits. Grok Build reported its served model as grok-4.6-build, while Pi reported grok-4.6, so this is a comparison of routes I can actually use rather than one identical checkpoint behind two wrappers. The tasks were also small, one-turn repairs without tools or repository discovery. This says nothing about which harness is better for long agentic coding. Grok Build may earn its larger context when the job needs its loop.

It does change how I will read coding-agent comparisons. A pass rate belongs to the model, prompt, harness, tools, and verifier together. A token report should separate input, cache reads, output, and the reasoning share of output before anyone calls one model more efficient. If two routes produce the same answer quality and similar output volume, a sixteen-fold total-token gap is not a model result hiding in the plumbing. The plumbing is the result.

Related by topic
  1. Summarisation is where the judgment should sit
  2. Masking doesn't declassify
  3. Try Widely, Build Narrowly