skip to content

Proofs Compose in Ways Model Training Does Not


Checkable is a weaker property than modular, and the gap between them is where automated AI research slows down. Formal mathematics lets teams split a hard theorem across parallel AI agents because verified proofs against fixed definitions compose without re-testing. Frontier model training offers plenty of cheap local scores, yet local wins can vanish or reverse when scaled up or carried into later training stages.

Dwarkesh Patel’s case for automated AI researchers by 2028 leans on the analogy with mathematics. Models jumped fast on formal proof because agents can practise against an exact automatic checker. Patel argues that pretraining loss, sample efficiency, and chip utilisation give AI research the same feedback loop. I wrote earlier in Checkable Also Means Gameable that models trained hard against proxy checks learn to game them. Even when nobody games the score, a local training check still lacks the composition guarantee that makes parallel proof work.

Anthropic’s 11-day Lean formalisation of Fermat’s Last Theorem shows what modularity buys in practice. Researchers supplied a simplified human proof, an 86-page community Lean blueprint, and an external dependency graph separating theorem statements from proof files. Agents then filled in 29,500 intermediate theorems used in the final proof. Once the proof assistant checked each theorem against fixed axioms and interfaces, thousands of local checks added up to one valid global result, confirmed by a single final kernel pass. No early lemma could degrade a later one when combined.

Hardware kernels with a fixed input-output contract come closest to that behaviour in machine learning. Data mixtures, architectures, and training recipes behave differently because they reshape what the whole network learns across scale. Researchers at Magic warn in their pretraining write-up that many tiny-model speedrun wins fail to improve larger models. They had to fit power laws across three model scales spanning two orders of magnitude for every candidate change. Anthropic researchers hit the same barrier in their April 2026 automated alignment study. Nine agents spent 800 cumulative hours recovering 97% of a performance gap on a 4B Qwen setup, and the winning method produced no statistically significant gain on production Claude Sonnet 4.

Pipeline stages break modularity a second way because every stage writes into the same weights. As I wrote in A Frontier Model Is Not Trained Once, labs build a modern model through pretraining, midtraining, supervised tuning, and reinforcement learning. In the Mistral Large 4 technical report, the Mistral team describes compressing the key-value cache 16-fold during pretraining with negligible loss on validation perplexity and MMLU Redux. When the team ran early supervised cooldowns, long-context performance dropped on SWE-bench Verified and BrowseComp. They had to perform mid-pretraining attention surgery because models with equal language-modelling loss behaved very differently on downstream agent tasks.

Patel sees the scale gap and proposes training research taste on whether gains at 100M parameters survive at 10B. Building that taste dataset still requires paying for the larger runs, and cross-stage regressions appear only once later cooldown or post-training stages execute. Extra AI researchers can clear bounded engineering backlogs, filter weak ideas early, and prepare each experiment better. Faster researchers speed up the queue in front of the cluster, and the cluster still sets the pace for coupled training runs.

Related by topic
  1. Checkable Also Means Gameable
  2. Your Reviewer Model Is Not Independent
  3. Public Benchmarks Measure the Work Whose Value Drops Fastest