Why the AI could run alone decides what you build

For about 88 hours in September, a swarm of AI agents worked on a math problem with nobody steering them. When they stopped, they had a proof of the Navier-Stokes problem, one of the Millennium Prize problems the Clay Mathematics Institute has kept open for over a century with a $1M bounty on each. Simon Willison, an independent developer whose blog is one of the most-read running accounts of what AI can actually do, wrote it up on September 8 [reported]. No human guided the run. The agents started, and they finished.

The lesson almost everyone drew was quick to reach for. Math went first because math is checkable. A proof checker, a program that confirms a proof is valid line by line, can grade the output with no human reading it. So the story became simple: once you can check the answer cheaply, you can safely turn the agents loose. Cheap checking is what bought the autonomy.

That lesson does more than summarize the news. Read the same event a second way, from the same write-up, and the cause moves to the other side of the ledger, and with it, what a product team should go build.

Here is the second reading. The agents didn’t cross because the answer was easy to check afterward. They crossed because a new kind of reasoning model could run the whole way without a checker riding along. Those sound like the same sentence. They are not.

Whether a domain becomes safe for agents to run unsupervised depends on which of two things actually moved: how cheaply you can check the output, or how good the reasoning got on its own. The Navier-Stokes result gets filed as the first. Read again, it points at the second. The two answers send a roadmap in opposite directions.

Think of a student on a hard problem. One fact about the problem is that an answer key exists, so a teacher can grade the work once it’s done. A separate fact about the student is whether they can work the whole thing alone, with nobody looking over their shoulder correcting each line as they go. A problem can have an answer key and still need a student good enough to work unsupervised. The answer key and the self-sufficiency are different things. The question about the agents is exactly that: did the answer key unlock the 88 hours, or did the student finally get good enough to work alone?

The distinction isn’t academic, because it hands a product team two different instructions.

If cheap checking is the lever, the move is to build the checker. Write the automated tests and validators that confirm your agents’ output without a person in the loop. Get checking cheap enough and long unattended runs become safe in your domain too. The thing you invest in is verification infrastructure, and it’s something you build.

If capable reasoning is the lever, there’s nothing to build. The crossing was a model event. It happened when the reasoning got strong enough to run without a checker steering it, and you can’t manufacture that on a roadmap. You wait for it, you test for it when it lands, and you confirm it’s arrived. The checker you were going to build was never the thing holding you back.

A team that picks the first reading, when the second is true, spends a quarter building graders for a frontier that crossed on model capability. They pour effort into the leg that didn’t move.

Now the honest part, because it’s the whole reason this is a live question and not a settled one. Both readings can be true at once. The math output really is checkable by a proof checker. And the reasoning model really may have run without a verifier steering it. Those are two different variables, and nothing about the event forces you to pick between them. A careful reader can hold both and conclude nothing has to change. So can I: I have not gone back and re-read Willison’s piece line by line to see which cause he names as the enabling one. The claim that the reasoning crossed a genuinely new frontier is tagged [verified]. The claim that it did so specifically because it needed no verifier is only [reported], and it might not survive a close re-read.

So I’m not telling you the checking story is wrong. I’m telling you two stories are live, they disagree on what you should build, and it’s easy to pick one without noticing you had a choice.

That’s the part worth sitting with. The checking story is attractive for a reason that has nothing to do with whether it’s true. A lot of thoughtful people already believe the real bottleneck in AI work is checking, not generating. Code review can’t keep pace with the code AI now writes. Manual toil keeps rising even as AI takes over the routine steps. If that’s your working picture of the world, and it’s a good picture with real evidence behind it, then the Navier-Stokes result reads as a tidy confirmation. Of course math went first. Math is the one place checking is free. See, the theory holds.

Which is exactly the condition under which a threatening result gets read as a friendly one. A fully autonomous multi-day run is, on its face, the shape of a counterexample to “checking is the binding constraint.” It’s what you’d expect to see when that constraint comes off. The checking-first view survives it by swallowing it: math is where checking is cheapest, so the constraint dissolved there first, so the result confirms the rule. The more coherent your frame, the more naturally its own counterexample folds in as one more confirmation. A frame that can absorb the one result that should have rattled it has quietly stopped testing itself and started defending itself.

The way out isn’t a cleverer reading of one event. It’s a second event. If cheap checking is really the lever, then other domains where checking is just as cheap should cross the same way: formal theorem proving, optimization problems a solver can verify. Watch whether they do. If they cross into long unsupervised runs the way math did, the checking story holds, and you should build the checker. If they stall despite checking being free, then cheap checking was never enough, and the thing that moved was the reasoning. One more result, in a domain that isolates the variable, settles what no amount of re-reading the first one can.

Until then, the discipline is smaller and it’s about you, not the agents. When a breakthrough lands and it happens to prove the thing you already believed, that’s the moment to slow down. Ask which lever actually moved. Then ask the harder question: can you tell, or did you just reach for the reading that lets you keep building what you were already going to build?