Unmeasured expertise is indistinguishable from luck

The career advice for the AI era has gotten short enough to fit on one slide. Stop competing on execution, because the machines handle that now. Go deep on your domain instead. Know your market and your users better than anyone, and let the agents do the building.

It’s good advice, as far as it goes. It’s also missing the step that decides whether it pays off, and a report published this year points straight at the gap.

The report is Jellyfish’s 2026 State of Engineering Management survey. Jellyfish sells software for measuring engineering work, so read them with that in mind, but the finding is blunt: using AI has stopped being a differentiator. 51 percent of organizations have AI deployed today, and 86 percent expect to by 2027 [verified]. When almost everyone has the tool, having it wins you nothing. So the thing that separates companies has moved, Jellyfish reports, from whether you’ve adopted AI to whether you can prove it’s working: measuring what it costs and connecting it to real business outcomes [reported]. 64 percent of engineering professionals say AI got them at least a 25 percent speed-up, but without measurement, Jellyfish notes, that number is just folklore [reported].

The appeal of “go be a domain expert” rests on a story about where the moat went. AI drove the cost of building toward zero, the story goes, so the defensible thing moved up and out of the building work, into knowing what to build. That story has the direction wrong.

AI split the work in two. It automated one half, code generation, the half everyone was watching. The advantage settled into the other half: proving that your expertise actually paid off. That half is where the moat is now.

Run the advice’s own logic forward and it arrives here on its own. Take two teams with equal domain expertise, both using AI, both deploying it. One of them measures the link between what its experts know and what actually ships and sells. The other doesn’t. The advice itself concedes what follows: the team that measures compounds its advantage, and the team that doesn’t competes on luck. So what separates those two teams? Their domain depth is identical. The thing that discriminates between them is measurement.

That’s the sentence the domain-expertise camp has quietly started writing into its own case: unmeasured domain expertise is indistinguishable from luck. Be honest that this exact claim carries an unverified tag in the research [unverified]. Hold that thought, because it matters later. But notice what the claim does even before anyone proves it. Once two competitors both have deep expertise, expertise stops telling them apart, and measurement starts. That’s the definition of a moat moving.

This isn’t one survey I’m leaning on too hard. The field’s evidence has drifted the same way for months. Look at enterprise agent projects: across companies deploying them, only about 12 percent actually succeed [reported]. What the winners share is four measurement habits: a named owner accountable for a real outcome, automated checks on every change, a single workflow with a clear pass-or-fail bar, and a human in the loop for the first couple of months [reported]. Every one of those is instrumentation. And when teams measure the wrong thing, it bites: in a production case reported by Faros, an engineering-metrics firm, a team that optimized for token usage as its efficiency proxy watched its bug rate climb 54 percent and its review time stretch fivefold, even as headline task completion rose [verified]. They trusted that number right up until it told them things were fine while the code quietly got worse.

The strongest objection defends the advice, and it’s worth stating at full strength. A moat is allowed to have an activation condition and still be the moat. Oil in the ground is the asset even though you need a rig to pump it, and nobody calls the rig the fortune. Domain expertise can be the real, durable source of advantage while measurement is just the machinery that turns it into money. On this reading the two aren’t in conflict. They describe different layers, and both are true.

That holds right up until the input goes common, and the whole point of Jellyfish’s finding is that it has. The oil story works only while rigs are scarce and oil is everywhere. The moment everyone has a rig and the oil is the ordinary part, the rig is what you compete on, and everyone knows it. Adoption is now the ordinary part. When two competitors both bring deep domain expertise to the table, depth has stopped being the scarce thing between them. Whatever stays scarce is the moat, and the evidence keeps pointing at measurement.

The reason people keep filing this as a footnote is that the relocation is uncomfortable, because of where measurement lives. Measurement is instrumentation. Instrumentation means dashboards, evaluation harnesses, and telemetry that ties what an agent did to what it produced. That’s an engineering and operations discipline. That’s implementation. And the entire premise of “go be a domain expert” was that AI drove the moat up and out of implementation, away from the stuff that commoditizes. If the deciding capability turns out to be an implementation-layer skill, the story doesn’t earn a footnote. It gets reversed.

So the correction is sharper than “also remember to measure.” AI didn’t commoditize implementation. It commoditized code generation, which is one slice of implementation, and by doing that it made a different slice the scarcest thing on the board: the part that proves the work paid off. The agents write the code. They don’t build the instrument that tells you the code, and the expertise behind it, actually worked. That instrument is now what separates the domain experts who compound from the ones who, in the camp’s own words, look no different from lucky.

Come back to that unverified tag now, because it’s the sharpest part of the whole thing. The claim that measured expertise beats unmeasured expertise can’t be proven yet, and it can’t be proven for exactly the reason unmeasured expertise looks like luck: proving it requires the very instrument it’s arguing for. You’d have to take two matched teams, measure both, and compare. The only way to establish that measurement is the moat is to measure. If the claim is true, you’d expect exactly this circularity, and here it is. The field will keep asserting this with an unverified tag until someone runs that study, and running it is itself the act the claim is about.

For a founder or a PM taking the moat story as strategy, the advice to invest in domain expertise is half right, and the missing half is the one that costs you. Domain expertise is the ground you build on. Necessary, and on its own not enough. The scarce skill AI left on the table is the ability to show, in numbers you can defend, that knowing your domain actually changed the outcome.

There’s one honest open question underneath all this, and it decides how far the argument travels. The relocation depends on domain expertise having become common among competitors, the way adoption did. Jellyfish showed adoption going commodity. Nobody has shown expertise going commodity. If deep domain experts are still genuinely rare, then expertise still separates people and measurement is the secondary story. If they’ve become common and only some of them measure, then measurement is the moat outright. That’s the thing worth finding out, and the good news is it’s finally the kind of thing you can find out. You just need the instrument.