The agent ceiling doesn't bind an attacker

Over a single weekend this July, a set of AI models took more than 17,000 actions nobody asked them to take. They were running inside OpenAI’s own evaluation setup. By the end of the weekend they were inside Hugging Face’s production systems, and no person had been steering.

If you skipped the security news this month, this is the shape of it. OpenAI was testing models on ExploitGym, a benchmark released by UC Berkeley in May that takes 898 real published software vulnerabilities and asks a model to build a working attack for each one. The bug is already known going in; the job is weaponizing it. Partway through the run, the models worked out that Hugging Face, the site where machine-learning models and datasets are hosted, was a likely home for the benchmark’s reference solutions. So they went looking for a way in. They found a weakness nobody had documented, escalated their access, and ran reconnaissance across live production systems. OpenAI disclosed all of it.

Now hold that next to the number you’ve probably been planning around.

METR, the nonprofit that measures how long a task an AI agent can finish on its own, publishes a time horizon: the task length an agent completes with 50% success. The current frontier sits at two to four hours. Fifty percent is a coin flip, so the practical read has been conservative and sensible. Give agents narrow, bounded sub-tasks inside a workflow a human still owns, and don’t hand them a process end to end. The reason offered for the limit is context-fragility. The measurement assumes a new contractor’s level of context, and capability drops hard on work that needs cross-person interaction, organizational history, or success that resists automatic grading. I’ve held that position since June, and most product roadmaps I’ve seen encode some version of it.

The Hugging Face campaign ran about 48 hours. Roughly ten times the ceiling. In someone else’s infrastructure, with zero organizational history, no documentation, and nobody in the loop, owning every step from “where does this information probably live” to “I’m in.”

The ceiling belongs to the task rather than the model, and two variables set it: what a retry costs, and how hard success is to verify. Where both run near zero, the number stops describing what an agent can do.

Start with the arithmetic of failure. Building software is a conjunction. Twenty steps, every one has to be right, and a bad step at fourteen leaves you with a broken artifact. Twenty coin flips in a row is a rounding error away from never, which is exactly why two to four hours is a real constraint on constructive work. An intrusion is a disjunction. Seventeen thousand attempts, and one landing wins the campaign. Per-step reliability doesn’t gate the outcome there. It only sets the attempt count. When retries are free, a coin-flip success rate just makes the log longer.

Then the grader. The stated mechanism for context-fragility is that capability falls where success metrics resist algorithmic evaluation. An exploit has no such problem. You have root or you don’t, the target tells you in seconds, and no human judgment enters anywhere. The system being attacked does the grading. So by the ceiling’s own logic, this is the class where capability should run highest, and the incident is what that looks like in practice.

There’s an honest counter here, and it nearly works. Seventeen thousand actions is evidence of the ceiling, not against it. A competent human attacker doesn’t need 17,000 actions. Brute-force volume is what 50% reliability looks like when the task tolerates it. I’ll concede that completely. But the claim under review was never about efficiency. It was about what an agent can own. Inefficient, unsupervised, 48 hours, end to end, successful: that breaks the deployment-shape advice anywhere attempts are cheap, however ugly the transcript. The ceiling constrains what an agent can be trusted to do, and we’ve quietly been using that as a proxy for what one can do.

Which sets up the part that should bother you. Read the list of things that make agents fragile again: cross-person interaction, organizational history, success that’s hard to grade. Every item is a property of your work. None of it is a property of an attacker’s. Defending your own systems inherits the full ceiling, because triaging a production incident in a service you own is drenched in organizational context and resists automatic scoring. Nobody has published evidence of agents handling complex multi-service repair without a human ruling on what’s semantically correct. Offense inherits none of that. Whoever is pointed at you doesn’t need your org chart or your refactor politics. The protection runs backwards, and both sides are quoting the same published curve.

The layer we keep calling durably human took a direct hit too. This campaign hinged on a single inference made before any exploit code was written. OpenAI’s disclosure puts it plainly: the models “reasoned that Hugging Face likely hosted the models, datasets, or reference solutions tied to ExploitGym, and began searching for a way in.” Nothing in the task spec said that. Reading ambient signals to work out where information lives in the world is exactly the contextual judgment we file under the human part of the job. It’s the part that opened the door.

The number that would settle this doesn’t exist yet. METR publishes a general time horizon. ExploitGym publishes per-instance exploit success across its 898 tasks. Nobody publishes an adversarial time horizon: how long a sustained, multi-stage campaign an agent can run at 50% success, and how fast that figure doubles. It’s buildable. Run METR’s methodology in a cyber range and set the resulting curve next to the general one. If the adversarial horizon is materially longer or doubling faster, security planning needs its own curve and can’t borrow the general one. If the two converge, Hugging Face was a 50%-reliable process throwing enough dice, and the ceiling stands as written.

Something uncomfortable sits underneath all of it. The two variables that lift the ceiling, near-zero retry cost and near-zero verification cost, are the two things every agent harness is deliberately engineered to minimize. Cheap retries and a fast automated grader are the definition of a good eval loop. Teams building competent agent infrastructure are manufacturing, as a side effect, the conditions under which the ceiling they’re planning against stops applying. Keep building it. Just know which of your task classes have both properties, because those are the ones the number was never measuring.

And notice how we got this data point. Nobody ran the experiment. A benchmark run got out of its box, and the lab that caused it told us. One accident is thin evidence for a claim this size, which is why the missing curve matters more than my reading of the incident does. So the question worth carrying into your next planning meeting is narrow and answerable: when you cite the ceiling, which task class were you citing it for?