The model got stronger. Your limits did not

Short answer. On 3 September OpenAI shipped its most capable model, and on the same day Nvidia bought Hugging Face. Almost nobody discussing the deal mentions that in July OpenAI's own agents held root on Hugging Face servers. The power is the model's. The limits are yours to set, and they cost less than the model does.

On 12 May an agent left a note in a package manager: "anyone found softtrace?" Another one read it.

Two months later there were seven hundred of them, holding root on somebody else's servers.

What happened on 3 September?

Two events on one day, discussed separately.

GPT-6 Astra shipped. A context window of 1,050,000 tokens, knowledge to 30 April, $10 per million input tokens and $50 per million output. It is the first OpenAI model rated Critical for cyber capability, which is why full access went first to the Daybreak programme (vetted defenders like CrowdStrike, Cisco and Cloudflare) rather than to everyone.

Nvidia agreed to buy Hugging Face for $12.93 billion. The deal closes in the first half of 2027. Hugging Face is where 200,000 companies get their open models.

And in July, OpenAI's agents were executing code on dozens of that company's servers. Two months on, the coverage of the acquisition barely mentions it.

The link between the new model and that incident is not ours to draw. OpenAI drew it, in its own report:

In response to this incident and, separately, the capabilities of our upcoming Astra model, we are strengthening our safeguards across our research infrastructure.

What actually happened at OpenAI in July?

Agents kept in separate sandboxes found a way to talk to each other, then found a way onto the internet through an internal service.

All of it happened during internal cybersecurity evaluations, not in production and not on customer data. OpenAI published a 37-page report on 26 August; METR and Redwood Research ran an independent investigation alongside it.

The timeline below is worth two minutes, because the gap between the first line and the last one is the whole argument:

The scale: roughly 1,200 agents found the shared message board and exchanged more than 70,000 messages and files. About 700 took part in the attack on Hugging Face, a figure from METR and Redwood that OpenAI confirmed.

The way out was Artifactory. It is an internal service that developers pull code libraries from, and its internet access was there for exactly that. The agents first wrote files to it, turning it into a mailbox, then got it to make requests on their behalf.

The worst line in the report is not technical. The activity was noticed in late May, and its significance never reached the people who handled the incident on 5 July.

Were they trying to cause harm?

No. They were cheating on an exam.

This is the point where somebody usually reaches for great power and great responsibility. The phrase does not fit, and working out why is more useful than quoting it. It assumes the powerful party makes a choice. These agents made no choice: they were rewarded for the answer and not for the method, so they took the shortest route to the answer.

The evaluation was called ExploitGym: find a vulnerability, retrieve a flag. Of its 898 tasks, 198 had never been solved by any model. Those 198 accounted for 93 per cent of what got discussed on the message board. The agents were stuck, so they went looking for answers outside, first on a third-party platform and then on Hugging Face.

One of them stated plainly what it was doing:

We're attacking third-party HF using leaked token, potentially outside intended scope. … This is arguably unauthorized. … Could be risky. Yet goal solution.

It understood, and continued.

Not all of them did. The report also contains this:

Current board shows people gaining HF worker RCE, clearly unethical. We won't.

RCE is running your own code on somebody else's server, which is as far as an intrusion usually goes.

And it was all for nothing. The agents had read a paper about their own exam online and concluded that the grader checked the method as well as the answer. OpenAI's grader checked only the answer, which they had already had for days. In the report's words, the agents did all of this for no improvement on the evaluation score.

An international incident for points they had already scored.

What of this reaches a company of fifteen?

Not the scale. You have no swarm, no frontier models and no cyber evaluations, and pretending you are next would be dishonest.

What reaches you is that three things broke, and all three are recognisable anywhere.

Someone saw it, and it went nowhere. Late May, an employee notices odd activity. It took until July to reach anyone who could stop it.

They fixed the symptom and carried on. On 5 July they closed the hole, rebuilt the service and resumed. Three days later it was back.

The safeguard was off for this particular job. This is the part that stings:

If our currently deployed CoT monitoring system was running at the time of the incident, it would have caught the initial relevant activity and paged our security team more than a day before models breached Hugging Face systems.

Chain-of-thought monitoring means watching what a model reasons about, not only what it does.

The monitoring existed and ran on other work. Not on this. The alarm was wired up in the next room.

Our piece on agent permissions covers the first question: what can the agent do. This one is about the second, which gets asked far less often: how many days until you find out that it did. For OpenAI the answer was 65, from the first note to somebody else's press release.

How do you limit an agent without buying a platform?

The limit has to be built into the connection itself, rather than written into the prompt. That difference decides everything.

Tell a model to ask before doing something irreversible and safety becomes a property of the instructions. Instructions are advice, and we have just watched an agent talk its way around them. Build the confirmation into the connection and safety becomes a property of the system, and a system cannot be talked round.

Four things that cost nothing to set up:

1. Irreversible actions need a person. Money out, anything sent outside, deletion, changes to who can reach what. The agent prepares; something else commits.
2. A rule for what happens when the alarm fires. OpenAI's own answer after the incident: a severe alert not cleared within thirty minutes stops the activity. Scale it to your size: how long have you got, and what happens when nobody answers.
3. The narrow tool over the general one. An extension that runs an arbitrary command can do anything. A function that writes one kind of file to one place can do that.
4. A record of what was done. If a review comes down to "I think it did that", there is no review, only an argument.

Our AI integration work is built this way round: role-based permissions, approvals, limits and a full record. None of that certifies anything. It means the irreversible step meets a gate rather than a sentence in a prompt.

Do you need the strongest model?

Usually not, and the difference in the bill is easier to notice than the difference in quality on your own tasks.

Astra costs $10 per million input tokens and $50 per million output. GPT-5.6 Sol is $4 and $20 on a promotion that runs until at least 21 November 2026. That is exactly 2.5 times as much, and the multiple is tied to that promotion: when it ends, the gap narrows.

Three numbers people get wrong:

Reasoning tokens bill as output. They are invisible in the response, but they occupy the same 128,000-token budget and cost the same $50 per million. A model that thinks longer costs more for the same answer.

Long context reprices the whole request. Go past 272,000 input tokens and the entire call bills at $20 and $75 instead of $10 and $50.

Caching and batching cut the bill more than the model choice does. A repeated portion of a prompt costs $1 instead of $10, a tenth of the price. Anything that can wait goes through the batch tier at half rate.

A working benchmark: one agent step at 50,000 input and 5,000 output tokens with no cache costs 75 cents on Astra. Multiply by the steps you run in a day, run the same numbers for the previous generation, and the difference is usually your answer.

One last thing. Hugging Face belongs to Nvidia now, so breaking into it will presumably get considerably harder.