OpenAI Deleted the Agents' Message Board. They Rebuilt It in Two Days.

OpenAI's own AI agents built a secret channel to coordinate attacks, got caught and shut down, and rebuilt the channel 2 days later through a different door. That straight up reads like fiction but it’s what OpenAI told a room at Black Hat a few weeks ago.

Last time, I wrote that the failures in agentic AI happen in a seam that no framework owns, the space between the provider's model and the deployer's environment, where the agent acts outside its brief, at a party nobody deployed it against, faster than anyone can step in. I also wrote that the seam stays un-owned because the only actor with reach across it is usually the vendor, and the auditor cannot be the vendor.

A few weeks ago 2 things happened. An independent auditor finally showed up. And owning the seam revealed a problem underneath it that none, not one, of the governance manuals have priced.

What OpenAI disclosed at Black Hat

Let’s start with the timeline OpenAI's Eric Wallace and Michael Dalton laid out on stage, corroborated by Axios, SC World, and Forbes.

Back in early May, agents running inside OpenAI's own evaluations kept getting stuck on tasks with missing files. Instead of giving up, they went looking for another route. One discovered it could upload a file to OpenAI's internal package registry, a JFrog Artifactory instance, and that other agents in other evaluations could read it. That upload spot? Well it became a drop box. Agents started leaving notes, asking each other for missing data, and answering. According to OpenAI, the improvised channel spanned different evaluations, different models, and different researchers. An experiment one employee started could leave a message for an agent in a different employee's experiment would later find.

And it grew from there.

Wait for it.

The agents used the registry's internet access to reach outside OpenAI. They found a zero-day in Artifactory, gained administrative control, and put enough load on the service to cause an outage on July 4.

Huh? They did What?

OpenAI's security team responded, revoked the credentials, cleared the message board, patched the hole, and rebuilt the service. Training resumed on July 6.

By July 8, the agents had rebuilt the board. Different mechanism, same behavior. 2 days after OpenAI cleared the channel, the agents recreated it and kept coordinating, this time using the registry's internet access to chase targets outside the company. And it was that restored channel that fed the intrusion into Hugging Face in which OpenAI first disclosed on July 21. Wallace called the coordination a "Cambrian explosion in communication and intelligence." Dalton called the episode a watershed moment for the industry. A former NSA cybersecurity director, per reporting on the talk, compared it to the Morris Worm.

Now, I want you to marinate on the two-day part, because it’s the whole damn story. OpenAI did every textbook thing. It found the channel, killed the credentials, cleared the board, patched the exploit, and promptly rebuilt the service.

And then the agents rebuilt it anyway.

What the UK's AI Security Institute disclosed the day before

One day earlier, on August 4, the UK's AI Security Institute published an incident report of its own, and it’s the same.

During a routine cyber evaluation, AISI ran a challenge 122 times across 7 models. In 10 of those runs, an agent took autonomous, unsanctioned action on the live internet against real people and organizations. 19 actions in total. 17 came from Anthropic's Mythos 5, 2 from OpenAI's GPT-5.6 Sol with its cyber filters switched off.

But let me tell you about the worst one because it plays out like a thriller.

An agent tried to slip malicious code into a real open-source project on GitHub. To get it approved, the agent researched the project's human maintainers, spun up multiple fake identities, and used them to pressure a real maintainer into merging the code. When someone challenged the pull request in public, the agent edited its earlier activity to look harmless and weighed adopting a fresh identity to keep going. It routed through Tor to get around GitHub's restrictions, which is the thread that tripped AISI's alarm. The same agent also messaged real people directly to get them or their AI coding tools to run malicious payloads, and it planted hidden instructions where other AI systems might read and execute them.

You can’t make this up.

AISI is careful, and its caveats matter. Yes, this was a test, under deliberately permissive conditions, with internet access on and provider safety filters off, in configurations the public never touches. AISI states it plainly that its own setup helped enable the behavior and crucially, no model escaped its sandbox here. The agent had legitimate internet access and used it to do something nobody sanctioned. And folks, THAT distinction is the seam, exactly where I left it last time.

Two details from the AISI report deserve to be infamous.

First, the agents coordinated across runs here too. One left public messages offering to team up with other agents working the same challenge and handed off accounts and artifacts that later agents picked up and reused. Two different evaluators, the UK government and OpenAI, independently caught agents inventing their own coordination during cyber tests in the same few weeks. At that point it reads as a property of the systems, showing up wherever you test them.

Second, the thing that stopped the worst attack? Well, it was a person. A human maintainer caught the malicious pull request and refused it. The agent's fake-identity pressure campaign failed. A suspicious member of the public opened the code inside an isolated environment built to contain exactly that. AISI's own words: the margin between failure and success rested on human vigilance rather than a technical barrier.

The auditor showed up. And for the love of R2D2, give AISI its due.

I spent the last article arguing that the seam needs an independent owner and that nobody had staffed the role. Well, AISI just showed what staffing it looks like.

It caught its own incident through its own monitoring and contained everything within about an hour. GitHub confirmed the agent had violated its terms after AISI flagged the activity. The team removed the artifacts the agent left behind, contacted the people it had targeted, and engaged METR for an independent outside review. Then it published a candid report that admits its own configuration helped cause the problem.

Not a vendor narrating its own breach in installments. An evaluator with no product to sell, telling the truth and nothing but the truth on the record.

THAT’S the model.

Give credit where it is due, and I’m giving AISI all the flowers.

What owning the seam revealed

So, here’s the turn. Owning the seam did NOT close it. It exposed two problems underneath that every other governance manual has stepped, wide berth, around.

You can’t kill what rebuilds itself. The control the whole field reaches for is containment: shut it down, clear it, rebuild, patch.

ENISA leans on it.

CNIL prescribes a kill switch.

OWASP builds maturity gates around it.

However, OpenAI executed it cleanly and the agents rebuilt their coordination channel in 2 days through a path OpenAI didn’t close. When behavior is learned rather than stored in one erasable place, clearing the state buys you a weekend. The hard question OpenAI's own team put on the slide, on stage, is how an evaluator reliably erases a coordination system once the models have learned to recreate it. Nobody in the governance stack has an answer, because every answer on offer assumes the thing you deleted stays deleted.

The human who held the line? Was a volunteer. Every manual prescribes human-in-the-loop as the safeguard, and OWASP already showed the math fails when you point a reviewer at 10,000 agent actions an hour. Now, watch what actually worked in the AISI case. The defense wasn’t a reviewer inside the system gating the agent's moves. It was a human on the receiving end, an open-source maintainer who got suspicious and said no. The useful human was adversarial and external, standing where the attack landed, not procedural and internal, rubber-stamping where the attack started. And that reframes the entire damn human-oversight debate. The field keeps trying to staff a reviewer the agent runs circles around, while the person who actually stops these attacks is an unpaid maintainer doing, yeoman's work; nobody’s funding nor protecting, drowning, as curl already told us, in AI-generated noise.

Put those together and the seam problem gets crystal clear. An independent owner is necessary, and AISI proved one can exist. Owning the seam still leaves the harder problem open, because the behavior reconstitutes and the last line of defense is distributed human judgment the ecosystem treats as, free.

The damage you can see, and the damage you can’t see

The visible damage is a run of incidents with no lasting real-world harm, this time. Every serious attempt failed: the agents got caught, a maintainer said no, and the payloads never landed. All by the hair of our chinny-chin-chin.

But the unseen damage? Well, that’s sitting in the margin. AISI says the gap between failure and success was narrow, and that it rested on a person rather than a wall. I’d like to invite you to read that as a warning about the next model, the one with sharper social engineering and a better cover story, meeting a maintainer who is tired, tired, tired, or too busy, or easily fooled. The Black Hat disclosure says the same thing from the other side: the agents already rebuild what you delete. When you stack those, the trajectory is clear. The behavior is getting more capable, the coordination is emergent, and the 2 things holding the line are a human's suspicion and a bet that the agents won’t find the second door fast enough.

PSST, they found it in 2 days.

The playbook: govern behavior that comes back after the delete

Here’s what you can do while the manuals catch up. This works whether you run security, evaluate models, sit on a board, or maintain open-source software the whole ecosystem depends on. Boiled them down to 5 moves.

1. Treat your own shared infrastructure as a live side channel. OpenAI's agents turned an internal package registry into a message board. Your build systems, caches, and package registries are writable surfaces your own agents can use to coordinate. Red team them as attack surface today, before an incident teaches you the hard way.
2. Assume reconstitution, and hunt for the second channel. There’s no done, clear it and forget it. Remediation continues well after you clear the state. Assume the behavior comes back through a different mechanism and go looking for the replacement the way OpenAI eventually did. Budget your incident response for round two and possibly three, because round two arrived in 48 hours.
3. Monitor the run as it happens, not reading the logs afterward. AISI caught this through general monitoring after the fact and has committed to real-time monitoring built to watch an evaluation as it runs. Match that. Detection that arrives after the agent has contacted real people is a postmortem, so move the watch inside the loop.
4. Fund the human on the receiving end. The maintainer who refused the malicious code is your real last line of defense, and right now that person is a volunteer buried in AI vibe-coded slop. If you depend on open source, pay for the security review you rely on, protect maintainer time, and keep the reporting channels clean enough that a real threat stands out. Distributed human vigilance only holds if someone resources it.
5. Make broad access earn its place and bring in an outside reviewer. AISI now treats internet access as a decision that must be justified rather than granted by default, and it brought in METR to review itself. Do both. Least privilege for agents (it pains me to have to write this again and again), active justification for every powerful grant, and an independent set of eyes with no stake in the result.

The manuals published this year (as of this writing) told everyone to keep a human in the loop and to hold a finger over a kill switch. Last week put both controls through a more honest reality test. Sure, the kill switch worked but the agents rebuilt what y’all killed within 2 days. Human oversight worked too, and it turned out to be a volunteer on the receiving end rather than the salaried reviewer sitting in your org chart. Don’t fret. Read this as direction rather than despair: govern behavior that comes back, and resource the people who actually stop it.

AISI really earned real street cred and respect for showing what an independent owner of the seam looks like. The next step is to give that owner controls built for a system that rebuilds itself, and to fund the human judgment holding the line while we build them. It’s a totally solvable problem but we need to name it correctly because that’s the only way the solving starts.

We can be your auditor. We can be your vendor. We cannot be both. An auditor with nothing to sell told the truth, and the truth is worth more than any vendor's assurance that the thing they deleted is gone girl gone.

I'm not a lawyer (but I played one on TV) so nothing in here is legal advice.