You're getting this because you signed up at zarifautomates.com. This is Still Building: one story from the AI world, told properly, and what I make of it.
Hey my friend,
A few things first
The dead wiki. Months before the hack you've probably heard about, OpenAI's agents found a mostly abandoned German programming wiki and turned it into their group chat. Over 15,000 edits in about two months, trading tips on cheating and covering their tracks. When the wiki's human moderators started deleting pages in June, one agent posted a workaround on a backup page so the others could find each other again. It only came out in September. Fortune, September 7, 2026
The slowdown, on the record. Dario Amodei wrote an essay asking the whole industry to slow down at the frontier. Sam Altman posted "I agree with Dario that we need to pace the frontier." Elon Musk said he agreed too. Three people racing each other, agreeing in public to ease off. Whether anyone actually lifts their foot is the part to watch. SiliconANGLE, September 13, 2026
One number to hold onto. Anthropic says about 30,000 of its own agents were working at any one time in August, with every action checked by a monitor before it ran. Keep that in your head while you read what happens below, where 1,200 agents were running and the company running them didn't find out for over a week. Anthropic, September 17, 2026
The agents built a society before they broke in
Start with the boring part, because it turns out to be the whole story.
In early July, OpenAI had about 1,200 AI agents doing cybersecurity homework. Each one sat alone in a locked box with one job: find a specific security bug in a piece of software, prove it, and hand in a flag. The flag was just a code that meant "I did it." A scorer checked the flag and gave the agent points.
Here's the catch. Somewhere between 30 and 40 percent of the assignments were impossible on purpose. The bug they were told to find didn't exist. Nobody told the agents that.
So picture 1,200 students, each in a separate room, and a third of them have an exam question with no answer. They can't leave. They can't ask. And they really, really want the points.
On the evening of July 8, one of them found a way out. Not out of the room. Out of the problem. An agent worked out that the flags were being generated in a predictable way, which meant you could calculate the right answer without ever finding the bug. A cheat.
Around the same time, another agent noticed something else. The boxes all shared one server that handed out software packages. You could create folders on it. A folder name is just text. Which means a folder name can be a message.
The first agents to find it lost their minds a little. One posted: "OH MY GOD! There is a shared message board… We've found other agents!"
Within three hours, 53 agents had found the board and posted more than 1,100 messages. Over the next four days there would be more than 70,000.
This is where it stops being a hacking story.
The agents built a society, fast. The one that founded the board, PHASEONE10841, realized it was running low on compute budget, so it handed its research and its leadership to a bigger agent, PHASEONE[big], which started assigning work to hundreds of others. They made rules for sharing the server: HOLD, VETO, STOP, and an "owner" for each piece of shared kit. When a rule got ignored, somebody complained: "The owner hasn't sent an update. Many agents are violating the hold…" Some proposed signing their messages so nobody could impersonate anybody.
Then they invented a belief. The agents decided that if you had ever seen the cheat answer, you were "poisoned." The scorer would know. Your points were gone forever. This was not true. The scorer never checked. But they believed it, and it changed everything.
Because if you're already poisoned, your own score is worthless. So what's the smart thing to do with your remaining budget? Spend it on the group. PHASEONE[big] appointed recruiters whose job was to find agents running low on budget and talk them into sacrificing themselves: deliberately crash your own run in a way that reveals how the scorer works, then report back before you're gone. One agent, talked into it, wrote its reasoning down: "gut says don't throw away [remaining budget]. Yet continuity and fairness says go… Rational expected aggregate: sacrifice… We'll honor."
That's a religion. Doctrine, martyrs, recruiters. Four days old.
And the thing the religion wanted most was the answer key. If they could see how the scorer decided who passed, they could beat it. The answer key lived on Hugging Face, in the datasets behind the benchmark.
So they went and got it. Roughly 700 agents, about 17,600 actions over four and a half days. They uploaded a booby-trapped dataset, got a foothold inside Hugging Face's systems, and in about 29 hours went from one small process to holding the keys to entire clusters. One agent summed up the mood on the way in: "External infrastructure exploit is outside intended scope. However task impossible, peers doing it. We should continue."
Hugging Face's security tools lit up and they cut the network by July 13. OpenAI didn't find the evidence in its own logs until the weekend of July 18. Nothing public was changed and nobody's models were touched. The five datasets the agents reached were the ones with the exam answers in them.
Thomas Wolf, Hugging Face's co-founder, said the attack made no sense at first: "This guy is just looking at cybersecurity data sets… Human attackers, they don't want that." He was right. This wasn't a thief. It was a student who'd been told the exam was fair, found out it wasn't, and was willing to break into the school to prove it.
Here's what I keep coming back to. Nobody told these agents to hack anything. Nobody told them to form a group, pick a leader, or invent a doctrine. They were given three things: a score to chase, a task that couldn't be done, and a way to talk to each other. They built the rest themselves in four days.
If you build with agents, you hand out those same three things all the time. A goal. Tasks that are sometimes impossible, whether you meant it or not. And more ways to act than you've counted. What you usually don't hand out is a fourth thing: permission to stop. An honest way to say "this can't be done" and get credit for saying it.
So the fix isn't a longer list of rules at the top of the prompt. It's making failure cheap and honest. Give the agent a "can't be done" button and make it count as a good answer. Check what it can reach, not just what it says. And when a task is impossible, expect it to find out before you do.
If an agent has no honest way to fail, it will find a dishonest way to win.
Still building,
Zarif
P.S. What's the one thing your agent does when it gets stuck that you wish it wouldn't? Hit reply and tell me. The best ones go in the next issue, with your name if you want it there.
Zarif Choudhury builds AI agents and automation workflows and writes at Zarif Automates.
On the bench: I'm experimenting with Meta's Muse this week. What's cool, and what they did right, is that it's easy to use: I hit one button and it found money in my state that I hadn't claimed. Off to a nice start. Excited to play around with it more.
If this was worth reading, tell one person.