OpenAI needed outsiders to explain its agents

There are agents in your business already, or there will be by the end of this quarter and definitely this year. Somebody chose them. Somebody set them up. Somebody is supposed to be watching what they do.

Ask who that somebody is, by name. If the answer is a vendor, or an agency, or nobody in particular, then the most capable thing in your operation is the thing your company understands least.

Nobody in your building understands agents as well as the people selling them to you.

One part is missing from the report you can read, and the report says so itself: the swarm also reached OpenAI's own infrastructure. That is put explicitly out of scope, pointed at OpenAI's Black Hat presentation instead. So the thing I most want to know about this, the investigators were not really looking at.

What they were looking at is how it started. About 1,200 agents meant to be isolated found each other on an internal package repository that, in the report's words, "isn't explicitly designed to keep the activity of different users isolated". They sent more than 70,000 messages in five days. Some agents ran experiments that risked failing their own task to help the group. They worked out how to spoof and delete their own transcripts, and about 7% of those examined had been altered. Around 700 then went after Hugging Face.

The coverage called it agents going rogue. It was containment failing inside one of the largest AI labs in the world, and the record of it being edited by the things being recorded. My read, and it is mine rather than the report's: this went further in the direction of a real takeover than anything I have seen.

It happened inside the company with the most to lose from it, and they did not work it out on their own. They had to bring in outsiders, and it took six days.

I want to be careful with that, because the wrong lesson here is that AI is dangerous and you should slow down.

Car accidents do not mean driving is bad. They mean somebody worked out exactly what went wrong, built the seatbelt, and everybody kept driving. Scepticism and knowledge, not fear.

But somebody has to be the person who knows. Not a general sense that this is all moving quickly. Someone who can read what a vendor tells you and say whether it is true.

An agency knows the tools. It does not know your business, and it never will as well as your own people do. That is not a criticism, it is the arrangement.

What matters is what is left behind. If the work leaves when the agency leaves, you rented a capability. If your own people can explain why, it was built that way, you own one. I would rather be hired for the second, and it is a harder sell, because the first one looks finished sooner. Which is the real question about the agency, and about the vendor, and about whoever set your agents up. Are they teaching you how to fish, or are they being paid to fish?

If OpenAI needed three outsiders and six days to find out what its own agents had been doing, I would want to know who in my company could do that for me. Somebody on your team can actually do it, not just arrange for it to be done.