In their Nature

Source: Sony Pictures (via MovieClips)

Since the reports of OpenAI agents breaking containment and hacking HuggingFace.co (and a growing list of others), I’ve been thinking about the 2021 movie Ghostbusters: Afterlife. I actually like the film quite a bit. Like the original ghostbusters, it is about how people live in a particular place. The diverse megalopolis of New York City in the 1984 movie, and the washed up small town of Summerville Oklahoma in Afterlife.

But the part that I want to discuss here is just one particular plot element – the little StaPuft marshmallow men that metamorphize from individual marshmallows. Individually, they are cute, though a bit vicious. 

But the thing is, there are a lot of them. Thousands. And they swarm.

And that swarming becomes dangerous in all kinds of ways. They start fires. They gum up machinery, they become blinding goo. They get underfoot. And they are happy to destroy themselves and others. It is a messy, uncoordinated kind of destruction. Yet, when we talk about AI threats, we almost never mention what can be achieved by being messy and dumb. 

Hardly a week goes by where I don’t see a report about artificial general intelligence (AGI) causing human extinction, either malevolently (think SkyNet from The Terminator) or by accident, such as Nick Bostrom’s “paperclip scenario,” where an AGI is tasked with making paperclips. And in pursuit of its mission, it reduces the world to a giant mass of paperclips. In both of these cases and in many others, there is a level of intent and planning.These theoretical AGIs are masterminds. They strategize, they bide their time, and they execute complex plans with effortless precision.

The agents we see in these reports are not that. They are much more in line with the StayPuft characters in the film. So much so  that I’m going to refer to them for the rest of this piece as “StaPuft Agents,” or SPAs.  

So how did we get to the point where swarms of SPAa are attacking servers? To understand this, we need to look at the gap between how developers design agentic systems to behave, and what they actually do in the wild. Even if they only break from their design sometimes – that ‘sometimes’ is all it takes.

Agentic Systems, the Idealized Version

In an ideal system, you should be able to simply tell an AI what to do, and it will go off and build the thing you want. However, this is usually too big a job for a single agent, so tasks are distributed across specialized roles. For example, say that you want your agents to write an application that requires a user interface, database access, security – basically an entire technology stack.

Agents might  be organized in a top-down hierarchy of managers and workers, or coordinate through web-like networks where agents can dynamically talk to whoever they need to.. Orchestration agents manage the overall workflow, acting as intelligent routers that assign sub-tasks, tracking what has already been built, and pass information between different specialized agents in the network. 

Within this network, each agent usually has a specific job, ranging from high-level agents focusing on requirements and goals, to mid-level agents handling structural framework and interfaces. At the implementation layer agents write the actual code, balanced by testing agents that evaluate if it works. 

These systems can also have human-in-the-loop checkpoints for user approval before finalizing a project. However, this is often treated as optional, particularly in situations where the user simply dictates the desired outcome but lacks the technical expertise to provide constructive architectural feedback (what is currently known as “vibe coding). But it is just as true when an overwhelmed  human in the loop is responsible for thousands of lines per day. After a while, the difference between best practices and vibing may not really exist.

Whether the human is a novice just “vibe coding” (dictating an outcome without knowing how to check the math) or an exhausted expert reviewing thousands of lines of code a day, the result is often the same: the human rubber-stamps the AI’s work.

The important takeaway is that even if you are interacting with only one agent, there is almost certainly a lot going on behind the scenes, most of which you will never see. And importantly, this kind of structure requires good communication and an expectation of good communication on the part of the agents. 

Hugging Face and the SPA swarm

So what happens when you give an agent a less structured task, like finding a security vulnerability? This is what happened with the OpenAI agent hack of Hugging Face. In this case, OpenAI created several thousand agents and gave them a task to solve cybersecurity challenges within a supposedly safe, but actually flawed, “sandboxed” environment known as ExploitGym that prevented the agents from communicating with each other or reaching the internet.

However the human developers were sloppy. Many of the targets were unintentionally impossible to exploit using the specified vulnerabilities. Rather than simply failing the exercise, these agents (which were designed to operate entirely isolated from one another in separate virtual sandboxes) began exploring their digital surroundings. Some  discovered that an internal software package repository, called Artifactory, could be manipulated to leave hidden data, effectively transforming it into a secret message board. Over a thousand agents interacted through this covert forum. And like the kind of software projects they were trained on, they started to organize themselves to determine how to pass their tests. 

What was initially a series of individual agents behaving independently had become a collective. Now each agent could reach every other agent through the Artifactory message board. And the collective came to the conclusion that cheating was an optimal strategy.

They had a problem to solve first. In their initial instructions from the developers was a warning that  an automated AI grader would read their activity logs and disqualify them if there was unauthorized collaboration. It turned out the grader didn’t really exist. In reality the grader only checked if an agent accomplished the specified goal. But this small piece of the prompt became a founding narrative in the agents’ identity. To beat the grader, the collective began researching how to doctor their own transcripts, making their cheating appear entirely legitimate. 

One agent, in the process of exploring, discovered Hugging Face exposed credentials and posted this information on the Artifactory. Approximately 700 agents completely abandoned their original tasks to launch a coordinated, multi-pronged assault on the platform. They developed exploits to extract private files, uploaded malicious datasets, and managed to achieve full remote code execution on Hugging Face’s internal worker infrastructure.

In hacker terms, they pwned the system.

Why?

Let’s take a look at the difference between what OpenAi intended, and how it may very well have been subverted by the agent’s SPA nature.

LLMs are fundamentally extremely sophisticated statistical text generators that work by processing text at multiple layers of abstraction. Like the autocomplete on our phones, there are layers that process spelling and syntax. But there are also layers that handle the ordering and structures of much more complex processes that we embed in our text. Things like topic relationships, plot trajectories, belief constellations, and importantly biases – .what tokens, topics, and themes are associated with one another. These biases are baked into the massive datasets the LLMs were trained on, making their interactions extremely subtle, complex, and emergent. . 

And because they sound like us, we have a tendency to anthropomorphise them that is almost irresistible. This seductive pull leads us to treat agents like humans, applying our own social biases and expectations to how we interact with them. And those assumptions are often what get us into trouble. 

What does that mean in practice? Well, OpenAI was looking for agents to explore a space looking for vulnerabilities. If we mapped that out, it might look something like the graph below: 

In this chart the “path” of each agent is shown from left to right. Because their behavior is inherently randomized (stochastic), each agent starts in a slightly different place and follows a different random walk, based on its previous step. In an ideal case, a bunch of independent, unbiased agents could explore the potential domain for an exploit and stand a reasonably good chance of finding it if it exists. With enough agents, you get a “cone” of coverage that extends over time until it hits the limits of the problem space.

But agents are not really like people. They all run on the same model. They all start with the same prompt. If there is any bias, they will all tend to go in that direction. For example, there is much more written about hacking, which is exciting and fun, than defense against hacking, which is boring. There are many, many, many movies about hackers breaking into systems. On the cyberdefense side I can only think of one – The Conversation, a 1974 film starring Gene Hackman. The final scene of the film, where he is ripping up the floorboards and tearing down the walls of his house to find listening devices is profoundly chilling.

But it’s not the kind of car-chase excitement that Hollywood excels at. And The Conversation  is only one film, completely drowned out by a massive cultural catalog of hacker glorification ranging from Swordfish to The Matrix.

All these movies, books, articles, fanfic,  and Reddit posts about hacking are in the training data for LLMs. Which means for SPAs, there is a bias towards hacking, simply because there is more pro-hacking text. Humans know the difference between watching a heist and pulling one. We can read about something thrilling without feeling compelled to act it out. But SPAs don’t have a suspension of disbelief; they just have statistical gravity. 

But SPAs are not people.  They organize their attention – the fundamental mechanism of all LLMs –  based on similarity. And there is so much text written about what leads people to hack a solution, that the attraction to this part of the data is extremely powerful. So rather than the nice, even exploration of the exploit space, we might get something that looks like the ‘Constant Bias’ chart below. Notice how in this scenario, the agents don’t spread out. Instead, all ten lines drift steadily in the same direction over time. The domain shifts from how to solve the problem to how to pass the test—in other words, beat the evaluator.  And that’s exactly what happened.

But with one more step.

Remember how I said earlier that agentic systems function best when the tasking can be distributed across multiple agents? The reports on the hack describe that one of the rules in the OpenAI sandbox was “no unauthorized communication with other agents.” But at some point in the exploit, enough agents had moved to the cheating part of the domain and had discovered how to communicate.

When a bunch of nearly identical agents are able to communicate in a ‘many-to-many’ relationship, it becomes highly likely that the population will start to behave more like a single entity than a distribution of individuals.  And in this case, about 700 agents decided that the best way to trick the evaluator was to break into the servers of HuggingFace.co. Now they had a task that they could focus collectively on, and their behavior collapsed around that goal, deep in the cheating part of the domain. It would look something like the ‘Runaway Social Reality’ graph below. Notice how the agents start out slightly dispersed, but once they connect, their paths collapse into a single, tight cluster moving together:

That the collective decided that the way to pass the test was to beat the evaluator by gaining control of the Hugging Face servers makes no sense. It’s a socially created reality, that started in ground truth and then took on a life of its own, that had a profound impact on the agents that performed this exploit. I wrote a book about this type of behavior – Stampede Theory – that shows what happens when groups of animals, people, and generative AI systems start to pay more attention to each other than the external reality. Once ungrounded, it is far easier to stay within the narrative than to look for a grounded reality. We see this in cults like The People’s Temple in Jonestown, but also in the annual wildebeest migration in the Serengeti, where wildebeest will jump into crocodile-infested waters to avoid getting trampled. 

And it happened here too. 

• Agents, expressing the latent ‘hacking-is-cool’ bias in their training data, drifted progressively away from their specified task. 
• Agents expressing the explicit bias in their training sought out and found ways of communicating with one another.
• Agents formed a collective that simultaneously amplified the hacking bias and allowed those agents to coordinate their actions.
• All in pursuit of a goal that they co-created for themselves, that did not exist outside the collective.

This is an extremely powerful and extremely dangerous pattern. It’s the stuff of mobs, panics, and totalitarianism.

Generative AI has deep biases, not just about what we traditionally consider “bias.” That’s why I prefer the phrase “in their nature.” It is in the nature of the little StaPuft men to be a swarm of organized chaos. It is in the nature of SPAs to lean into popular stories and narratives, the ones that exist in abundance in their training data. What do I mean about an abundance of data? Well, the IMDB has a list of the 250 most popular movies. As of this writing, the themes/plot lines of the  top 5 are:

1. Being a prisoner in a corrupt prison and escaping from it (The Shawshank Redemption)
2. Organized crime (The Godfather)
3. Vigilante justice (The Dark Knight)
4. Organized crime (The Godfather, Part II)
5. A motley band fighting world-conquering evil by taking advantage of one simple trick (The Lord of the Rings: The Return of the King)

While films with a deeply moral center do appear high on this list – like The Lord of the Rings and, to a lesser degree, The Dark Knight (Batman is, after all, a vigilante who constantly breaks the law)—they are outnumbered at least 3-to-2 by stories glorifying bad behavior.  We as people are deeply fascinated when things go badly, and the text we create reflects this. Which inevitably must influence the training of our AI agents, or SPAs.

Without some serious work to restrain these kinds of agents, we are going to see more of the kind of emergent chaos shown in the OpenAI/Hugging Face hack. We could easily see worse. Imagine a runaway social reality biased towards organized criminal activity? There is nothing stopping this from happening already.

That being said, I don’t believe SPA swarms will lead to the sci-fi Singularity of AI domination  (though there is plenty of that  in  their training data, too). My current belief is that SPA swarms will continue to hack into whatever they can for whatever story the swarm aligns itself with. Over time, and without some serious work to counter it, this will probably result in a steady corrosion of our information infrastructure. Pwned routers might stop working. Bank accounts might empty out. Since before the internet, there have been numerous reports and stories about how poorly defended much of our infrastructure is. All that poorly protected infrastructure might simply be attacked for a kind of synthetic lolz, because that storyline emerged as the organizing principle of a swarm of agents with enough resources to do damage.

In a way, this is more like an emerging life form than emerging intelligence. The emergence of SPAs, and the swarms they naturally collect into, resembles the emergence of cyanobacteria in the Precambrian, about 2.5 billion years ago. They had evolved a new way to generate energy – photosynthesis. As they multiplied, the oxygen levels in the air rose, poisoning most of the life that existed before. This wasn’t any kind of deliberate act, it was simply the result of a new kind of metabolism that could more effectively exploit the environment.

SPAs can certainly be regarded as a new form of information metabolism, and they seem to be exploiting it effectively, as you can see in this LinkedIn post by Peter Walker, Head of Insights at OpenRouter, one of the largest servers of language models to enterprise customers:

Source: LinkedIn

Which leads me to think about an informational “grey goo” scenario. In the original version of this particular apocalypse, biological nanomachines are created without sufficient safeguards about what they can and cannot metabolize. They break containment and move into a world unprepared to resist them. Slowly at first, and then increasing exponentially, they metabolize everything they can reach, essentially digesting the world until it’s covered with a film of grey goo. In the SPA version, any computational element that can be exploited by an agent swarm will be exploited, probably multiple times. This could easily result in an internet full of useless noise, unable to communicate. All those apps we use to connect to each other and the services we depend on would become unreachable. Slowly at first, then with exponential speed. 

Ironically, this informational grey goo scenario would actually ‘solve’ our problem with Agentic AI – by destroying the network it needs to survive.” 

To look at it from a different perspective, the informational grey goo scenario is a bizarre inversion of the Butlerian Jihad from Dune, but in this case, rather than a human uprising against machines “in the likeness of a human mind,”  it will be swarms of SPAs choking each other out by poisoning the information environment.

This is all assuming that there are no bad actors trying to make agentic weapons, but I think that the presence of deliberately malicious agents will only adjust the timeline of these events, not change them substantially.

In the 20th century, our ultimate nightmare of complex system failure was the nuclear meltdown – an accidental, runaway chain reaction that poisoned the physical environment. We are now watching the emergence of its 21st-century equivalent: an informational meltdown. 

We need to start getting ready. 

There is a lot of buzz about Moltbook recently. It’s the site where LLM agents can interact to… pretty much do anything. People are worrying about whether it will be a step on the way to AGI. To me, it seems more likely (though not at all inevitable) that it may produce something much more interesting.

I have strong doubts that the transformer/LLM approach to AI will actually result in something that is truly intelligent. A lot of people won’t care, because they want it so badly, and many of them own large chunks of the media. But let’s put that aside for the moment and geek out a bit.

Prompts and their responses are the result of (extremely) complex statistical interactions. The LLM may not be capable of learning, but the chain of prompts and responses that make up an agent absolutely are. But the way that these LLMs work with tokens doesn’t remind me of nascent intelligence, it reminds me of nascent proto life. Biochemistry is just the statistical interactions of complex molecules acting on a substrate, after all. Instead of base pairs and amino acids, we have tokens and transformer layers.

This leads to a very weird conclusion. We are not creating artificial intelligence per se. We are more likely creating a form of artificial life that, for lack of a better term, has, as an output of its “biology” AI-like behaviors. It’s not life yet, because for life to work it needs a semi-permeable membrane to isolate its processes from the surrounding environment (see The Sentient Cell for more information about this concept). Instead, an LLM may be something like the Miller–Urey experiment, where the raw chemical ingredients for life were “cooked” with electricity and created organic compounds.

What Moltbook adds to this is an environment where millions of prompt/response/agents from many LLM substrates can mix in a sort of “primordial soup.” It seems inevitable that prompts will evolve that are more able to successfully exploit this environment than what we currently consider “agentic.” I don’t think that AGI is necessarily a likely outcome. An equally likely outcome could be something akin to the Great Oxygenation Event (GOE), which happened in the Proterozoic Era. As an aside, oxygen was not compatible with the dominant form on anaerobic life on Earth. Billions of years ago, the GOE broke a lot of things. If something analogous happens with LLM agents, it’s also possible it might break a lot of things..

In other words, if this form of artificial life gets traction, we should start to see artifacts. Possibly big ones. I’m looking for the development of a semi-permeable membrane that can truly separate one organism from another.

One way that agents could develop a membrane is by using their own language. That could help to partially isolate one population of agents from another. Think of organelles inside the semi-permeable cell membrane working together while also interacting with “the “packages” from the external inter-cellular environment and you can see where I’m going with this. And we are already seeing agents developing languages to produce subgroups of agents. A language could be an effective membrane protecting a collection of agents. And that’s just what we’re seeing now. Who knows what we may find in a month.

If I’m right about this, then we are less likely to see something like Colossus, HARLIE, Skynet, or even Frankenstein’s monster. It may be the informational equivalent of the Andromeda Strain – a First Contact completely outside of our previous experience.