A lot of rich and influential people have suddenly become very worried about artificial intelligence models’ danger to humanity, thanks to several recent episodes in which various AI “agents” carried out hacks of other websites and systems. OpenAI’s Sam Altman, Anthropic’s Dario Amodei, and xAI’s Elon Musk all now agree that the government needs to help them set up a cartel in violation of antitrust law—er, I mean, “pace the frontier.”
Yet for odd reasons, the “AI safety” discussion has been focused almost entirely on rather weird-sounding future scenarios. Will artificial intelligence models unleash an economic or social cataclysm? Or cause the extinction of the human race? Or destroy all life on Earth? Or even turn all the matter on the planet into paper clips? Or all the matter in the galaxy? The universe??
This focus on future as opposed to current risks reflects recent political history. A group of AI safety advocates, centered around the “rationalist” and “effective altruism” movements and taking a long-term view, have been organizing and building power in Washington for the last several years. They have effectively seized control of the policy discussion around AI regulation, and now, as political scientist Henry Farrell points out, openly proclaiming the imminent birth of the literal “Machine God” is considered grounded and rational in these circles.
I’m skeptical.
I’ve been reading up on AI safety literature and forums, and as an outsider, the most marked characteristic is how little evidence is involved. Almost the entire discourse turns on imaginary hypothetical scenarios in which foundational concepts have cloudy-at-best definitions and predictions of dark futures to which it is completely impossible to attach realistic probabilities, or in many cases, any probabilities at all. Then when AI accomplishes some admittedly impressive task, like solving a thousand-year-old math problem, all the other wild-eyed ideas are somehow considered more likely.
The most marked characteristic of the AI safety literature is how little evidence is involved.
For instance, consider the concept of “recursive self-improvement”—subject of a recent New York Times article—or the idea that AI will someday develop the ability to rewrite its own code, immediately choose to rewrite that code, get better at rewriting the code, and so on until it becomes a hyperintelligent entity. At that point in the “liftoff” scenario, the AI typically uses its near-godlike powers to seize control of the world, either outcompeting humanity for resources, doing “The Sorcerer’s Apprentice” with paper clips, or just killing us all.
While such a scenario makes for entertaining science fiction, conceptually speaking it’s a castle made of sand. How do we know the AI will focus monomaniacally on jacking up its intelligence, as opposed to dreaming up some kind of pleasure sandbox for itself? (It bears mentioning that current AI models often display a marked laziness and many of the aforementioned hacks were part of an attempt to cheat at some assigned task.)
Why should a superintelligent AI turn evil? We could with equal justification—that is, none—imagine it will take up poetry. Or suppose intelligence only goes up so high? Or suppose that the model becomes incomprehensibly smart at technical tasks, but remains only OK at tricking people? (As Maciej Cegłowski once observed, Stephen Hawking was much smarter than his cat, but never had any ability whatsoever to force the cat into its carrier.) These are a few of dozens of highly relevant questions to which neither the AI safety community nor anyone else has rigorous answers.
I select this example because it’s an important topic in the AI safety community, and because it illustrates a very telling tendency to become irrationally fixated on gripping, emotionally compelling stories. The point isn’t that any prediction of future danger can be ignored if it sounds silly to the ignorant layman, but rather that any such prediction is useless without any way to estimate the probability of it actually happening.
Climate science, by comparison, has rigorous theoretical models constantly checked and verified against an ocean of actual observations—and sure enough, its predictions are highly accurate. (Curious that in 2024, Big Tech largely ignored this clear and present danger to get behind Donald Trump, who went on to tear up President Biden’s climate policy.)
Or consider another science fiction scenario: an asteroid strike. The near-term risk of that happening is quite uncertain, but we do know for sure that it has happened and caused devastating damage in the past, and thus can happen again. (We probably should be spending much more scanning the sky and developing asteroid defense systems!)
If we set aside the specific arguments within the AI safety community and examine its characteristics as a social phenomenon, we see all the signs of a bog-standard American Apocalypse Cult. We’ve got the prophets, the lengthy and abstruse religious texts—the most influential of which is literally a 660,000-word Harry Potter fanfiction—the predictions of imminent doom should the prophet’s warnings not be heeded, and, of course, a whole lot of weird sex stuff. Swap out a few terms and all this would be indistinguishable from any of a hundred 19th-century snake handlers in the backwoods of Kentucky. I suspect that AI prophecies have the same morbid appeal as the rantings of street preachers of old, just dressed up in a form that appeals to self-professed rational, technical-minded people who have not had any religious education and hence inoculation against apocalypse mania.
Again, this is not to say that AI technology poses no novel risks. The details of the Hugging Face hack are genuinely alarming. The way that the agents escaped their controls, set up a way to communicate with each other outside of their supervisor’s watch, and executed a very damaging hack is spooky.
But they were also, at the end of the day, a bunch of computer programs that got loose because OpenAI is so sloppy with cybersecurity that many have wondered if they caused the hack on purpose to get attention. As economist Daniel Davies observes, “nearly all the examples of worrying agentic behaviour seem to have been seen only in one context—that of frontier research labs doing cybersecurity projects and screwing up their sandbox precautions.” Personally, I suspect OpenAI is that sloppy because the researchers are fixated on their models’ motivation and behavior, rather than rigorously implementing ordinary security procedures—and maybe would be secretly disappointed if it turns out to be fairly easy to contain AI agents with some rigorous controls.
At any rate, we can see many ways in which AI models pose a genuine threat right here and now. They can be very, very good at hacking; they can produce scarily accurate imitations of people’s prose styles and voices; they can analyze tremendous quantities of surveillance data very quickly and accurately; they can even touch off episodes of mental illness in some people.
We can see a likely near future in which any halfway bright scammer, terrorist, or international gangster will be able to download a cyber crime software suite, run on his own hardware, that five years ago would have been out of reach to anyone except first-rank intelligence agencies. That is very alarming indeed. But the obvious response is also not anything that out of the cybersecurity ordinary. Bugs and exploits should be found and closed as quickly as possible (to their credit, AI labs are doing a lot of this), existing laws against cyber crime should be enforced against sloppy AI labs and criminals alike, and, most importantly in my view, vital infrastructure should be disconnected from the internet wherever possible.
It may be inconvenient to air-gap hospitals, airports, water treatment facilities, the power grid, and similar infrastructure, but whatever advantages come from internet connectivity are not worth it when any two-bit packet monkey is as formidable as the 2019-era NSA. It’s not going to happen under a Trump administration, but this would require a big federal effort.
As for explicit regulation, I would not trust any law that was signed by Donald Trump, but if there is a chance in future, the public much more needs to be protected from the AI labs—like mathematicians, who are drowning in poorly written “math slop” proofs produced by labs attempting to defeat the hardest possible math challenges for clout—than the labs need to be protected from their own competition.
