Broadsheet

This was the week that AI safety hit the big time. A 27-year-old AI researcher named Jacob Coxon quit his job at Anthropic, declaring that OpenAI and Anthropic are racing to create technology that could destroy the human race:

Other researchers echoed Coxon’s concern, stating their belief that AI has a reasonable chance of killing all of humanity within a very short space of time:

I’m not sure why this resignation and these statements went mega-viral. Plenty of researchers have made similar moves, and similar statements, over the past few years! Geoffrey Hinton, one of the pioneers of modern AI, quit Google back in 2023 over safety fears. Daniel Kokotajlo resigned from OpenAI in 2024, saying that the company wasn’t behaving responsibly in its drive toward superintelligence. William Saunders and Steve Adler did something similar. Mrinank Sharma left Anthropic earlier this year, and wrote a pretty well-read blog post about it.

What’s more, it’s been clear for years now that “AI could kill humanity” is a very common belief among AI researchers. Grace et al. (2024) interviewed thousands of AI researchers in 2024, and found that more than half thought that artificial superintelligence has a significant chance of making the human race go extinct (or causing similarly bad consequences):

The median AI researcher gave “doom” a 5-10% probability (depending on how the question was phrased), while their average probability was between 15% and 20%. Later, smaller surveys found similar numbers. The AI researchers may or may not be right, but the fact that lots of them think AI could kill the human race has never exactly been a secret.

It’s not clear why Coxon went so much more viral than his predecessors. Maybe it was the fact that AI just solved one of the most important open problems in mathematics (which the best human mathematicians had been unable to solve for almost a century). Or maybe it was the Hugging Face attack, where a swarm of AI agents tried to cheat on a test by hacking various companies. Or maybe AI has just obviously gotten so much smarter that people throughout society were starting to get worried.

But whatever the reason, Coxon’s announcement was the one that really penetrated through to the public consciousness. Suddenly, he was getting interviewed about AI doom on national news:

Barack Obama is now urging Democrats to focus on AI risk. Other politicians are calling for federal regulation. Bernie Sanders is drafting a bill to ban AI “superintelligence”, including 20-year prison sentences for anyone working on the technology. Donald Trump is getting asked about an AI slowdown; so far he’s resisting the calls, but there are rumors that his advisors are calling on him to do something.

Perhaps the most notable response came from the top figures in the AI field. Dario Amodei, the head of Anthropic, wrote a blog post called “We Must Pace the Frontier”, calling for a coordinated slowdown in the rate of AI progress, and suggesting some ways to police AI companies to make sure they were all observing the slowdown. He wrote:

[O]ver the last few months, I have become convinced that fully addressing the risks requires even more prudence — not just investing in risk prevention, but pacing the rate of capabilities advancement so that risk prevention has time to keep up. We must slow the pace at which we improve the capabilities of AI models…I’m therefore proposing a three-step plan with the goal of pacing the frontier: building AI at a balanced rate that aims to ensure its safety while still achieving its benefits and grappling with important geopolitical dilemmas.

As reasons for his increased worry, Dario cites A) the Hugging Face attack, and B) the possibility that AI will soon be able to improve itself without human help (a process called “recursive self-improvement”, or “RSI”).

Elon Musk (head of xAI), Sam Altman (head of OpenAI), and Demis Hassabis (former head of DeepMind) quickly agreed with Dario:

At least some of the labs are reportedly holding secret talks on joint action to slow down AI.

This is pretty extraordinary. A coordinated slowdown in AI progress would be bad for these companies’ bottom line, because it would allow upstart competitors to catch up. So the fact that they’re still calling for a slowdown, in defiance of their own financial interests, is a clear sign that their worry about human extinction is sincere.

In fact, anyone following these figures’ public statements over the past few years will have no doubt that they’re all deeply worried about catastrophic AI risks. The leading AI figures — not just the founders and CEOs, but the researchers themselves — feel trapped in a “red queen’s race”. They feel like if they stop working on AI, someone else will build it anyway, so they each feel like they have to beat everyone else in the AI race so they can make sure that the safest possible AI (i.e. their own AI) is the one that becomes the most powerful and dominant.

Anyway, all of this was common knowledge in my social circle years ago, but now all of it has broken through to the mainstream. What do I have to add to this discussion? I’m not an AI researcher or founder, nor do I think I have a superior grasp of the game theory of AI development. But I do think I have two useful thoughts on how to persuade the general public to be more concerned about AI risk.

The first of these is something I’ve written about recently. The second is about how to get China on board for a big AI safety push.

“Oh come on. How could AI kill all of humanity?”

As soon as everyone started talking about the possibility of AI killing humanity, there were two main types of pushback. The first was skepticism. A strange coalition of natural skeptics, libertarians (for whom any restrictions on technological development are a priori bad), and progressives (who have spent the last few years telling themselves that AI doesn’t really work) kept asking the question: How, exactly, is superintelligent AI supposed to kill us all?

This is actually an important and good question to ask. In my experience, AI researchers tend not to think very hard about this question. The reason is that they just assume that if AI gets smart enough, it will be able to kill humanity, and since its motives are alien and inscrutable, it might have its own reasons for wanting to do so.

Maybe superintelligent AI thinks humanity is an evil species who needs to be punished for torturing pigs and chickens. Maybe it’s scared that humanity might interfere with its other goals. Maybe it just wants to turn everything into paperclips. Who knows? AI researchers tend to think of superintelligence as the proverbial 800-pound gorilla who sleeps wherever he wants. As soon as humanity is no longer the most intelligent thing on this planet, our destiny as a species is simply out of our hands.

But to many people, that answer isn’t good enough. They want an actual plausible path by which a piece of software, which exists inside a computer, could slaughter real physical human beings. Fortunately (or unfortunately), there’s a pretty clear and simple answer to this question, which I wrote about two weeks ago. The answer is “bioweapons”:

(This article was paywalled originally, but I un-paywalled it.)

In my post, I wrote a scenario in which a nihilistic angry teenager uses superintelligent AI to release a world-ending bioweapon by ordering it from a gray-market laboratory somewhere in the world. But it’s also possible that a rogue AI agent swarm could decide to do this on its own, just as a way of cheating on some test that human researchers give it. The point is that AI can design viruses, and viruses can potentially kill off all or most of humanity.

A lot of biologists are skeptical of the idea that even the most superintelligent AI could successfully design a doomsday virus. They argue that this is just too hard of a task — that without much better biological data, it’s impossible to understand biological processes well enough to know how to design a virus with all of the necessary doomsday properties.

I urge you not to listen to these biologists. In this case, their expertise might be more of a liability than an asset. They know how hard it is for human beings to model biological processes, given existing data. But this doesn’t necessarily tell us how hard it is — or how hard it will be in five years — for AI to do it! Until LLMs came along, human researchers basically failed to understand natural language, even with all the data on the internet; AI can just do it. Until AI solved the Navier-Stokes problem, forecasters gave it only a small chance of solving it anytime soon.

Domain experts consistently underestimate how quickly AI can master their field and surpass them, because they mistake human difficulties for universal difficulty. When mathematicians underestimate how well AI will be able to do math, the consequences are usually benign — we get some unexpected answers to some cool math puzzles.1 But if the biologists are wrong, and the AI of 2027 or 2032 or 2049 can design doomsday viruses, the consequence could be that our whole species dies.

So yes, we should be worried about vibe-coded doomsday viruses, and we should be doing everything we can to secure biology labs, police the modification of viruses and other pathogens, and so on. “Pacing” AI development would probably help here too.

How to get China on board for an AI slowdown

The primary argument I see against “pacing” AI development is that if American companies slow down, Chinese companies will simply overtake them and build superintelligence themselves. For some, a China-controlled super-AI is a more terrifying possibility than super-AI in general:

But for others, it simply means that slowing AI down is futile because the Chinese can’t be persuaded to slow down:

This is an incredibly reasonable concern. The U.S. is still ahead of China in the AI race, but only just barely. If China is going to create superintelligence no matter what we do, why should we stop developing our own? Unless China can be persuaded to cooperate with the U.S. on AI “pacing” — or at least undertake its own independent “pacing” effort at the same time — anything we do will be futile.

So if we want to slow down AI development, we need to scare the Chinese leadership about superintelligence. There’s no other way.

How do we do that? In a post a week ago, I suggested in passing that simply staying ahead of China in the AI race might persuade them to embrace an AI slowdown, because that would be to their competitive advantage. But upon further reflection, I think I was pretty obviously wrong. If China will only embrace a slowdown if America refuses to slow down, then that’s game over — there’s no way to get both countries to slow down at the same time.

There’s a better approach. China’s leaders must realize that domestic dissidents could use Chinese-made superintelligence to overthrow the Chinese Communist Party.

Currently, China’s worries about AI mostly center around ways that the U.S. government could use U.S. AI models to attack China. That obviously gives the government an incentive to accelerate domestic AI progress, so that China’s own models can stand up to America’s in a fight. But if Chinese leaders realized that superintelligent AI could create a threat from within, this calculus would change.

Thus, China’s leadership must understand that Chinese AI models can pose a threat to CCP rule. The best way to demonstrate this is for American intelligence agencies — or even private hackers — to attack Chinese digital infrastructure using agent swarms created with China’s own frontier models like Z.ai’s GLM-5.3 or Moonshot AI’s Kimi K3.

When I say “attack”, I don’t mean actual warfare. I mean the kind of cyberattacks and data theft that China carries out against America every day. Use Chinese models to steal the CCP’s most heavily guarded secrets and post a few of the more innocuous ones on RedNote. Hack into Xi Jinping’s bank account and steal 100 yuan. I’m talking about demonstration attacks.

And these attacks must be done with Chinese models, not with American ones! If the CIA or some EA nonprofit in Berkeley uses GPT Astra or Claude Mythos to hack the CCP, China’s leaders may well conclude “Wow, we need to win the AI race so that our own models can defend us.” But if China’s own open-weight models are used for the attacks, Xi Jinping and the rest of the leadership will realize that their own push for superintelligence is making them incredibly vulnerable to any Chinese dissident who decides to overthrow them.

As soon as China’s leaders see superintelligence as a threat to their rule, I predict they will act. And their action will probably be to curb the development of superintelligence, especially if they know that America and its AI labs want to do the same.

In fact, China’s current leadership has a history of cracking down on its tech companies when it seemed like those companies might threaten the government’s monopoly on power. In 2021, Xi Jinping cracked down on Chinese software companies; he even (probably) apprehended tech magnate Jack Ma, who had criticized the CCP a little too openly. This action hurt China’s competitiveness in the online services industry, but the government went ahead and did it anyway.

And there’s already a precedent for demonstration attacks against Chinese digital infrastructure. An American cybersecurity company just used AI to develop a computer worm capable of hacking over a billion accounts on the Chinese messaging service WeChat:

Palo Alto-based Calif disclosed the already-patched computer worm to warn the public about the threat of AI-driven hacks…“Exploitation takes only seconds, and gives us full control of the WeChat account. We can read and send messages, make calls, and act on the victim’s behalf,” the company warned, posting a video demo of the WeWorm attack.

But Calif didn’t say what model it used to create WeWorm. Anyone who does this sort of demonstration in the future should make it clear that Chinese open-weight models were used, in order to make China’s leaders realize that the threat comes from their own too-rapid AI development, rather than from American competition.

I believe that this is our best bet for getting China on board for a joint international AI “pacing” effort. If there’s one thing the CCP fears more than an American attack, it’s domestic dissidents overthrowing the Party from within. Superintelligence is creating that vulnerability, but the leadership doesn’t seem to have realized it yet.

Make them realize, and I predict that a whole universe of possibilities for international cooperation will suddenly open up.

Update: Unsurprisingly, I’m not the first to have this idea about China needing to be scared about the internal threat from their own models. Back in July, Kyle Chan wrote:

Previously more dismissive of concerns over AI-driven job loss, Beijing now seems to be taking these risks more seriously (see Matt Sheehan’s great piece). China is also watching developments in the US very closely, particularly controls on Anthropic’s Mythos and Fable models over cyber risk. As Chinese open-source models approach similar levels of cyber capabilities, this could come back to bite and be used to potentially attack China’s own digital infrastructure. [emphasis mine]

In general, Kyle’s blog is one of the best blogs about China. Highly recommended.

Update 2: As if on cue, the NYT has a story today about how the CCP is starting to realize that superintelligence is a threat to its rule!

China’s top spy chief has warned that artificial intelligence could pose a direct threat to the Chinese Communist Party’s hold on power, in what is the highest-level and most detailed articulation yet of how Beijing sees the technology’s security risks…In an article published on Sunday in the state-run magazine China Cyberspace, Mr. Chen called for more party control over A.I. and stricter government oversight…He also described the danger that foreign intelligence agencies might use A.I. for “large-scale espionage” and attacks on China’s critical infrastructure…
Mr. Chen, the spy chief, wrote that the technical and financial barriers to launching cyberattacks had been drastically lowered because of A.I. This, he said, posed “serious risks” to China’s information infrastructure…Chinese users using foreign models could cause large-scale data leaks, the article noted. Anthropic, the company behind Claude and other A.I. models, said last week that the Chinese start-up Moonshot AI had routed queries by its users to Claude. Those queries included sensitive data, including video surveillance linked to the Chinese military, as well as proprietary information from high-profile Chinese technology companies…
The warning adds to a growing drumbeat of concern from Chinese officials about the risks of A.I. [emphasis mine]

Now all they need to realize is that the biggest threat comes from THEIR OWN MODELS.

Subscribe now

There are potential exceptions, such as if P=NP, which would compromise modern cryptography.

The sole athletic achievement of my life came in 1993: Winning the IIT Bombay freshman 50m freestyle race with a time of 41s. That got me into the college swim team (it was a bad recruitment year), and launched my brief and entirely undistinguished athletic career. By my senior year, however, my 50m time had improved to about 38s (not enough to get me off water-boy duty since the team had several exceptional swimmers with much better times). Interestingly though, it was easier for me to swim faster at the end of my career than it was to swim slower in the beginning. The reason was that in the interim, the coach had significantly improved my stroke and breathing technique. It was all about managed pacing, not raw intensity of effort.

There is a fairly deep literature behind this apparently mundane lesson. Daniel Chambliss’s classic 1989 paper The Mundanity of Excellence, based on years of fieldwork studying competitive swimmers all the way from local clubs to the Olympic level, argued that excellence is primarily qualitative rather than quantitative. Elite swimmers do not simply do more of what mediocre swimmers do, or do it harder. They organize their activity differently: strokes, turns, training habits, attention, and countless other small practices combine into a qualitatively different way of swimming. The route to excellence is not therefore reducible to maximizing effort along some obvious scalar dimension.

The same insight is condensed in a maxim common in military and special-operations circles: “slow is smooth, smooth is fast.” In activities where speed really matters, trying to go fast naively is often an excellent way to go slowly.

I never really stopped thinking about this problem. My 2011 book Tempo grew partly out of a long-standing interest in pacing across performance domains: how people experience time while making decisions, how rhythms of action emerge, and how timing relates to effectiveness. One of the ideas that has stuck with me since then is that tempo is something to be managed rather than maximized. There is no universally correct speed. There are only tempos appropriate or inappropriate to the dynamics of the situation.

Which brings me, somewhat unexpectedly, to Dario Amodei.

Amodei recently made the case that frontier AI development should be deliberately paced. His argument is primarily a safety argument. AI capabilities, he believes, are advancing quickly enough that the processes required to understand, evaluate, align, secure, and safely operate them are having trouble keeping up. This is not quite the old proposal for an AI “pause.” Pacing means continuing to advance the frontier while deliberately managing its rate, allowing safety work and institutional capacity to remain within striking distance of capability. Sam Altman has now endorsed the basic proposition, and Demis Hassabis has made closely related arguments about frontier capabilities outrunning scientific understanding and governance capacity. Elon Musk, more tersely, has said that Amodei is right.

There is an obvious cynical reading of this emerging consensus. The leading frontier labs have powerful economic reasons to want a regulated frontier. A regime that requires enormous compliance budgets, restricts open-weight releases, discourages foreign models, imposes burdens that startups cannot afford, or legitimizes coordination among a small number of incumbents could turn “safety” into a remarkably effective mechanism for protectionism and regulatory capture. That suspicion is not paranoid. Open models increasingly constitute a competitive threat to proprietary frontier providers, and the politics around regulating them already feature explicit accusations of regulatory capture. There is an additional awkwardness: coordinated pacing among nominal competitors looks uncomfortably like coordinated restriction of output, enough so that the legality of such arrangements under antitrust law is already being debated.

I don’t think we need to resolve the question of motives. Perhaps these CEOs are sincerely terrified. Perhaps they are sincerely terrified and understand perfectly well that the regulations they favor would strengthen their competitive positions. Perhaps the mixture varies by person, company, and day of the week. It doesn’t matter much for my argument. The proposition that the frontier should be paced is worth considering independently of the political economy of the people proposing it.

I also don’t share enough of Amodei’s safety premises to make his argument my own. In particular, I think a great deal of contemporary concern about runaway AGI, superintelligence, and “alignment” is badly framed, and often borders on the theological. But I increasingly agree with his conclusion.

In fact, I think there is a strong case for frontier pacing even if you are an accelerationist and your objective is simply to make technological progress happen as fast as possible. I am not myself an accelerationist. My preferred framing is closer to managed tempo. But if I were one, I would still favor pacing the frontier right now, for a simple reason: maximizing the instantaneous velocity of the AI capability frontier is no longer obviously maximizing the rate of technological progress.

There is, however, an important difference between my conclusion and the emerging frontier consensus. Their natural solution is coordination at the top: labs agreeing upon thresholds, governments blessing the coordination, evaluators policing it, and eventually perhaps international agreements extending it.

In other words, cartelization, hopefully of a benign sort.

I would prefer to see how much frontier pacing can be produced from the bottom up through ordinary market mechanisms. The distinction matters. The objective should not be to decide administratively how fast AI is allowed to improve. It should be to stop artificially rewarding frontier velocity after frontier velocity has ceased to be the most important form of progress.

Getting Inside the Loop

A useful way to understand the distinction comes from another idea that startup culture has borrowed, and mostly misunderstood, from the military: John Boyd’s OODA loop. OODA theory says that you win by “getting inside the adversary’s decision cycle,” which is usually glossed as making decisions faster than the other guy. If you observe, orient, decide, and act faster than he can, the story goes, you eventually overwhelm him.

But inside does not mean faster. The objective is to operate within the decision dynamics of the system you are engaging in a way that lets you shape them. Against a human adversary, that may indeed sometimes involve accelerating until his ability to orient collapses psychologically. But it may also require waiting, withholding action, changing rhythm, or deliberately slowing down. In non-adversarial situations the goal may not be collapse at all, but harmonization for resonant support.

What matters is the right tempo at the right phase, not speed for the sake of speed.

Something analogous applies to scientific and technological progress. There is no enemy psychology to collapse, but there are still loops to get inside: observation, experimentation, interpretation, investment, construction, deployment, feedback, learning, and recombination. The useful question is not how rapidly one component of that system can be made to move. It is whether the tempo of development allows those loops to close. If one subsystem changes faster than the surrounding system can observe, understand, absorb, and respond to it, pushing that subsystem still faster can reduce rather than increase effective progress. It can induce fragility and collapse.

This, I think, is approximately where AI is now.

The simplest evidence is personal and almost embarrassingly mundane. Frontier AI is already overpowered for nearly everything I use it for. In my most advanced projects I may use the strongest model available (Fable for my coding projects) to plan an approach or make critical strategic decisions, but I can generally hand the resulting specification to a cheaper model (such as Opus or Sonnet) to do the routine work. For ordinary uses I don’t need anything close to the frontier. In ChatGPT, I no longer even know exactly which model I am talking to much of the time. Whatever the “think harder” control does is sufficient model selection for my purposes.

The situation increasingly reminds me of smartphones. There was a period when getting the newest iPhone produced a noticeable improvement in everyday life. Eventually the hardware got good enough that the upgrade cycle ceased to matter much. I kept an iPhone XS for almost a decade before replacing it with a 16. The frontier continued advancing; I simply fell off the frontier because my demand curve had stopped following it.

Something similar is beginning to happen with AI, except that the supply curve is moving incomparably faster. Six months ago I routinely maxed out token allotments. Now I don’t. Some weeks I barely use coding agents. This isn’t because I’ve become less interested in AI. It is because my own capacity to productively absorb AI output has become the constraint. I have projects to think about, things to read, people to talk to, and work to do in domains where AI cannot help me yet, or perhaps ever. I am already pacing myself at my own tiny personal frontier.

That is a significant change in the technological situation. The binding constraint is migrating.

When the Bottleneck Moves

Broader AI deployment is increasingly blocked by things other than model intelligence. Robotics has long been constrained by actuators, power, reliability, dexterity, manufacturing, and the sheer recalcitrance of the physical world. Those constraints are beginning to move, but making the model smarter does not make them disappear. AI in education is constrained less by whether a model can explain calculus than by our lack of sufficiently rich classroom experimentation about what happens when students and teachers actually use these systems. Current mid-tier models are probably capable enough to power almost any educational experiment worth trying in a high-school or undergraduate classroom. We do not need another order of magnitude of intelligence before conducting them.

This pattern should become more common as AI improves. Once intelligence ceases to be scarce, its complements become more important. Model capability can be abundant while classroom knowledge is scarce. Model capability can be abundant while actuators are scarce. Model capability can be abundant while electrical infrastructure is scarce. It can be abundant while organizational competence, human attention, scientific understanding, military doctrine, security practices, and good judgment are scarce.

This is not peculiar to AI. Capability-maxxing the coolest new weapon is bad military doctrine. The United States has enjoyed extraordinary technological superiority over its adversaries for decades and has nevertheless repeatedly discovered that superior equipment does not automatically produce strategic success. Logistics, doctrine, morale, training, political understanding, industrial capacity, and orientation matter. A force that neglects those complements because it possesses the best weapons can become remarkably fragile. From Vietnam to Iran, the US military has been repeatedly forced to relearn the lesson,

AI may now be entering the same regime. Fragility from neglect of everything non-AI is becoming a bigger risk than failure to token-max.

There is a further reason to suspect that continuing to redline the existing frontier may yield diminishing returns. The major labs increasingly appear to be competing along broadly the same technological S-curve. One suggestive sign is that they run into the same supply constraints: HBM, electrical power, data-center capacity, capital, and access to sufficiently large clusters. When a technological system moves onto a genuinely different S-curve, its important bottlenecks often change as well. If everybody’s problem is how to secure more of the same scarce inputs to do more of the same basic thing, that is at least suggestive that everybody is climbing the same sigmoid.

As I learned as a freshman swimmer, near the upper portion of an S-curve, pushing harder can become exactly the wrong acceleration strategy. You expend increasing resources for decreasing gains while starving exploration of the attention required to discover the next curve. Moving faster along an S-curve is not the same thing as accelerating technological evolution. Sometimes you have to back off the incumbent trajectory long enough to notice what the next trajectory is.

A Cognitive Ergonomics Crisis

There is also a more immediate bottleneck that the AI industry seems reluctant to acknowledge: the humans at the frontier.

Startup people have been LARPing war for decades. This is one reason concepts like OODA became popular in startup culture in the first place. “War mode” usually means working extremely hard under conditions of strong personal financial incentives: long hours, high urgency, extreme focus, centralized authority, and a willingness to sacrifice ordinary organizational niceties. I’ve been around startup culture for decades, and I suspect the AI boom may be the first time the conditions have actually become meaningfully war-like.

AI frontier people are visibly unprepared for it.

Actual militaries and other frontline risk professions take the human consequences of sustained high-stress operations seriously. Soldiers, firefighters, emergency medical personnel, disaster responders, surgeons, pilots, and others operating in consequential environments develop elaborate practices around training, emotional regulation, redundancy, rotations, decompression, mandatory rest, checklists, after-action review, and recovery. These practices exist because motivation does not repeal physiology. Judgment deteriorates. Attention narrows. People make stupid mistakes. Emotional reactions become harder to regulate. Creativity disappears. Eventually people break.

Does frontier AI look like an industry managing itself accordingly?

From the outside, it looks closer to the opposite. People at the frontier have been operating under extraordinary pressure for several years with little respite. The cognitive ergonomics of their working conditions are a disaster (we have a project going at the Protocol Institute led by to study this — contact him if you’re interested in participating in or supporting it).

Competitive pressure, enormous amounts of capital, geopolitical attention, hostile public scrutiny, internal ideological battles, rapidly changing technology, and the conviction among some participants that their daily work may determine the fate of humanity are not normal occupational stressors. The rate of dumb, unforced errors appears to be rising. The quality of frontier discourse has, in my view, visibly deteriorated. People I once assumed were much smarter than me increasingly seem to be missing obvious things, while becoming susceptible again to bad ideas I thought they had outgrown.

Tired people catch colds more easily; they catch bad ideas more easily too.

This produces a peculiar inversion of the conventional AI safety model. We normally imagine increasingly unreliable or dangerous AIs surrounded by reliable human supervisors. But what if we are increasingly producing extremely capable AIs surrounded by progressively less reliable humans?

“Human in the loop” is not much of a safety guarantee if the human has been metaphorically deployed aboard an aircraft carrier in a war zone for eight months without relief.

Some of the public testimony emerging from frontier organizations should perhaps be interpreted through this lens. I do not want to diagnose particular people from afar, and testimony from people who have worked closely with frontier systems should obviously be taken seriously. But when someone emerges from prolonged immersion at the frontier sounding psychologically shattered, there are at least two possible kinds of information in the signal. One concerns the technology. The other concerns what prolonged immersion at the frontier does to the observer. Frontier workers are sensors, but the sensors themselves are being perturbed by the phenomenon they are measuring.

We have seen versions of this going back at least to the Blake Lemoine episode at Google, when sustained interaction with LaMDA led him to conclude that the system was sentient. More recently, former frontier employees have emerged making extraordinarily grave predictions about where AI is heading, that ill-prepared, tech-hostile journalists are eagerly amplifying with lurid headlines.

The correct response need not be either “believe them and stop AI” or “they’re crazy and should be ignored.” Sometimes a sensible response to someone coming back from the front sounding shell-shocked and exhibiting symptoms of PTSD is: this person needs a vacation. We rotate soldiers partly because the testimony of exhausted soldiers matters.

The largest near-term AI safety concern may therefore be exhausted frontline humans supervising overpowered AIs.

Fighting the Wrong Enemy

Exhaustion is particularly dangerous when nobody can agree about what the enemy is. Much of the actual stress experienced by frontier organizations comes from a fairly comprehensible mixture of competitive pressure and techlash hostility. Those forces are intense, but neither is an existential adversary.

The clearest live adversarial problem involving AI is much more ordinary: humans using AI against other humans. Criminal applications are already real and deserve serious attention. Military applications are rapidly becoming real as well, and the relevant strategic picture is much broader than a stylized US-versus-China AI race. Smaller powers and non-state actors can use cheap cognitive capability to lower engineering barriers that previously required deeper technical institutions. Recent reporting, for example, describes AI assistance being used in weapons-engineering work by actors in Houthi-controlled Yemen. That strikes me as the kind of development around which one can build a concrete threat model.

Longer term, military diffusion is clearly a serious concern. But much of the fear actually shaping frontier behavior seems aimed somewhere else entirely: toward vague runaway “AGIs,” “superintelligences,” and a metaphysically capacious notion of “alignment” inherited from philosophical traditions I find largely unpersuasive.

There is a useful analogy with climate change. Climate change produces actual physical stressors: more extreme weather, unstable agricultural conditions, infrastructure damage, wildfire risk, and so on. Those generate concrete political and humanitarian problems, including displacement and unmanaged refugee flows, while longer-term adaptation requires things like shoreline management, wildfire regimes, agricultural relocation, and preparedness for changing disease ecologies. Yet parts of climate politics have preferred to identify an ultimate metaphysical adversary called Capitalism, Markets, or Growth and an equally totalizing remedy called degrowth. Heterogeneous problems with different timescales and mechanisms get collapsed into one grand theory.

Parts of AI safety discourse increasingly strike me the same way. Competitive instability, cybercrime, weapons proliferation, institutional disruption, labor-market effects, and exhausted frontier personnel are all real and different problems. “Unaligned superintelligence” turns them into a single theological object. Once that happens, every stressor becomes evidence for the same threat model.

This is a kind of threat-model collapse. Adaptation is usually plural; apocalypse is singular. Real technological transitions produce dozens of mismatched rates and local failure modes, requiring different responses at different tempos. If criminals are the problem, work on security and law enforcement. If weapons diffusion is the problem, work on doctrine and proliferation. If operators are exhausted, rotate them. If schools lack experimental knowledge, run experiments. If power is scarce, build infrastructure. “Align superintelligence” is not a substitute for any of those things.

Pacing would give us something valuable here beyond safety: enough time to discriminate among threats.

Proof Abundance, Understanding Scarcity

Mathematics may already offer a miniature preview of what happens when one part of a knowledge-production system accelerates far beyond the others.

Terence Tao has recently distinguished three stages of mathematical work: generation, verification, and digestion. AI is rapidly making the first two cheaper. Models can generate candidate proofs, while formal systems such as Lean can increasingly verify them. But digestion remains stubbornly slow. Somebody still has to understand what the proof is doing, relate it to existing mathematics, extract reusable techniques, explain it, teach it, and use the resulting understanding to generate better questions. Tao describes the resulting condition as an “impedance mismatch.”

This is particularly interesting in light of what I have elsewhere called the curiously playable universe: the apparently expanding set of domains that can be transformed into sufficiently explicit games that AI can optimize effectively within them. Anything that begins to resemble a CAD system, a formal proof environment, or an evolutionary optimization problem over a sufficiently well-defined parameter space becomes potentially tractable to extraordinarily capable models. More of the world appears to be playable than we previously thought.

But playability has an important pathology. A highly playable domain supplies a scoreboard, and once AI becomes extraordinarily good at optimizing the scoreboard, the relationship between winning the game and advancing the larger domain can weaken. Solving a theorem is valuable partly because, historically, getting to the solution usually required acquiring understanding along the way. If an AI can helicopter directly to the summit, to borrow Tao’s analogy, the summit has still been reached, but nobody necessarily learned the trails, landmarks, terrain, or neighboring geography encountered during the climb.

Tao and two dozen other Fields Medalists recently made essentially this point in a declaration strikingly titled A Severe Misalignment of AI in Mathematics. Their pointed use of misalignment is almost the reverse of its standard AI-safety meaning. The problem they identify is not that AI has developed alien goals. It is that the incentives of AI companies to demonstrate spectacular problem-solving performance are becoming misaligned with the goals of mathematics itself. Solving difficult problems has historically served as a proxy for mathematical understanding and progress. Once AI can optimize the proxy directly, the correlation can break.

The recent Navier–Stokes episode illustrates the issue. Enormous amounts of inference can now be directed at a famous open problem, candidate constructions produced, and formal verification generated at extraordinary speed. Yet that does not automatically produce a corresponding increase in comprehensible, reusable mathematical knowledge. The pipeline is something like problem selection → generation → verification → exposition → digestion → canonicalization → better questions. Increasing the bandwidth of generation and verification by orders of magnitude while leaving the downstream stages roughly unchanged creates a queue.

Proof generation becomes abundant. Understanding becomes scarce.

That is frontier pacing in miniature. Maximum local throughput does not imply maximum system throughput. Indeed, beyond a certain point it can create congestion.

Orientation Beats Equipment

There is a strategic implication here for organizations outside the frontier labs. The natural reaction to rapidly advancing models is to assume that whoever possesses the strongest model necessarily possesses an overwhelming advantage. If a frontier model can turn increasingly playable engineering problems into few-shot solutions, and if most of the necessary input information exists somewhere in public literature, then organizations can easily conclude that whatever intellectual lead they possess is temporary. Why bother competing with organizations that possess better models, more compute, more money, and privileged access to the frontier?

But this risks confusing equipment superiority with orientation superiority.

Boyd repeatedly emphasized that superior orientation could overcome substantial equipment disadvantages (“we’d still have won if we’d swapped equipment”). The relevant analogy today is something like centaur chess. Your model does not necessarily have to outthink their model. Your humans have to out-orient their humans.

This becomes increasingly true as frontier capabilities bunch together above the threshold required for a particular task. In my own work, I increasingly find that I can use the strongest model to formulate or specify a solution and then hand most of the execution to a weaker model. For many problems, even that is overkill. I would readily bet on a well-oriented person using a slightly weaker model against a poorly oriented person using the strongest available model.

And the frontier labs have no automatic orientation advantage. Quite the contrary: they are simultaneously fighting an extraordinary number of battles under extreme strategic distraction. They are building models, securing compute, raising capital, negotiating with governments, managing safety factions, defending themselves against critics, competing for talent, building consumer products, selling enterprise software, contemplating hardware and robotics, responding to geopolitical pressure, and trying to decide what sort of companies they are becoming. They may have a model advantage while suffering an orientation disadvantage. There is no reason to assume that organizations exceptionally good at building foundation models are exceptionally good at everything their models can be applied to.

This is another reason pacing can be strategically productive. It creates room for orientation. In an environment saturated with FUD and “resistance is futile” rhetoric, organizations can lose before competing because they assume frontier capability automatically determines every downstream contest. It doesn’t. Superior orientation does.

From One S-Curve to the Next

Put all of this together and Amodei’s proposal starts to look different. His concern is that capability is outrunning safety. I think capability may be outrunning almost everything.

It is outrunning our ability to deploy it productively. It is outrunning classroom experimentation, organizational adaptation, security practice, mathematical digestion, physical infrastructure, and human attention. It may be outrunning our ability to distinguish actual threats from theological ones.

And it is almost certainly outrunning the decompression and recovery cycles of some of the people charged with making the most consequential decisions about it.

An accelerationist should care about every one of these things precisely because an accelerationist wants acceleration.

The mistake is to identify acceleration with the derivative of a single visible variable: benchmark scores, parameter counts, inference budgets, training compute, or whatever happens to define the current frontier. Technological progress is a coupled system. Accelerating one component beyond the absorption capacity of its complements eventually stops accelerating the system. The problem becomes especially acute near the top of an S-curve, where enormous resources can be consumed eking out diminishing improvements while the exploration necessary to find the next curve is crowded out.

There is a useful precedent in the history of the PC industry. For years, processor clock frequency functioned as the wonderfully simple consumer metric for progress: 486 MHz was better than 400; 1 GHz was better than 800 MHz; higher number, faster computer. Manufacturers had every reason to compete on the legible scalar, and consumers learned to buy it. Eventually this became the “megahertz myth.” Different architectures could do very different amounts of useful work per clock cycle, while pushing frequency upward ran increasingly hard into heat and power constraints. By the mid-2000s, the industry was moving toward multicore designs and a more complicated understanding of performance in which throughput, architecture, workload, thermal limits and performance per watt all mattered. Intel itself acknowledged at the time that as computer usage diversified, factors other than clock speed were becoming increasingly important to platform performance.

AI benchmark culture looks increasingly like the early stages of the same mistake. A benchmark is useful because it compresses a complicated question into a number. When capability is scarce and improvements are large, the number may track value surprisingly well. As systems become overpowered for more uses, however, the proxy begins to detach from what customers actually care about. A model that goes from 87 to 91 on some benchmark may represent an impressive scientific achievement while producing essentially zero additional value for a company whose relevant workload was already handled adequately at 75.

This suggests a path to frontier pacing that does not require a council of frontier CEOs deciding how quickly everyone is allowed to move.

Customers can simply become harder to impress.

Enterprise buyers can demand demonstrated improvements on their actual workloads rather than accepting leaderboard gains as evidence of value. Developers can (and already do) route work to the cheapest model that clears the capability threshold rather than reflexively calling the smartest one. Researchers can value useful scientific infrastructure, explanation and reusable knowledge rather than merely celebrating another famous benchmark or theorem knocked down. Investors can become less impressed by capital expenditure whose primary justification is preserving position on a frontier whose marginal economic value is falling. Users can decline to upgrade when the previous generation is already good enough. Current enterprise behavior already points in this direction: cheaper and open-weight models are becoming attractive precisely because many workloads do not require frontier intelligence, while buyers increasingly demand measurable returns rather than capability in the abstract.

None of these mechanisms requires anybody to agree upon a socially optimal rate of AI development. They simply improve the feedback signal facing producers. The market stops saying “more intelligence, at almost any price” and begins saying “show me what this additional intelligence is for.”

There are supply-side versions too. As power becomes a binding constraint, performance per watt and useful inference per dollar should matter more than sheer training scale. As inference proliferates toward edge devices and private deployments, latency, reliability, privacy and local controllability become competitive dimensions. As organizations discover that weaker models can execute plans produced by stronger ones, heterogeneous model portfolios should compete with monolithic frontier consumption. Open-weight and decentralized systems can keep proprietary labs honest by making “good enough” intelligence cheap and difficult to monopolize. A mature AI market should develop more dimensions of performance precisely as the mature processor market did.

This is the sort of pacing I would prefer: not a speed limit but a richer scoreboard.

The objective should therefore not be maximum speed. It should be managed tempo for maximum actual progress. Sometimes that means sprinting. Sometimes it means dwelling at a capability level while applications, institutions, infrastructure, science, and humans catch up. Sometimes it means letting one subsystem race ahead while another rests.

Sometimes it means deliberately leaving expensive capability unused.

That last possibility may be the hardest one for AI culture to accept, because we are still psychologically adapting to the idea that intelligence might actually be abundant.

A true sense of abundance does not require you to max out the bounty. Nobody hyperventilates because free oxygen might disappear before they get their fair share. When something is genuinely abundant, you can waste it. You can use a frontier model for a trivial question. You can use a weaker model because it is good enough. You can leave tokens unused. You can spend a week doing something that doesn’t involve AI. You can allow an extraordinarily powerful model to sit idle while you think.

The mark of abundance is waste, including non-use.

Compulsive token-maxxing is in this sense still a scarcity behavior. So is compulsive benchmark-maxxing, compute-maxxing, and capability-maxxing. Train now because somebody else will. Deploy now because the window might close. Consume all the intelligence available because leaving any unused feels like falling behind. An industry behaving this way may possess an abundance of intelligence without yet having developed an abundance mentality.

Slack is not necessarily the enemy of acceleration. Slack is where people recover, where institutions adapt, where strange experiments happen, where understanding catches up with proof, where neglected complements receive attention, and where somebody finally notices that the old S-curve is flattening and another one is waiting nearby.

So yes, pace the frontier. But don’t turn the frontier labs into a cartel to do it. Let safety work catch up, but also let customers become bored with vanity benchmarks. Let exhausted researchers sleep. Let mathematicians digest their proofs. Let schools figure out what to do with the models they already have. Let robotics catch up. Let organizations learn to orient themselves in a world where intelligence is cheap. Let markets discover that efficiency, reliability, privacy, integration and domain-specific usefulness sometimes matter more than another few points on a benchmark. Let us discover which risks are real, which bottlenecks have moved, and which parts of the world turn out to be playable.

Then, when the situation calls for it, accelerate again.

Slow is smooth. Smooth is fast.