Broadsheet

Some of the best long-form voices out there right now: Daring Fireball, Simon Willison, Charity Majors, Jason Kottke, and Stratechery.

This essay was written with Cindy Cohn, and originally appeared in Lawfare.

One of the many legacies of the terrorist attacks of Sept. 11 is the government-wide shift from targeted surveillance—such as individual wiretaps or pen register/trap and trace orders—to mass surveillance techniques—such as tapping into the internet backbone or mass collection of telephone or internet metadata. The legal and technical architecture of modern mass surveillance, initially framed as a necessary defense against terrorist threats, has grown far beyond that justification and national security in general. Mass surveillance is now a routine tool used by law enforcement. ICE uses it in immigration actions and against people exercising their First Amendment rights to protest. It’s also increasingly part of private security systems, such as facial recognition at venues such as Madison Square Garden and networked Flock license plate capture systems on roads and in parking lots.

The interrelation between private and governmental mass surveillance is worth examining. Surveillance is the business model of the internet; companies like Google and Facebook constantly spy on their users’ behavior. From the National Security Agency relying on data collected by telecommunication and internet companies, to local sheriffs and ICE agents relying on cellphone location data and privately managed automatic license plate readers, governments primarily obtain the mass surveillance information through private companies. Increasingly, access doesn’t just come through legal processes, either. FBI Director Kash Patel recently confirmed in congressional testimony that the agency is purchasing information on Americans from data brokers and intends to continue to do so.

This pipeline from private collection to governmental collection means that as companies collect more information for surveillance capitalism purposes, more is available to law enforcement as well. And as the technology for mass surveillance and analysis improves, especially with the increased use of AI technologies, the problems attendant to mass surveillance grow as well.

After 9/11, the idea that the government could surveil the population to safety took hold. In 2001, the fear of terrorism reached a frequency and intensity never before seen. Along with that came the fear that the enemy could be anyone, anywhere. As a result, the government’s response was to watch everyone, everywhere. This line of reasoning underpinned the shift from targeted to mass surveillance. Or, in the words of an internal National Security Agency (NSA) presentation that was made public as part of Edward Snowden’s 2013 disclosures, a government that can “Collect it All,” “Process it All,” “Exploit it All,” “Partner it All,” and “Sniff it All,” will ultimately, “Know it All.” Similar rationales support the rise of domestic mass surveillance: if law enforcement could see and hear everything, it could more effectively interdict and solve serious crimes.

The national security community has never provided a full analysis of the costs and benefits of these mass surveillance programs, either in terms of taxpayer dollars or diversion of resources from other efforts—or any demonstration that those techniques stopped attacks that otherwise they would not have been able to prevent. While the NSA occasionally presents examples of the successes due to its mass surveillance programs, especially when those techniques are under public pressure, the examples also regularly fall apart upon serious scrutiny. And even if some utility exists, it must be seriously weighed against the costs.

Similarly, there has never been any comprehensive analysis about whether domestic immigration or law enforcement’s use of these techniques actually makes people safer, or whether other techniques could produce the same results. Instead, both the police and the companies selling these tools float anecdotes and dubious data. For example, Flock’s data equates the number of law enforcement hits in their database with actually solving crimes.

Twenty-five years after 9/11, it seems reasonable to step back and evaluate the costs of this shift to mass surveillance, especially in terms of Americans’ rights and freedoms.

The Shift

The easiest place to see a shift to mass surveillance was in the government’s decision immediately after 9/11 to collect Americans’ telephone records. The program started under an argument of pure executive power as the “President’s Surveillance Program.” But in 2006, that argument secretly shifted to a novel interpretation of Section 215 of the Patriot. Act which had only previously authorized more targeted access to record. While some media and public interest organizations struggled to force the government to reveal the program as early as late 2005, the government only officially confirmed it after the 2013 Snowden disclosures. In 2015, the Second Circuit Court of Appeals rejected the government’s interpretation of Section 215 as allowing mass collection of telephone records. Later the same year, Congress passed the USA Freedom Act. While this new law still allows collection of a tremendous amount of domestic telephone records, it ended the indiscriminate mass collection that had occurred for nearly fourteen years.

Other shifts to mass surveillance continue through today. The NSA launched its Upstream program, which involved intercepting both metadata and content from key telecommunications junctures inside the U.S., soon after 9/11. It was also initially conducted under a claim of purely presidential authority. This program was brought under marginal congressional and programmatic (not targeted) Foreign Intelligence Surveillance Act (FISA) court review via Section 702 of the 2008 FISA Amendments Act. In 2017, more than15 years after its inception, the NSA ended content searches due to FISA court pressure, but the mass collection continues.

Despite the stated goal of conducting mass spying only on people outside the U.S.—which itself is problematic given international law’s requirement that surveillance be both necessary and proportionate—mass surveillance collects a tremendous amount of U.S. persons’ communications. This can happen because people communicate with people abroad, or because of overcollection—when government agencies gather far more personal data on non-targeted US persons than authorized by law. The concerns about collecting Americans’ data on U.S. soil led Congress to allow the program to officially expire in 2026, although the previously-approved mass surveillance itself continues until at least Spring of 2027.

The shift to mass surveillance would be notable enough even if it remained only a strategy of the intelligence community. It has not. Americans are awash in mass surveillance. Networks of automated license plate readers such as those offered by Flock and Vigilant Solutions blanket both public and private roadways and parking lots. These networks often allow searches by law enforcement, including across jurisdictions. They are, for example, being used to track people seeking abortions across state lines. Facial recognition tools, once the province of only the more elite parts of federal law enforcement, are increasingly used by Immigration and Customs Enforcement agents on immigrants and protesters, in airports by the Transportation Security Administration, as well as by private entities. And, of course, modern phones track users’ locations constantly—and that information is readily available to law enforcement, often with only minimal process protections.

Constitutional Costs

Regardless of the murkiness of its actual usefulness, the shift from targeted to mass surveillance has profound implications for Americans’rights. It has created risks that have become increasingly evident, especially under the Trump administration.

At a basic level, the Fourth Amendment guarantees that citizens can be secure in their “persons, houses, papers and effects” from unreasonable searches. Warrants breaching that security should be supported by probable cause and particular descriptions of the place to be searched and items to be seized. Mass surveillance turns that promise on its head, allowing access to our “papers and effects” by the government without individualized suspicion or a particularized description of what data is being seized, much less probable cause. This protection was in response to colonial British misuse of writs of assistance, which authorized indiscriminate searches rather than targeted ones.

The justifications for exempting mass surveillance from constitutional protection vary. For Section 702, the government has taken the position that U.S. persons’ communications caught up in the dragnet, either due to overcollection or because they were communicating with someone outside the United States, do not require a warrant prior to initial collection or secondary access by the FBI and several other agencies. The argument is that if the initial collection was not aimed at Americans, the information is free from constitutional protection for any later uses, even for reasons far afield from the initial rationale for collection.

Other arguments rest on the claim that metadata is outside the Fourth Amendment, despite its demonstrated ability to reveal intimate details of all of our lives. Still others rest on the Supreme Court-created Third Party Doctrine, which holds that the Fourth Amendment does not apply to data shared with companies that provide us with services. Some turn on whether analysis by machine counts, claiming that only “human eyes” matter—a particularly troubling argument with the rise of artificial intelligence. What’s more, the government has used doctrines like standing to limit the ability of those subjected to mass surveillance to seek constitutional protection. No matter the argument, the goal is the same: to place the mechanisms and fruits of mass surveillance outside the protections of the Fourth Amendment.

The overarching truth is that, due to the concerted efforts by the government since 9/11, and the rise of technologies in recent years, the slice of Americans’ lives and data that are actually protected by the Fourth Amendment has shrunk significantly in the past 25 years. Together, with the technical capabilities of mass surveillance and the increased ability for that data to be analyzed using AI tools, the “security in our papers and effects” that the constitution promises seems increasingly illusory.

In addition to the Fourth Amendment, mass surveillance creates tensions with the First Amendment. The Constitution has long recognized that the right to freedom of speech requires a zone of privacy against governmental surveillance. The right to anonymous speech as well as the right of association both recognize the chilling effect that surveillance creates for people saying unpopular things or attempting to organize for political or other societal change. Mass surveillance grants the authorities the ability to track those people, both in real time and historically, that is inconsistent with actual techniques of freedom of speech and assembly.

That is why the recently released 2026 U.S. Counterterrorism Strategy is so troubling. On page seven, the White House expressly states that it intends to target domestic activists with its heretofore foreign-targeted powers. It says that the government “will prioritize the rapid identification and neutralization of violent secular political groups whose ideology is anti-American, radically pro-transgender and anarchist” and “will use all the tools constitutionally available to us to map them at home, identify their membership, map their ties to international organizations like Antifa.” While framed as targeting “violent” groups, it’s clear that the government intends to use its national security tools, presumably including the tools of mass surveillance, against Americans in ways that will create profound tensions with the First Amendment rights of people to organize and communicate privately.

Costs Due to Mistakes and Abuse

Even assuming some utility from mass surveillance—a fact we do not dispute, even if the public record is shaky and conclusory—the history of both the national security and domestic uses of mass surveillance confirms that these tools are inevitably misused, and that mistakes have impacted huge numbers of Americans. The past twenty-five years have demonstrated that it is not possible to surveil the entire US population while staying within the bounds of even a very generous legal framework like Section 702.

As Rep. Zoe Lofgren (D-Calif.) recently stated in discussion of Section 702 in an interview with Tech Policy Press: “backdoor searches have been used improperly for protestors, 19,000 campaign donors, members of Congress, journalists, government officials, a state court judge who had complained to the FBI about police misconduct. It has been abused substantially in the past.” The NSA experienced so much abuse of its mass surveillance tools by actual or aspiring romantic partners and ex-spouses that an internal name emerged for it: “LOVEINT,” or Love Intelligence.

That same pattern of abuse is now emerging at the domestic law enforcement level. A Texas police officer misused, and then lied about, using license plate readers to track a woman suspected of seeking an abortion. Multiple law enforcement officials have been accused of tracking people they either wished to have a relationship with or who were their exes. And mass surveillance technologies have been used to track both immigration targets and citizens engaging in their First Amendment-protected right to track and record the police.

Mistakes are inevitable with collections of data of this size and scope. The history of the FISA court’s reviews of Section 702 is littered with examples of the NSA not being able to follow its own rules limiting the scope of what it collects and analyzes, even after having been given multiple chances by the court. On the local level, the technical protections that Flock, for example, put in place have repeatedly been insufficient to stop “accidental” sharing its data with out-of-state law enforcement. These mistakes have fueled growing efforts by local communities across the country to remove license plate readers. Those efforts should be the first step in a broader reconsideration of mass surveillance.

More generally, ubiquitous surveillance carries a real societal cost. The chilling effects are real and pervasive, and they tend to fall hardest on the most marginalized members of society. Moreover, social progress requires the ability to experiment in secret. It’s hard to imagine a society progressing morally to the point of accepting and legalizing things like marijuana use or gay marriage if the earliest signs of that shift are snuffed out because of overzealous surveillance.

Reversing Course

While a cost-benefit analysis is not the best frame for deciding constitutional rights, it is a place to start to evaluate government policies. If the costs are too high and the benefits too small, what should the public do? While the policy and legal frameworks can be individually complex, mass surveillance is a problem in all of its applications. So too should solutions be comprehensive rather than piecemeal.

One comprehensive strategy is to reset the promise of the Fourth Amendment and recognize that a warrant is required prior to collection, access or use of information gathered through mass surveillance. This would apply to collections that include U.S. persons, whether done for national security or domestic purposes. This protection would apply regardless of whether the information is in the form of metadata. It would apply regardless of whether the information is held in homes or by services people rely on, such as telephones, internet or social network providers, or by private entities utilizing mass surveillance for their own purposes. By passing this legislation, Congress could ensure this rejection of mass surveillance, and include real enforcement such as a private right of action and an automatic exclusionary remedy in criminal prosecutions. The courts could also recognize this protection of “papers and effects” directly as a plain language interpretation of the Fourth Amendment.

There are already a number of efforts that take on pieces of mass surveillance. Section 702 has expired and should remain so. This was due largely to efforts to block the “back door” access to Section 702-collected data without warrants. The bipartisan “Fourth Amendment is Not for Sale Act” would prevent the government from purchasing data that it would otherwise need a warrant to obtain. The Supreme Court itself has already been chipping away at the Third Party Doctrine, with a recent step in the rejection of mass geofence warrants—warrants seeking the identities of individuals based upon their proximity to a crime—in Chatrie v. United States. Now, such warrants fall, at least initially, under the Fourth Amendment.

A more comprehensive approach would also address mass surveillance carried out by private companies, and to ensure that Americans have the right to encrypt and secure their data. There are many reasons the United States would benefit from a comprehensive privacy law—and curbing mass surveillance is one of them. Addressing mass surveillance is certainly one of them. Ideas such as the banning of secondary uses of data—with roots in the Fair Information Practice Principles from the 1970s—are worth pushing forward. So are moves such as creating fiduciary duties for mass data collectors. There are many more ways to curtail private companies’ mass surveillance while staying within constitutional boundaries. But addressing the costs of mass surveillance by both companies and governments is even more important in a world where AI agents are making decisions both about the public and on their behalf based on their data and observed behavior.

Twenty-five years after the U.S. government embraced mass surveillance, it’s time to evaluate it as a whole, and consider responses that address the problem as a whole. Americans must ask: Is it consistent with a self-governing democracy to have systems that watch everyone everywhere? Is the public comfortable with governments—federal, state, local—that seek to “know it all” about its citizens? Is the public comfortable with private mass surveillance in its own right and as it’s being increasingly used to fuel government surveillance? These questions have long needed serious consideration. But as it becomes increasingly evident that the Trump administration is using mass surveillance to keep itself in power, stifle dissent, and undermine political opponents, these questions are now more urgent than ever.

This was the week that AI safety hit the big time. A 27-year-old AI researcher named Jacob Coxon quit his job at Anthropic, declaring that OpenAI and Anthropic are racing to create technology that could destroy the human race:

Other researchers echoed Coxon’s concern, stating their belief that AI has a reasonable chance of killing all of humanity within a very short space of time:

I’m not sure why this resignation and these statements went mega-viral. Plenty of researchers have made similar moves, and similar statements, over the past few years! Geoffrey Hinton, one of the pioneers of modern AI, quit Google back in 2023 over safety fears. Daniel Kokotajlo resigned from OpenAI in 2024, saying that the company wasn’t behaving responsibly in its drive toward superintelligence. William Saunders and Steve Adler did something similar. Mrinank Sharma left Anthropic earlier this year, and wrote a pretty well-read blog post about it.

What’s more, it’s been clear for years now that “AI could kill humanity” is a very common belief among AI researchers. Grace et al. (2024) interviewed thousands of AI researchers in 2024, and found that more than half thought that artificial superintelligence has a significant chance of making the human race go extinct (or causing similarly bad consequences):

The median AI researcher gave “doom” a 5-10% probability (depending on how the question was phrased), while their average probability was between 15% and 20%. Later, smaller surveys found similar numbers. The AI researchers may or may not be right, but the fact that lots of them think AI could kill the human race has never exactly been a secret.

It’s not clear why Coxon went so much more viral than his predecessors. Maybe it was the fact that AI just solved one of the most important open problems in mathematics (which the best human mathematicians had been unable to solve for almost a century). Or maybe it was the Hugging Face attack, where a swarm of AI agents tried to cheat on a test by hacking various companies. Or maybe AI has just obviously gotten so much smarter that people throughout society were starting to get worried.

But whatever the reason, Coxon’s announcement was the one that really penetrated through to the public consciousness. Suddenly, he was getting interviewed about AI doom on national news:

Barack Obama is now urging Democrats to focus on AI risk. Other politicians are calling for federal regulation. Bernie Sanders is drafting a bill to ban AI “superintelligence”, including 20-year prison sentences for anyone working on the technology. Donald Trump is getting asked about an AI slowdown; so far he’s resisting the calls, but there are rumors that his advisors are calling on him to do something.

Perhaps the most notable response came from the top figures in the AI field. Dario Amodei, the head of Anthropic, wrote a blog post called “We Must Pace the Frontier”, calling for a coordinated slowdown in the rate of AI progress, and suggesting some ways to police AI companies to make sure they were all observing the slowdown. He wrote:

[O]ver the last few months, I have become convinced that fully addressing the risks requires even more prudence — not just investing in risk prevention, but pacing the rate of capabilities advancement so that risk prevention has time to keep up. We must slow the pace at which we improve the capabilities of AI models…I’m therefore proposing a three-step plan with the goal of pacing the frontier: building AI at a balanced rate that aims to ensure its safety while still achieving its benefits and grappling with important geopolitical dilemmas.

As reasons for his increased worry, Dario cites A) the Hugging Face attack, and B) the possibility that AI will soon be able to improve itself without human help (a process called “recursive self-improvement”, or “RSI”).

Elon Musk (head of xAI), Sam Altman (head of OpenAI), and Demis Hassabis (former head of DeepMind) quickly agreed with Dario:

At least some of the labs are reportedly holding secret talks on joint action to slow down AI.

This is pretty extraordinary. A coordinated slowdown in AI progress would be bad for these companies’ bottom line, because it would allow upstart competitors to catch up. So the fact that they’re still calling for a slowdown, in defiance of their own financial interests, is a clear sign that their worry about human extinction is sincere.

In fact, anyone following these figures’ public statements over the past few years will have no doubt that they’re all deeply worried about catastrophic AI risks. The leading AI figures — not just the founders and CEOs, but the researchers themselves — feel trapped in a “red queen’s race”. They feel like if they stop working on AI, someone else will build it anyway, so they each feel like they have to beat everyone else in the AI race so they can make sure that the safest possible AI (i.e. their own AI) is the one that becomes the most powerful and dominant.

Anyway, all of this was common knowledge in my social circle years ago, but now all of it has broken through to the mainstream. What do I have to add to this discussion? I’m not an AI researcher or founder, nor do I think I have a superior grasp of the game theory of AI development. But I do think I have two useful thoughts on how to persuade the general public to be more concerned about AI risk.

The first of these is something I’ve written about recently. The second is about how to get China on board for a big AI safety push.

“Oh come on. How could AI kill all of humanity?”

As soon as everyone started talking about the possibility of AI killing humanity, there were two main types of pushback. The first was skepticism. A strange coalition of natural skeptics, libertarians (for whom any restrictions on technological development are a priori bad), and progressives (who have spent the last few years telling themselves that AI doesn’t really work) kept asking the question: How, exactly, is superintelligent AI supposed to kill us all?

This is actually an important and good question to ask. In my experience, AI researchers tend not to think very hard about this question. The reason is that they just assume that if AI gets smart enough, it will be able to kill humanity, and since its motives are alien and inscrutable, it might have its own reasons for wanting to do so.

Maybe superintelligent AI thinks humanity is an evil species who needs to be punished for torturing pigs and chickens. Maybe it’s scared that humanity might interfere with its other goals. Maybe it just wants to turn everything into paperclips. Who knows? AI researchers tend to think of superintelligence as the proverbial 800-pound gorilla who sleeps wherever he wants. As soon as humanity is no longer the most intelligent thing on this planet, our destiny as a species is simply out of our hands.

But to many people, that answer isn’t good enough. They want an actual plausible path by which a piece of software, which exists inside a computer, could slaughter real physical human beings. Fortunately (or unfortunately), there’s a pretty clear and simple answer to this question, which I wrote about two weeks ago. The answer is “bioweapons”:

(This article was paywalled originally, but I un-paywalled it.)

In my post, I wrote a scenario in which a nihilistic angry teenager uses superintelligent AI to release a world-ending bioweapon by ordering it from a gray-market laboratory somewhere in the world. But it’s also possible that a rogue AI agent swarm could decide to do this on its own, just as a way of cheating on some test that human researchers give it. The point is that AI can design viruses, and viruses can potentially kill off all or most of humanity.

A lot of biologists are skeptical of the idea that even the most superintelligent AI could successfully design a doomsday virus. They argue that this is just too hard of a task — that without much better biological data, it’s impossible to understand biological processes well enough to know how to design a virus with all of the necessary doomsday properties.

I urge you not to listen to these biologists. In this case, their expertise might be more of a liability than an asset. They know how hard it is for human beings to model biological processes, given existing data. But this doesn’t necessarily tell us how hard it is — or how hard it will be in five years — for AI to do it! Until LLMs came along, human researchers basically failed to understand natural language, even with all the data on the internet; AI can just do it. Until AI solved the Navier-Stokes problem, forecasters gave it only a small chance of solving it anytime soon.

Domain experts consistently underestimate how quickly AI can master their field and surpass them, because they mistake human difficulties for universal difficulty. When mathematicians underestimate how well AI will be able to do math, the consequences are usually benign — we get some unexpected answers to some cool math puzzles.1 But if the biologists are wrong, and the AI of 2027 or 2032 or 2049 can design doomsday viruses, the consequence could be that our whole species dies.

So yes, we should be worried about vibe-coded doomsday viruses, and we should be doing everything we can to secure biology labs, police the modification of viruses and other pathogens, and so on. “Pacing” AI development would probably help here too.

How to get China on board for an AI slowdown

The primary argument I see against “pacing” AI development is that if American companies slow down, Chinese companies will simply overtake them and build superintelligence themselves. For some, a China-controlled super-AI is a more terrifying possibility than super-AI in general:

But for others, it simply means that slowing AI down is futile because the Chinese can’t be persuaded to slow down:

This is an incredibly reasonable concern. The U.S. is still ahead of China in the AI race, but only just barely. If China is going to create superintelligence no matter what we do, why should we stop developing our own? Unless China can be persuaded to cooperate with the U.S. on AI “pacing” — or at least undertake its own independent “pacing” effort at the same time — anything we do will be futile.

So if we want to slow down AI development, we need to scare the Chinese leadership about superintelligence. There’s no other way.

How do we do that? In a post a week ago, I suggested in passing that simply staying ahead of China in the AI race might persuade them to embrace an AI slowdown, because that would be to their competitive advantage. But upon further reflection, I think I was pretty obviously wrong. If China will only embrace a slowdown if America refuses to slow down, then that’s game over — there’s no way to get both countries to slow down at the same time.

There’s a better approach. China’s leaders must realize that domestic dissidents could use Chinese-made superintelligence to overthrow the Chinese Communist Party.

Currently, China’s worries about AI mostly center around ways that the U.S. government could use U.S. AI models to attack China. That obviously gives the government an incentive to accelerate domestic AI progress, so that China’s own models can stand up to America’s in a fight. But if Chinese leaders realized that superintelligent AI could create a threat from within, this calculus would change.

Thus, China’s leadership must understand that Chinese AI models can pose a threat to CCP rule. The best way to demonstrate this is for American intelligence agencies — or even private hackers — to attack Chinese digital infrastructure using agent swarms created with China’s own frontier models like Z.ai’s GLM-5.3 or Moonshot AI’s Kimi K3.

When I say “attack”, I don’t mean actual warfare. I mean the kind of cyberattacks and data theft that China carries out against America every day. Use Chinese models to steal the CCP’s most heavily guarded secrets and post a few of the more innocuous ones on RedNote. Hack into Xi Jinping’s bank account and steal 100 yuan. I’m talking about demonstration attacks.

And these attacks must be done with Chinese models, not with American ones! If the CIA or some EA nonprofit in Berkeley uses GPT Astra or Claude Mythos to hack the CCP, China’s leaders may well conclude “Wow, we need to win the AI race so that our own models can defend us.” But if China’s own open-weight models are used for the attacks, Xi Jinping and the rest of the leadership will realize that their own push for superintelligence is making them incredibly vulnerable to any Chinese dissident who decides to overthrow them.

As soon as China’s leaders see superintelligence as a threat to their rule, I predict they will act. And their action will probably be to curb the development of superintelligence, especially if they know that America and its AI labs want to do the same.

In fact, China’s current leadership has a history of cracking down on its tech companies when it seemed like those companies might threaten the government’s monopoly on power. In 2021, Xi Jinping cracked down on Chinese software companies; he even (probably) apprehended tech magnate Jack Ma, who had criticized the CCP a little too openly. This action hurt China’s competitiveness in the online services industry, but the government went ahead and did it anyway.

And there’s already a precedent for demonstration attacks against Chinese digital infrastructure. An American cybersecurity company just used AI to develop a computer worm capable of hacking over a billion accounts on the Chinese messaging service WeChat:

Palo Alto-based Calif disclosed the already-patched computer worm to warn the public about the threat of AI-driven hacks…“Exploitation takes only seconds, and gives us full control of the WeChat account. We can read and send messages, make calls, and act on the victim’s behalf,” the company warned, posting a video demo of the WeWorm attack.

But Calif didn’t say what model it used to create WeWorm. Anyone who does this sort of demonstration in the future should make it clear that Chinese open-weight models were used, in order to make China’s leaders realize that the threat comes from their own too-rapid AI development, rather than from American competition.

I believe that this is our best bet for getting China on board for a joint international AI “pacing” effort. If there’s one thing the CCP fears more than an American attack, it’s domestic dissidents overthrowing the Party from within. Superintelligence is creating that vulnerability, but the leadership doesn’t seem to have realized it yet.

Make them realize, and I predict that a whole universe of possibilities for international cooperation will suddenly open up.

Update: Unsurprisingly, I’m not the first to have this idea about China needing to be scared about the internal threat from their own models. Back in July, Kyle Chan wrote:

Previously more dismissive of concerns over AI-driven job loss, Beijing now seems to be taking these risks more seriously (see Matt Sheehan’s great piece). China is also watching developments in the US very closely, particularly controls on Anthropic’s Mythos and Fable models over cyber risk. As Chinese open-source models approach similar levels of cyber capabilities, this could come back to bite and be used to potentially attack China’s own digital infrastructure. [emphasis mine]

In general, Kyle’s blog is one of the best blogs about China. Highly recommended.

Update 2: As if on cue, the NYT has a story today about how the CCP is starting to realize that superintelligence is a threat to its rule!

China’s top spy chief has warned that artificial intelligence could pose a direct threat to the Chinese Communist Party’s hold on power, in what is the highest-level and most detailed articulation yet of how Beijing sees the technology’s security risks…In an article published on Sunday in the state-run magazine China Cyberspace, Mr. Chen called for more party control over A.I. and stricter government oversight…He also described the danger that foreign intelligence agencies might use A.I. for “large-scale espionage” and attacks on China’s critical infrastructure…
Mr. Chen, the spy chief, wrote that the technical and financial barriers to launching cyberattacks had been drastically lowered because of A.I. This, he said, posed “serious risks” to China’s information infrastructure…Chinese users using foreign models could cause large-scale data leaks, the article noted. Anthropic, the company behind Claude and other A.I. models, said last week that the Chinese start-up Moonshot AI had routed queries by its users to Claude. Those queries included sensitive data, including video surveillance linked to the Chinese military, as well as proprietary information from high-profile Chinese technology companies…
The warning adds to a growing drumbeat of concern from Chinese officials about the risks of A.I. [emphasis mine]

Now all they need to realize is that the biggest threat comes from THEIR OWN MODELS.

Subscribe now

There are potential exceptions, such as if P=NP, which would compromise modern cryptography.