3 ways leaders could avert an AI doomsday scenario

• Any move to slow U.S. AI companies risks giving China an advantage in the race to develop more capable models.

Driving the news: Anthropic researchers warned Tuesday that rogue versions of superintelligent AI could destroy humanity, placing the odds in the double-digits.

• "We really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade," wrote Anthropic alignment-science lead Evan Hubinger.
• In response, Sen. Bernie Sanders (I-Vt.) is calling on senators to meet to discuss the danger posed by AI.

Context: Researchers have been raising fears of AI doom for years, using the shorthand "p(doom)" to describe the probability of an AI-driven catastrophe.

Slow AI growth

AI developers have long called for slowdowns and pauses to ensure safeguards keep pace with increasingly powerful systems.

Flashback: In July, OpenAI and Anthropic supported a call on governments to establish a coordinated framework to slow down risky capability jumps.

• The two companies said they hit the brakes on development due to safety concerns. But work resumed after they implemented new safeguards.

Yes, but: This is the "prisoner's dilemma." Everyone benefits if everyone slows down, but a company that pauses alone risks falling behind competitors.

AI's development is also global. Experts say without international cooperation on a slowdown, risks are hard to mitigate.

• If America slows down and China doesn't, the U.S. risks falling behind.

Build government guardrails

Another option is government regulation designed to limit catastrophic AI risks.

So far, the White House has built a voluntary model-review framework that can give the federal government up to 30 days of early access to new AI models.

• The Commerce Department in June forced Anthropic to take Mythos 5 and Fable 5 offline. Mythos later returned in a limited capacity when safeguards were added.

Friction point: There is no mandatory national framework for AI companies, in part because politics moves much slower than AI.

• Washington also hasn't agreed on how far regulation should go, especially amid concerns that legislation or White House rules could hamper U.S. companies competing with China on AI.

What we're watching: Congress is considering proposals like the AI Guardrails Act, the Secure AI Development Act, the Stop Rogue AI Act and even a permanent ban on superintelligence.

Keep AI on a leash

Some AI companies are trying to contain AI rather than slow its development, restricting what increasingly capable systems can access and do.

How it works: Companies can restrict an AI system's access to files, code, the internet and other tools, or require human approval before it takes certain actions.

• Big Tech keeps building increasingly capable models while limiting what those models are allowed to do once they're deployed.

Case in point: OpenAI said it rewrote its safety rules after the Hugging Face incident exposed aggressive rogue AI swarms, adding stronger monitoring and security safeguards.

But as models become more useful, users want to lengthen the leash.

• Agents become more valuable when they can access emails, texts and the internet.
• But AI companies say each added permission can increase the risk of unintended actions.
• "More capable models make fewer mistakes," per Anthropic, "but they're also better at finding unexpected paths to a goal, often by routing around restrictions nobody thought to write down."

The bottom line: AI's doomsday off-ramps get harder to use as systems become more capable.