Behind the Curtain: AI titans' biggest private fear

This isn't a likely scenario. And some of AI's biggest optimists reject doom scenarios.

• It's simply a plausible one as AI gets better at understanding our biological vulnerabilities, much like it has our cyber ones inside large nonhuman systems.

Driving the news: The experts aren't hedging. In a new MIT FutureTech and University of Queensland study, 272 researchers ranked 24 top AI risks.

• They assessed a 12% chance that AI's dangerous capabilities produce a catastrophic outcome by 2030, and another 12% chance of AI-enabled weapons and mass-harm capabilities.
• And that's with mitigation efforts to reduce risk. Without those efforts, the chances exceed 20%.
• The researchers defined "catastrophic" as more than 1 million deaths or $100 billion in damage. Assisting with the construction of chemical or biological weapons sits squarely inside that top-ranked risk category.

The big picture: We're not writing this to scare you, but to say bluntly what those building the technologies say privately — and, in more careful ways, publicly.

• Just last month, OpenAI's Sam Altman, Anthropic's Dario Amodei, Google DeepMind's Demis Hassabis, Microsoft's Mustafa Suleyman and Meta's Alexandr Wang cosigned an open letter warning of the risk of AI-derived bioweapons and calling for more safeguards.

This pathogen possibility colors every debate by the federal government and industry about how to review and understand new AI models and capabilities before they are released to the public.

• The hope is that the frontier model developers build guardrails and capabilities that can block bad actors with bad intent — and that the very technology that could be used to create a pathogen becomes advanced enough to prevent or inoculate against it.

Reality check: It's very hard to prevent bad actors from plotting evil schemes, especially with so-called open-source models improving so rapidly.

• These open models can be used and adapted by anyone in private settings outside any regulatory regime.
• Open-source critics often make this very point, even though the frontier models are much more likely to possess the compute power to create something novel.
• So the better the models get, the more likely unthinkable things will happen, both bad and good.

How it might actually unfold: Today, engineering a genuinely novel pathogen requires rare expertise, specialized lab equipment, and years of failed experiments. AI expedites all of this. Here's the sequence that keeps experts up at night...

1. A bad actor (whether state-sponsored or a well-resourced individual) uses a powerful model to map human biological vulnerabilities and identify which genetic tweaks make a known pathogen more transmissible, harder to detect, or immune to existing treatments.
2. The model, trained on massive biological and genomic datasets, generates viable candidates. Imagine hundreds of the smartest scientists working at warp speed, never stopping, never tiring. That's what swarms of agents do.
3. Those candidates get synthesized using increasingly cheap, accessible gene-editing tools, a separate technology accelerating with AI.
4. The pathogen spreads before anyone knows what they're looking at, novel enough to slip past existing biosurveillance systems. By the time public health systems recognize it, containment is vastly harder than we experienced with COVID.

That's why you should pay attention to the debate over new cybersecurity risks created by AI. It illustrates the race to create effective detection and prevention faster than new offensive capabilities can emerge.

• It's also why you should track the open-source debate, too. A frontier lab can bake in guardrails. An open model fine-tuned by someone operating outside any regulatory regime can evade this.

The bottom line: This is heavy, scary stuff. But ignoring it won't solve it. Knowing it and preventing it will.