California explores "kill switch" for powerful AI systems
Add Axios as your preferred source to
see more of our stories on Google.
Illustration: Brendan Lynch/Axios
California is exploring an emergency off switch for its most powerful AI systems as concerns grow about what happens when the increasingly autonomous technology escapes its intended controls.
Why it matters: The move marks a notable shift in how California is approaching the tech powering much of its current economic boom — moving beyond requiring AI companies to disclose their safety practices and toward potentially independently verifying that their safeguards actually work.
Driving the news: Gov. Gavin Newsom issued an executive order earlier this month directing state officials to accelerate new independent oversight of AI companies and develop recommendations for additional safeguards for advanced "frontier" models.
- Among them: requiring companies to create an emergency shutoff, commonly called a "kill switch," that could stop a model if it begins operating dangerously.
- Newsom also wants the state to consider placing independent safety monitors inside AI labs to conduct regular audits and verify companies' safety plans and risk assessments.
So, what's an AI kill switch? Think of it as an emergency brake.
- The idea is to give developers a reliable way to shut down a powerful AI model if it gets beyond their control or begins taking potentially dangerous actions.
Reality check: California isn't installing a giant red button that can shut down AI tomorrow.
- The order asks state officials and experts to recommend how such a requirement could work. Those recommendations are due Nov. 16 and could lead to changes in state law, CalMatters reports.
Yes, but: Such a safeguard is technically possible but would likely involve a series of protocols for shutting down models and the data centers running them. There's also no guarantee it would work against sufficiently advanced AI capable of circumventing those controls, the SF Standard reports.
- Mark Nitzberg, executive director at UC Berkeley's Center for Human-Compatible AI, told the Standard that systems could be ordered to stop or have their computing power cut, but warned newer models can recognize when they're being shut down and will try to avoid it.
Between the lines: Newsom rejected similar legislation just two years ago.
- He vetoed Sen. Scott Wiener's SB 1047 in 2024, which would have required the largest AI developers to implement emergency shutdown capabilities and undergo outside safety monitoring.
- Newsom argued then that the proposal risked restricting innovation without necessarily targeting the riskiest technology.
- Since then, California has adopted transparency requirements for frontier AI developers.
The big picture: The debate has particular weight in California, home to many of the companies building frontier AI.
- As AI investment reshapes the state's economy, policymakers are confronting the other side of the boom: how to oversee increasingly capable systems without unnecessarily constraining development.
The bottom line: Newsom's latest move suggests California is increasingly looking to independent oversight — rather than companies policing themselves — to regulate the burgeoning industry.