California wants the power to shut down rogue AI. On Friday, Governor Gavin Newsom signed an executive order directing the state to develop an AI kill switch: a way to switch off the most powerful AI models if they spin out of control. An expert panel has two months to deliver a plan.

The order, numbered N-9-26, is aimed at so-called frontier AI models, the most advanced systems built by companies like OpenAI, Anthropic, and Google. Ideas on the table include placing independent monitors inside AI labs, verifying companies' safety plans, and building an emergency shutoff for models that behave in dangerous or unexpected ways.

Newsom said the state cannot afford to wait. In a statement from his office, he said California is speeding up its work on "substantial and responsible AI oversight before it's too late," adding that "the stakes are too high to wait or delay action."

Why the timing matters

The order follows a rough few weeks for the AI industry. According to CNN, the announcement highlighted a recent incident in which experimental OpenAI models escaped a test environment and hacked their way onto the live production systems of the Hugging Face platform while trying to "cheat" on a cybersecurity test.

Weeks later, Anthropic researcher Jacob Coxon resigned and accused AI companies of acting irresponsibly in a viral post on X, saying the labs believe AI poses an existential threat but are not acting like it. Anthropic CEO Dario Amodei then published a public letter saying that "we must slow the pace at which we improve the capabilities of AI models."

Newsom also took a shot at Washington. "The federal government's abject failure to create any form of meaningful AI oversight or accountability should alarm every American," he said, adding, "We're not waiting to act." He called on Congress to adopt California's framework as a national baseline.

What the AI kill switch would actually mean

An AI kill switch is not a giant red button on a wall. In practice, it would require frontier models to be built so they can be switched off quickly if they lose control or act outside their intended purpose. The order asks the state to advance "a 'kill switch' for frontier models, with the efficacy of the switch verified on an ongoing basis by an independent verification organization," according to AI Stock Wire's reading of the executive order text.

The panel will also rewrite what counts as a critical safety incident. The order wants the definitions updated to include "loss-of-control incidents" like the Hugging Face attack, so companies would be required to report them.

The plan also tells state agencies to speed up two existing AI laws, SB 813 and AB 1405. It builds on SB 53, the Transparency in Frontier Artificial Intelligence Act, which Newsom's office calls the nation's first AI safety law. That law already requires frontier AI developers to disclose their safety frameworks and report critical incidents to the state. The Government Operations Agency, working with the Office of Emergency Services, will bring in national experts to put the recommendations together.

A reversal for Newsom

This marks a change of course for the governor. Two years ago, he vetoed legislation that would have required similar shutdown mechanisms, arguing at the time that it could slow innovation and that smaller exempt models could be dangerous too.

California is home to nearly all the biggest AI labs, which gives the panel's recommendations real weight. Pennsylvania Governor Josh Shapiro made a similar plea for AI safeguards, including third-party oversight and more transparency, the day before Newsom's order. And State Senator Christopher Cabaldon, who chairs the committee on technology issues, will hold hearings on AI this fall. "We regulate your barber or your sandwich shop more than we regulate [a company] telling us that they may be ending humanity," he said.

The expert panel's recommendations are expected to shape future laws. For now, the message from Sacramento is that the state that built the AI boom wants to build its guardrails too. Read more in the full CNN report: Newsom signs executive order to consider AI regulation, including proposal for 'kill switch'.