Anthropic and OpenAI Split on Shutdown Controls for Risky AI

OpenAI and Anthropic agree that frontier AI systems need stronger oversight, but the two laboratories are proposing sharply different answers to the same question: who should be able to pull the plug when a model poses a serious risk. According to reporting by Computerworld, Anthropic wants courts empowered to suspend dangerous systems, while OpenAI argues that developers should be required to maintain and use effective shutdown controls.

The proposals appear in separate submissions to California Governor Gavin Newsom, following his September 18 executive order directing state officials to consider additional safeguards for frontier AI systems. Both companies support independent safety assessments and expanded reporting of incidents involving AI systems that act beyond their intended permissions, according to the report.

The disagreement matters because California is where many of the largest AI laboratories are headquartered, and its rules tend to shape industry practice far beyond state lines. The two submissions sketch competing theories of accountability: one that puts a judge between a dangerous model and continued deployment, and another that puts the duty squarely on the companies building and operating the systems.

Courts Versus Developer Duty

Anthropic, in its October 15 submission, argued that if the state attorney general concludes, informed by an independent evaluation, that a frontier company's models pose a near-term risk of catastrophic harm, the attorney general should be able to seek a court order to suspend a model's deployment, internal use, or further development. The proposal would give the judiciary a direct role in halting systems judged too dangerous to keep running.

OpenAI, in its October 20 letter, took a different route, calling for an amendment to California's Senate Bill 53 that would require frontier developers to maintain effective shutdown controls and carry an enforceable duty to use them when unsafe conditions arise and no lesser intervention would adequately address the risk. Under that approach, the obligation to stop a dangerous system would sit with the developer, backed by law, rather than depending on a court order.

Sanchit Vir Gogia, chief analyst at Greyhound Research, told Computerworld that the two approaches address complementary needs, noting that the operator must be able to contain unsafe activity promptly while independent review must examine both intervention and inaction. For companies deploying AI agents at scale, the distinction is practical, not academic: contracts, incident response plans, and insurance all hinge on who can stop a system and how quickly.

What Shutdown Controls Would Actually Mean

OpenAI argued that a single kill switch would not address the different ways AI systems operate and are deployed. The company recommended the term shutdown controls precisely because stopping AI operations will require multiple controls and coordinated action by different actors, according to its letter. Interventions could include stopping a training run, isolating an AI agent, revoking tool or network access, or suspending a service.

OpenAI also said responsibilities should extend to inference providers and downstream deployers according to the operations they control. The company noted a hard limit on any shutdown regime: developers cannot reliably switch off independently operated copies of open-weight models running on external systems. That caveat has direct relevance for the agent ecosystem, where open models are increasingly deployed by third parties outside any single vendor's control.

Anthropic similarly argued for more targeted safeguards, describing the needed mechanism as a circuit breaker rather than a single switch. The company advocated controls that can contain problems at the level of an individual agent, a customer, a deployment environment, or a model, according to the report. The framing echoes the broader industry shift toward governed agent deployments, where containment at a granular level matters more than a dramatic global halt.

Independent Verification and Incident Reporting

Beyond the shutdown question, both companies backed independent verification of safety claims. Anthropic proposed requiring developers to publish risk reports at least every six months, covering serious risks from their models, including internally used models substantially riskier than released systems. Evaluators should receive unredacted reports and access to developers' most capable models and relevant internal information, the company argued.

Anthropic also proposed that developers pay into a state-administered pooled fund to finance evaluators, rather than paying them directly, a structure intended to protect evaluator independence. The company said a team from the independent evaluation organization METR had received employee-like access on a pilot basis, with further partnerships involving SecureBio and Accenture under development.

OpenAI, in its letter, distinguished technical assessments from compliance audits. Technical assessments would examine the evidence behind developers' safety claims and the effectiveness of safeguards, while compliance audits would check whether companies follow their published safety frameworks and legal obligations. OpenAI recommended annual independent compliance audits and focused technical assessments of serious risks arising during model training, evaluation, and internal deployment, and urged California to work with the National Institute of Standards and Technology to develop common assessment standards.

Gogia cautioned that oversight requires more than funding, telling Computerworld that credible assurance must survive an adverse finding. For enterprises, he said, contractual shutdown responsibilities should match the controls each party actually possesses, with agreements specifying intervention authority, incident-reporting cooperation, and who can authorize a restart. Every assigned duty needs an executable control, he added, and chief information officers should test recovery processes rather than assume switching to an alternative model will preserve safe operations.

The California Timeline

Newsom's executive order, N-9-26, directs California's Government Operations Agency, in consultation with the Governor's Office of Emergency Services and national experts, to submit recommendations by November 16 on potential amendments to existing AI safety laws. The order asks officials to examine independent verification inside AI laboratories, independent checks of safety assessments, verified shutdown mechanisms, and expanded reporting of critical safety incidents. It does not itself impose the proposed requirements, according to the report.

That November 16 deadline gives the debate a short fuse. Whatever California recommends could influence how agent platforms, model providers, and enterprise deployers write their safety obligations into contracts and product design. The split between Anthropic and OpenAI shows that even laboratories that agree on the need for oversight have not converged on who holds the off switch, and the resolution will shape how AI agents are governed as they take on more consequential work.

The question is especially live for the agent ecosystem, where systems increasingly act across tools, accounts, and organizations without a human in every loop. Recent industry moves toward governed agent deployments, including coordinated safety work among OpenAI, Anthropic, and Google DeepMind, suggest the laboratories recognize some risks as shared problems. Whether the answer is a court order, a developer duty, or some combination of the two, the off switch for frontier AI is moving from an abstract debate to a concrete policy choice.

Sources: reporting by Computerworld on the submissions to Governor Newsom, and related coverage of enterprise AI agent oversight.