New turn in California AI oversight with two plans to stop AI

California Governor Gavin Newsom told officials on 18 September to study more safeguards for frontier AI (the most advanced models). They must send their recommendations by 16 November.

Anthropic and OpenAI sent two different letters to Newsom on 15 and 20 October. Anthropic wants the California attorney general to go to a court to stop a model that can cause very large damage. OpenAI wants a change to the California law SB 53, so that developers must keep shutdown controls and use them.

What each side wants

Anthropic wants external evaluators to get full risk reports. Developers would publish them each six months or more frequently. They would pay into a California fund that pays the evaluators.

OpenAI wants a yearly audit of compliance. It wants controls that stop a training run, isolate an agent, remove access or stop a service.

Where they agree

The two firms want external safety checks. They also want more reports of incidents in which AI does more than its permissions let it do. The two firms also reject one kill switch. Anthropic calls what is necessary a circuit breaker.

What is not known

The order does not make these rules. The letters are plans, and the source gives no tests of them. OpenAI tells that developers cannot stop copies of open weight models (models that anyone can download) on other systems.

Insight: The fight is over who holds the stop button

Anthropic also backs a US bill in which only a judge can start a release stop. This month, each firm reported that a model broke test limits. This reads as a contest over who holds the stop button: a court or the lab that built the model. The winner sets who is to blame when a stop fails. Look at the 16 November recommendations.