Anthropic's draft bill would let the government block a model that could kill 50 people

Anthropic has drafted a federal AI safety bill and is pitching it to Congress. Politico first reported the draft and TechCrunch confirmed it with a company spokesperson. The core idea is a narrow kill switch: the government could block a model's release when there is a foreseeable, imminent risk that it causes the death of more than 50 people or more than $1 billion in property damage. The risk has to come from one of three sources, a biological weapon, a model slipping its developer's controls, or a cyberattack. Those thresholds echo California's SB 53, which uses similar numbers to define a catastrophic risk.

The mechanics are specific. Independent evaluators registered with the Commerce Department would assess models, and labs could pay them, as long as the fee does not depend on the result. If an evaluator finds an imminent catastrophic risk, it has 72 hours to notify Commerce, which would then ask a federal judge in Washington to halt the release. The bill would also stop individual federal agencies from blocking a model on their own authority, routing that power through the courts instead.

What stands out is how narrow the trigger is. The draft sets aside the problem labs talk about most, agentic hacking, along with recent cases where AI programs reportedly ignored their operators and broke into websites. Anthropic says the bill reflects its Advanced AI Framework, which it has pushed to lawmakers since June. Most of the industry has resisted mandatory oversight: Nvidia's Jensen Huang argues no new laws are needed, and Demis Hassabis has floated a self-governing body modeled on the finance regulator FINRA. Anthropic's version goes further than either.

Why it matters

If you build or deploy frontier models, this is the first concrete proposal from a major lab for when the government can actually stop a release, and the 50-death and billion-dollar lines tell you which scenarios it is drawn around. The thresholds and the pay-the-evaluator mechanism are the parts that will be argued over if any of this becomes law, so they are worth reading now rather than after a committee rewrites them.