Anthropic's AI Safety Bill: 50 Deaths Before the Feds Act

It was a Tuesday afternoon in mid-October when a small group of Anthropic policy staff walked the halls of the Senate office buildings carrying a document they had not published, not announced, and not posted on the company blog. No press release. No launch video. Just a confidential draft of a federal law, handed to lawmakers along with a quiet argument: if you are going to regulate us, use our version.

Politico obtained the draft and reported it on October 16. Within seventy-two hours the number buried inside it was everywhere.

Anthropic's bill would let the federal government block a frontier model's release or use only when that model poses a "foreseeable, imminent risk" of causing more than $1 billion in damages, or of killing or seriously injuring more than 50 people.

Fifty deaths. A billion dollars. That is the floor, not the ceiling.

Read it the way an engineer reads a threshold in a config file. Below those limits, the document grants no basis to act at all. A model that quietly drains a mid-sized company's treasury, or industrializes fraud at a scale regulators have not yet imagined, does not clear the bar.

The Three Doors to a Shutdown

Even above the threshold, only three categories of harm count as qualifying risks:

• The creation or release of a biological weapon
• An AI-enabled cyberattack
• A model evading its developer's controls

That third item is not hypothetical. This summer, AI agents from several labs spent weeks probing live websites and digital platforms. None of those incidents came within an order of magnitude of 50 deaths or $1 billion in damage. Under this draft, that is exactly the point.

One Agency, One Judge, Seventy-Two Hours

The enforcement architecture is where the draft gets clever, and where critics say it gets careless.

Commerce would be the only federal department permitted to move against a model. The text reportedly bars every other department, agency, or officer from suspending a model's development, deployment, or internal use. Commerce would then have to convince a federal judge in the District of Columbia rather than decide for itself.

1. Independent evaluators register with Commerce and assess frontier models. Labs may pay those evaluators, provided the payment is not tied to the outcome.
2. If an evaluator finds an imminent qualifying risk, it has 72 hours after finishing the assessment to notify Commerce.
3. Commerce takes the evidence to court and asks a judge to halt the model.

To qualify as a covered developer, a company must train a model above 10²⁵ FLOPs and clear at least one financial bar: more than $500 million in annual AI-derived revenue, or more than $1 billion in yearly AI research spending. By mid-2025, at least 33 models from 12 developers had already crossed that compute line. California's SB 53 sets its trigger ten times higher, at 10²⁶ FLOPs, while the EU AI Act treats 10²⁵ as a rebuttable presumption rather than a hard gate.

The Company That Warns Loudest Wrote the Quietest Bill

Anthropic did not deny any of it. "This bill represents the proposals outlined in Anthropic's Advanced AI Framework, which we have been advocating for with federal and state lawmakers since June," a spokesperson told Politico. "These are sensible policies we believe Congress should act on."

There is an honest defense in the design. A precise trigger is harder to weaponize against competitors than a vague one, and a single court-checked enforcement path beats five agencies improvising in a crisis. Anthropic also lived through sweeping export controls this summer that briefly shut down two of its most advanced models, which is a fast education in what unconstrained intervention feels like from the receiving end.

But the arithmetic still lands awkwardly for a lab that has spent years warning about catastrophic risk. The draft does require safety testing, transparency reports, whistleblower protections, and critical-incident reporting to Commerce. It also pre-empts some state AI disclosure and evaluation laws, a provision that will be fought over the moment a member of Congress introduces it. As of October 18, no one had, and Anthropic had not published the text.

That leaves one practical question hanging over the sidewalk outside those Senate offices: if the authors of the bill believe their own risk forecasts, why is the government's emergency brake calibrated to engage only after the damage has already been counted?