Anthropic pitches AI safety bill that may block a model if 50+ people could be killed

Anthropic Proposes AI Safety Bill with Concrete Risk Thresholds

In a significant move to shape its own regulatory future, AI safety and research company Anthropic has circulated a draft bill among U.S. lawmakers. The proposed legislation outlines specific conditions under which a new AI model could be blocked from public release, defining a “catastrophic risk” as a foreseeable event that could lead to more than 50 deaths or over a billion dollars in property damage. This effort marks a tangible step by a major AI lab to translate abstract safety principles into concrete legislative language, directly influencing the ongoing debate on how to govern powerful artificial intelligence systems.

A Framework of Independent Evaluation and Judicial Oversight

The bill details a specific operational framework for pre-deployment evaluation. It suggests a system of “independent evaluators” who would be registered with the U.S. Commerce Department to assess AI models for potential catastrophic risks stemming from bioweapons, cyberattacks, or a model evading developer controls. While AI labs could pay these evaluators, the proposal states that payment could not be contingent on the outcome. If an evaluator identifies an imminent risk, they would have 72 hours to notify the Commerce Department, which would then need to seek a judicial order from a D.C. judge to halt the model's release, explicitly preventing any federal agency from acting unilaterally.

• Risk Definition: A “foreseeable, imminent risk” causing 50+ deaths or $1B+ in property damage.
• Risk Sources: Creation of biological weapons, model evasion of controls, or cyberattacks.
• Oversight Body: Independent evaluators registered with the U.S. Commerce Department.
• Enforcement: Commerce Department must obtain a judicial order to block a model's release.

Industry Impact and Proactive Regulation

Anthropic’s proposal goes further than the voluntary commitments and self-governance suggestions offered by many of its peers. It presents a stark contrast to the position of companies like Nvidia, whose CEO has argued against the need for new AI laws. By proactively drafting legislation, Anthropic is attempting to steer the regulatory conversation, potentially preempting more restrictive, government-led mandates. This strategy seeks to establish a regulatory floor that, while introducing new compliance hurdles, keeps significant procedural control within an ecosystem of industry-funded evaluators and judicial, rather than executive, oversight.

Anthropic's draft legislation is a calculated move to front-run government regulation, proposing a framework that establishes clear risk thresholds while retaining significant industry influence over the evaluation process.