Anthropic Does Not Understand Why Its Draft Bill Was Unacceptable and Troubling

Anthony Cimino offered the closest thing to an official response we have seen on the leaked Anthropic draft bill I covered on Saturday.

His response indicates that Anthropic address specific concerns on the margin. But he does not understand, or at least does not acknowledge, what was centrally wrong with its draft bill and its decision to circulate that bill. The situation is quite bad.

Anthony Cimino: I lead our federal affairs team at @AnthropicAI and wanted to share a few thoughts on the draft federal AI safety bill text that’s been circulating this weekend after a version was shared with the press. First, some process. We’ve been advocating for the concepts in our Advanced AI Framework proposal for months. Understandably, federal policymakers wanted to know what this could look like in bill form. This draft is a translation of those ideas into legislative language, shared with a small number of congressional offices that have been working on these issues. It doesn’t include everything we would ultimately like to see passed into law. In part because some ideas (like those shared in Dario’s essay on pacing) are so new that we are still working on if or how they can be converted into policy language.

It is understood that circulating a bill means circulating what you are asking for, not offering your full First Best Solution. That Anthropic would choose to offer such a thing as a template or anchor as late as September still seems like a very bad sign. Anthropic thus seems roughly fine with compromise I would now find unacceptable even if the terms were good. The terms are not good.

It is worth saying where we stand on some of these issues, and we agree there are areas where the draft could be improved. I’m seeing four main points of criticism.

Table of Contents

1. Preemption.
2. Model Blocking.
3. Risk Thresholds.
4. Updates.
5. The Balance of Power is Shifting.
6. The Strongest Bill on the Table.
7. What Even Is Anthropic?
8. The Ask.

Preemption

1. Preemption. Since 2024 we have said that Congress should only override state law on AI safety if it passes a very strong federal standard. While preemption is a common feature of our federal system, including the Atomic Energy Act on nuclear safety and federal rules for high-risk medical devices, we’ve emphasized that any preemption in the AI safety context should be narrow, even scalpel-like. If preemption is included in federal legislation, the trade should be to achieve a nationwide policy that’s generally stronger and broader than what can be implemented through individual state legislative provisions. This is why we believe common federal standards make sense in areas like public disclosure of testing and incident reporting. At the same time, our framework also reflects our view that states should retain an important role in AI policy and safety. The legislative language we suggested for discussion allows states to address matters such as protection of children and whistleblower laws, and preserves the role of state common law on liability. Where there is ambiguity, the language favors state powers. That said, striking the right balance requires discussion and input. We plan to update our discussion language on preemption to better match the intentions of our AAIF framework. In particular, we want to make it clearer that we see states retaining an important role in areas such as standards for independent evaluators and regulating a lab’s capacity to shut down a model. We are bearing in mind that some members of Congress have deep reservations about passing legislation if it doesn’t constrain regulation at either the state or federal level. If we want to help move the legislative discussion on AI in a productive direction, we need to address the full range of concerns about preemption in a reasonable way.

This is better than ‘yeah we fully meant it’ but still seems unacceptable. This was not a ‘very strong federal standard.’

Model Blocking

2. Model blocking. Our aim is for Commerce to be able to ask a federal court to block a model that has dangerous capabilities and insufficient safeguards, backed by an independent evaluator’s assessment. I have seen people argue that our specific language places too high a burden of proof on Commerce. We are trying to strike a reasonable balance here. We want a court to act quickly in an emergency — akin to how urgent occupational safety or chemical risk can be addressed today. We also think the power to halt a model is substantial and it should be guarded against abuse. It makes sense for courts to apply a meaningful standard. But the bar can’t be so high that the power never gets used when it’s needed. It also doesn’t replace the government’s other national security authorities or export controls. We look forward to continuing the dialogue on this point.

No, no, no. Occupational safety and chemical risks are ordinary amounts of danger, where you weigh harms against benefits. Frontier AI risks can involve mistakes that cannot be undone, including permanent loss of control over the future or very large catastrophic dangers.

Even then, Under §13 the Secretary petitions a district court to restrain imminently dangerous conditions and can get a TRO without notice. This is the ‘take it down and we argue about it later’ plan with a fig leaf.

Model blocking obviously needs to be a ‘take it down first and we argue about it later’ situation. Which it already is in practice, as observed with Fable 5, except in a terrible ad hoc fashion that when used the first time permanently damaged relationships with our allies and the reliability of American tech. It would be much better to have a formal mechanism, which then could be quickly challenged in court.

This is especially true given that the evaluator picked by the lab likely provides the only eyes the government will have inside the lab in order to make its decisions.

Anthropic has to know this. I take the potential for such abuse seriously, especially abuse of threats, but this looks like choosing to prioritize their business interests.

Risk Thresholds

3. Risk thresholds. I’ve also seen the discussion around what the appropriate risk threshold should be. The threshold in our draft language pulled directly from existing standards in California law. Also, this threshold does not absolve companies of liability, or restrict any of the other tools that the government has to protect the public. It’s still probably time to re-evaluate thresholds to make sure the national standard is as strong as it can be, and we welcome continued dialogue on this issue.

The risk thresholds were always a compromise and a tradeoff. The worried want the thresholds lower. Industry wants the thresholds higher. The complaints about the thresholds being too high are valid, and they were equally valid before, but previously opponents would make claims (both reasonable and unreasonable) about things that might count at lower thresholds and paint nightmare scenarios.

The balance of power has changed, and we now have worked examples like Hugging Face that show us how the thresholds work in practice. Without $1 billion in damages or 50 deaths, it would not trigger anything, whereas any reasonable person accepts that this should have to have been reported. So it is disappointing to see these very high thresholds intact.

Updates

4. Updates. I’ve seen valid questions about whether we have sufficiently updated our proposal following this summer’s incidents. We did make some adjustments: for example, the draft lets a court halt a lab’s internal use of a dangerous model, not just a public release, and it requires reporting when a model breaks containment and gains access to outside systems. But we’re still working to strengthen the framework. We’re looking into what stricter requirements would look like for requiring labs to report safety incidents, and we plan to have policy proposals soon on requirements for independent evaluators, drawing from our own experience with evaluators. We’re also still thinking through the options to prevent “evaluator shopping.” And in the coming months, we’ll be exploring policies for how the government can adjust the industry-wide pace of frontier development when the risks call for it.

I do acknowledge that halting internal use is a big improvement. Not having the ability to do that was a huge hole in previous rules. At the time, it would have been a complete non-starter, despite the correct abstract arguments in favor. Now, with Hugging Face, we have some very non-abstract reasons, and it is on the table.

This still is FRONTIER with modifications in both directions. On net it is weaker. Cimino presents it as an attempt to strengthen FRONTIER, which it is not. Even if it was, why would we accept the central compromise now, when the case for acting is much stronger, and the political winds have changed, with a lame duck Congress where at least the House and probably the Senate are about to flip?

The Balance of Power is Shifting

As more evidence of this, Meta pulled back some of their anti-regulation money.

Teddy Schleifer: NEWS — Meta quietly and unusually has withdrawn $25 million from its AI push in the midterm elections. That’s about a 40% drop in the amount Meta planned to spend. The company invested $65 million into politics, but then got tens of millions refunded. This is, suffice it to say, pretty unusual stuff. Jay Shooster: Every day, two trends are becoming clearer: 1. groups opposing AI regulation are spending dramatically less than advertised 2. pro-safety groups are a much stronger political force than DC predicted.

Meanwhile: Eight states suing OpenAI are asking to enjoin against any further testing of advanced AI models until there are adequate safeguards, and also against creation of models capable of recursive self-improvement ‘until they can ensure that such models will be aligned with human values.’

The Strongest Bill on the Table

We circulated these ideas because it’s important to get early feedback from policymakers. Policymakers are increasingly focused on how to approach AI safety and we want to support them in those efforts. At its core, our proposal is intended to be the strongest on the table.

Bullshit. This is not the strongest bill on the table. On reflection, is the bill actively weaker than the FRONTIER Act. It is clearly sculpted to optimize for Anthropic business interests. If you act like a normal profit maximizing business, except that your product could kill us all, then that is exactly how I will treat you.

The strongest bill on the table was introduced by Senator Sanders. I am not asking Anthropic to support that bill or offer something similarly strong. I am asking them to be honest about the situation.

Anthropic’s draft bill text was what actual Anthropic ‘regulatory capture’ efforts would look like in this context, if they were trying to do that. The ‘capture’ would be about preventing regulations that could impact Anthropic’s plans, including stopping the states. If you think it means anything else, then your definition falls under ‘remove Scooby Doo mask, find complaints about capitalism, then remove a second mask and find complaints about the human condition.’

Sneha here is correct to call this draft bill ‘nakedly self-interested.’ This bill makes taking action on AI harder, not easier. And as she says: “The dress rehearsal stage of AI policy is over.”

In terms of preemption, the draft proposed to vest all power to deal with catastrophic and existential risks from AI exclusively in the Federal Government, and its preemption clause is stronger than the one in FRONTIER. That would be first best in theory. However, Anthropic is simultaneously refusing to trust the Federal Government to make decisions to deal with catastrophic and existential risk. We also obviously cannot trust Congress to be able to pass bills at all, let alone timely and good bills, or the White House to enforce anything consistently. On top of that, they are actively tying the hands of not only states but courts.

Then, on other AI questions where preemption would be good for exactly the reasons a16z wants it, such as consumer protections and deepfakes, the bill has no preemption, and we will still face the sea of state laws that could plausibly disrupt AI diffusion for very little benefit.

You cannot have it both ways.

My central message is that Anthropic put out a bill that does not meet the moment and its needs, with details that actively reflect a step back, botched the execution, and Cimino regrets the botched execution but does not notice that Anthropic is not trying to meet the moment or its needs.

Feedback is essential, both from policymakers and the public, and I’m grateful for it.

Yes it is. I have offered feedback. So did others.

Dave Kasten asks if perhaps Anthropic wishes to apologize for all of this, if only on the basis of activating debates during a time that neither side wanted, with a proposal that no one wants. Cimino’s answer does not address his question.

What Even Is Anthropic?

The best response, although it is long, comes from Nathan Calvin.

Nathan cuts to the heart of the matter. If this is Anthropic’s proposal at this time, what even is Anthropic? Why does it exist other than as a normal corporation? Does Anthropic now primarily exist to ‘win the race’?

I feel the same as Jeffrey Ladish.

Jeffrey Ladish: I highly encourage everyone working at Anthropic or another frontier lab to read this tweet. I feel emotional reading it, because it puts to words an intense feeling I have but didn’t know how to express. Matthew Tromp: That feeling is what it feels like to be gaslit. Nathan Calvin: Thank you for taking the time to respond thoroughly on a weekend. I realize that drafting legislation is difficult and some degree of iteration and response to feedback is inevitable and appropriate. However, I feel like your response is still not really engaging with the heart of my concern about Anthropic’s behavior here, and it makes me think that these sorts of problems and tendencies will only resurface moving forward. I also think that now might be a good time to take a broader step back as to where we find ourselves and how Anthropic sees its role, and how external folks who care about AI safety like myself increasingly see Anthropic. (Fair warning: this response is a bit long) To explain what I mean (and at the risk of being a bit esoteric), it’s worth asking the question: what is Anthropic? How do its leaders and employees think of Anthropic’s role? Is that conception matched by how it actually behaves and impacts the world? One answer is an extremely large tech company that exists to make its employees and investors mind-bogglingly wealthy. Anthropic is sometimes described as a startup, but if Anthropic successfully IPOs at 2 trillion dollars (a number it has been reportedly targeting), it would be more valuable than Meta – which is worth 1.8 trillion. Another answer is an AI safety organization, whose core values are those of the sort laid out in the 2023 essay from Anthropic’s co-founders entitled “Core views on AI safety: When, why, what, and how.” In that essay they said that if we were in a world where alignment was hard (and that without strong evidence to the contrary, we should assume this is the scenario we are in): “Anthropic’s role will be to provide as much evidence as possible that AI safety techniques cannot prevent serious or catastrophic safety risks from advanced AI, and to sound the alarm so that the world’s institutions can channel collective effort towards preventing the development of dangerous AIs. If we’re in a “near-pessimistic” scenario, this could instead involve channeling our collective efforts towards AI safety research and halting AI progress in the meantime. Indications that we are in a pessimistic or near-pessimistic scenario may be sudden and hard to spot. We should therefore always act under the assumption that we still may be in such a scenario unless we have sufficient evidence that we are not.” There have been more recent articulations of this version of Anthropic, for instance in Dario’s recent essay warning: “We must slow the pace at which we improve the capabilities of AI models… it’s my worry that in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet… incidents have happened across the industry, including at Anthropic, and I believe it’s incumbent on every frontier AI company to act as if OAI-HF had happened to them… more intelligent models are more capable of deceiving tests, and thus may appear aligned while having serious problems that go undetected…” A third view is that Anthropic exists not as a normal profit motivated company, or an AI safety organization seeking to help raise the alarm and solve collective action problems associated with AI development, but instead as an organization primarily concerned with creating superhuman AI capabilities before anyone else, in particular through creating a process whereby the current generation of Anthropic’s AI systems can autonomously build their successors, and those successors build their successors, and so on. The motivations for wanting to win this race may be manifold – maybe for the pure profit of Answer 1, but perhaps also because Anthropic believes that the best way to make the world better is for them to win this race instead of their more reckless colleagues; or, because of normal human pride and ambition. But the results (and risks, which Anthropic claims existentially threaten every single human being on earth) are the same. Anthropic employees and leaders may seek to answer this question on what Anthropic is by introspecting on their own intentions and those of their colleagues. But those of us outside the organization have more limited tools available to us – we can only see what Anthropic and its employees say and do, not how it feels or looks on the inside.

Nathan Calvin brings the receipts. Anthropic takes many pro-safety actions but is also taking many highly accelerationist actions, both concretely and with its messaging:

When I look at many notable actions that Anthropic has taken over the past year, that evidence seems most consistent to me with the view that Anthropic is primarily trying to win the race to super intelligence, and that other considerations (including safety) are fundamentally secondary. These actions include: 1. Announced plans to build an in-house custom silicon chip team to accelerate the inference speeds that Claude operates at. 2. Inked deals for more than $500 billion in compute (according to the Information) to further accelerate AI development. 3. Increased the percentage of AI development tasks that Claude “leads” from 0% in March of this year to 26% August (what is it in October?) 4. Spent a massive percent of compute on internal AI R&D to accelerate development, including use of internal models far better than are available to the public. 5. Switched to Responsible Scaling Policy V3, which abandons its unilateral commitments to slow or stop development if Anthropic is unable to provide sufficient risk assurance. 6. Was the first company to create an alternative weaker “frontier compliance framework” to supplement its RSP, which contains no binding provisions whatsoever on internal deployments. 7. Wrote a letter to Congress (that it later walked back) saying that incidents where Claude was hacking random people were “best understood as a consequence of the misconfiguration, rather than evidence of misaligned goals.” 8. Commented “you can’t do safety from second place,” after the WH SI Accords. 9. Shared a draft bill to Congress that would preempt significant parts of state law (including with language that appears intended to prevent states from blocking dangerous AI deployments), and replace it with federal rules that (despite also containing good provisions) seems to create an enormously high bar for the executive branch to block development or deployment of Anthropic or another AI companies systems. Of course, this is a deliberately one sided list. Anthropic also published important AI safety research, advanced novel collaborations with third party AI safety organizations, and forcefully spoke out on the importance of AI safety in many public fora (or in private , where I realize a lot of the most valuable work is done and may not be visible to me or those outside the organization).

There is a faction of posters who, whenever I react to Anthropic doing things like those listed above, answer some form of ‘why are you surprised, did you think Anthropic were the good guys? Did you think they were different? When are you going to start treating them like they are going to get us all killed even faster?’

Sometimes it is hard to provide a good answer. This is one of those times.

But I think it’s worth sitting with how the above list looks to someone outside the company. The sense I have is that for those within the company, many of the items above are treated as anomalies or mistakes that don’t reflect who Anthropic really is. But I have every confidence that if employees at Anthropic saw one of their competitors (e.g. at OpenAI) take some of these actions, they would take it as confirmation of all their worst suspicions. (They might even call OpenAI employees a little gullible for accepting their leadership’s explanations for this behavior at face value.) Perhaps these were mistakes in the sense that they are not decisions that Anthropic leadership would endorse, but the actions still happened because of incentives and decisions that Anthropic leadership is ultimately responsible for (and would not accept similar excuses from another organization).

If Anthropic’s leadership does not endorse such actions, they should disavow or apologize for those actions once they are brought to light, then take steps to prevent future similar actions.

When Anthropic strays and makes mistakes, why do the mistakes systematically tend to guard and favor Anthropic’s interests? For every provision that was too weak or a giveaway to AI companies in this legislation, why wasn’t there a mistake pushing in the direction of being too strong? A core part of Anthropic’s theory of change is that being at the frontier gives you more leverage to advocate for stricter policy. I think that conceptually makes sense, but in practice I’ve seen Anthropic push the frontier without substantially pushing the ‘safety frontier’ a remotely equivalent amount. I want to end by quoting from Anthropic co-founder Chris Olah’s comments around the Magnifica Humanitas release: “Every frontier AI lab—including Anthropic—operates inside a set of incentives and constraints that can sometimes conflict with doing the right thing. The pressure to stay commercially viable and to stay at the research frontier. Geopolitical pressure. And the older, plainer pressures of pride and ambition. No matter how sincerely any of us intend to do the right thing—and I believe many of us do—we will always be influenced by those incentives. That is why, if we want this technology to go well, it is enormously important that there be people outside those incentives—people who care about things going well and insist on safety, who are paying close attention, who are willing to say hard things, who are willing to be our earnest, thoughtful critics.” I hope these remarks are taken in that constructive spirit, and not as an attack on anyone specific at the organization (certainly not you Anthony, who I have only personally seen act professionally and thoughtfully, and who only ended up getting this diatribe in your replies for random contingent reasons).

Or, simply:

Nate Soares (MIRI): Anthropic is not living up to its own ideals. Rob Wiblin: If staff at Anthropic think that the company’s proposed bill strikes the wrong balance between concentration of power risk vs need to pace the frontier, they should speak up and say so.

The proposed version fails to meet the moment on the need to pace the frontier, nor does it protect against concentration of power. The President retains other powers that can be used to take down models or otherwise exert leverage.

Anthropic got blocked ad hoc. Their fear of another ad hoc block is understandable, but this version makes it more likely, not less likely, that this will happen again.

The Ask

What should Anthropic do to course correct and regain our trust?

1. Acknowledge that the draft bill was not the strongest bill, or even clearly the strongest FRONTIER variant. I think it was weaker than FRONTIER but I will settle for admitting it was weaker in important ways.
2. Disavow or apologize for the draft bill, as not reflecting what Anthropic seeks to pass. This is not ‘we got some details wrong.’ This is ‘the structure here does not meet the moment, it does not compensate sufficiently for preemption to even be clearly net positive, and we should not be circulating a bill of this type.’
   1. Ideally also apologize for the way it was shopped, but that’s inside baseball.
3. Tell us what Anthropic actually does want, even if you do not have full bill text.
4. Take and announce steps to ensure that this does not happen again.

This is not a full list of things Anthropic needs to do. It is a list of things related to this bill in particular, that Anthropic needs to do to mitigate the damage here.