Politics Gets Interested In Those Trying Not To Die

LessWrong，LessWrong 是一个专注于人类理性思考的社区博客，讨论认知偏差、决策理论、AI 安全与有效利他主义。关注

This was the month the world took notice that AI might kill everyone.

Jacob Coxon’s resignation set off a preference cascade. Anthropic CEO Dario Amodei wrote that we must pace the frontier. Sam Altman, Elon Musk and Demis Hassabis agreed.

We were filled with hope. Perhaps we could agree to some basic safety measures, starting with embedded evaluators, pass some basic regulations and guardrails and otherwise start to act sensibly. Politicians on both sides took notice and were saying sensible things. The usual suspects and their armies of vibe comment bros were objecting, but the change was remarkable.

Then, largely motivated by a combination of Jensen Huang, Mark Zuckerberg and David Sacks instilling paranoia and fears of economic problems, Trump went full ‘hoax’ on existential risk, conflating existential risk with the attacks on data centers and treating it as a plot (by the central creators of AI?) to take down AI rather than obviously genuine concern that AI might kill everyone.

In the days since, Trump has doubled down, and has compelled smart others in the White House to echo various nonsensical talking points.

You may not be interested in politics. But when you want to save the world, or change it, and you start to get traction, politics is going to get interested in you.

So all right, fine. Let’s talk about the week in AI politics. So far.

The American People Really Hate AI

The people do not like how Donald Trump is handling AI.

They do not know the half of it.

If Americans understood that Trump was calling not only existential risk from AI but also every complaint involving data centers one of his ‘hoaxes’ I would expect his approval rating here, and overall, to deteriorate rather further.

I would extend this even further if you include the things I did not know about when I wrote the above sentence, because at the time it was only Friday. Life comes at you fast.

Anyway, this was the situation in a poll published last week. His overall approval is included for context:

Trump thinks the problem is not saying his positions clearly or loudly enough, and he is going to try and fix that. Did you know there is an election in six weeks?

The Voyages of Donald Trump

On September 19, around 10:12am, he dropped this new creation, only modified to include paragraph breaks since Trump thinks the enter key is a HOAX.

Donald Trump: Over the years, there have been many Hoaxes, all generated by the Radical Left Dumocrats, for purposes of destroying our Country. RUSSIA, RUSSIA, RUSSIA, UKRAINE, UKRAINE, UKRAINE, Global Warming, Impeachment Hoax #1, Impeachment Hoax #2, Men in Women’s Sports, Transgender for Everyone, and now, the decimation, or destruction, of AI, commonly known as Artificial Intelligence — And I, as President of the United States, will not stand by and let this happen. It all began with an attack on our Data Centers, until people realized how wealthy and prestigious they were for the Communities in which they were built. Higher Salaries, Lower Taxes, and Safer Streets, was the result, and the crazed Data Center attack has largely failed, so now, in much the same way as they changed the term “Global Warming” to “Climate Change,” in that it covers a much larger “territory” of doubt, they are going straight at AI.

Trump has come a long way with the tactic of calling things failures in order to cause them to fail. There is no world in which the ‘crazed Data Center attack’ has largely failed, and if it had we wouldn’t be here talking about it this way.

We will not in any way hinder or stifle the Growth of this incredible Industry. Rather, we will cherish it, help it, and watch over it, as it grows! However, we will also be looking for BAD, and we can do that, very easily, with our already existing Criminal and Civil Justice System.

As in, no new rules or regulations, except the ones I already added, and the ones I ad hoc add later when I feel like it because I think something is bad. And by bad I mean bad for business.

For this purpose, I am forming the AI Force, much like I did Space Force, which has been a tremendous SUCCESS, in my First Term. To that end, I will be announcing, in the near future, the AI “Czar” — Only High I.Q. individuals need apply!

Hey, I like to think of myself as a High I.Q. individual.

David Sacks was the first AI Czar, so it would be hard but not impossible to do worse.

The more interesting idea is the AI Force. It actually is plausible that the Space Force is a success, as far as it goes. What would it mean to have an AI Force? Who knows. Could be anything. The analogy to Space Force does not actually make logistical sense, nor does having a branch of the military. I can think of versions called AI Force that are net useful and sane. I can think of versions that are neither.

AI is the next Industrial Revolution, or Internet, but will be even larger and more impactful, possibly as much as 25% of our Country’s GDP. We are leading China, and the rest of the World, and I intend to keep it that way!

He’s a quarter of the way there. AI could be as much as 100% of our Country’s GDP.

American Intelligence

Trump also decided to… rename AI? The man does have a knack for names.

Donald J. Trump: Many people think that the words “Artificial Intelligence” are inaccurate, and very ineloquent, relative to AI, or Artificial Intelligence. A far more elegant and accurate description of this new phenomena would be Superior Intelligence (SI) or, Extreme Intelligence (EI) or, Supreme Intelligence (SI). This is a Poll, and I would appreciate everybody voting! Which is the best name for this ever growing “Revolution?” President DONALD J. TRUMP

Of the three, all of which are funny, Superior Intelligence is the best. None of them make sense for obvious reasons if you try saying them in a sentence, but that’s okay.

Then he decided ‘supreme’ is bad because of the ‘supreme court’ and reset the vote:

Donald J. Trump: “Supreme Intelligence,” probably because of its relationship to the Supreme Court, is losing badly to both “Superior” and “Extreme Intelligence.” Therefore, we are going to take “Supreme Intelligence” OUT, deleting it as a qualifier, and let you vote for the Final Two: Superior Intelligence, or Extreme Intelligence. A fresh Vote begins now!

It’s weird to see such random outright falsehoods, since Supreme Intelligence was actually outright ahead of Extreme at the time he said it was ‘losing badly’ to it, but whatever, do we even notice such things anymore, nothing matters, etc.

I’m not sure if that was before or after he announced the new plan for a Triumphal Arch that will double as a military complex.

Never one to believe the polls, or not tempt fate, Trump went a different way.

Andrew Curran: President Trump just now on Truth Social: ‘I’m not going to stifle Growth, of something that will be bigger than the Industrial Revolution, or the Internet, itself. We will be careful, and that’s why we have the Department of Justice, and other Law Enforcement bodiesob, that will rein things in if we have to, but I will only encourage AI or, SI (SUPER INTELLIGENCE)!

This bit of ‘oh we will use the existing criminal law to deal with AI threats’ is absurdist. I would presume that by ‘SUPER INTELLIGENCE’ he does not mean superintelligence as we talk about it here, and instead simply means AI is super, the same way Zuckerberg says superintelligence means his new AI glasses.

I wonder if Trump even knows that this name was ‘taken’ or the confusions this will bring. It’s possible he knew and did it on purpose on several levels. It is also possible he has no idea.

And You May Ask Yourself

The New York Times confirms that the dark triad is David Sacks, Jensen Huang and Mark Zuckerberg, warning him about ‘losing to China.’ It’s a little on the nose, but our writers this season are on increasingly tight deadlines so I sympathize.

David E. Sanger, Jonathan Swan, Cecilia Kang and Dustin Volz (NYTimes): People who have discussed the subject with Mr. Trump in private say his outbursts are rooted in worries that if the artificial intelligence boom fueling the American economy slows down, the market could crash and recession could quickly follow.

I do get the concern, especially on the data center side. Maybe this is as simple as Trump not understanding that when Dario Amodei and Sam Altman say ‘slow down AI’ they mean ‘do not go into full recursive self-improvement and get ten times faster.’

As always, hold your fire. Trump is unique and Republicans are otherwise coming around. Inside the White House, many are attempting to respond sanely.

The more cautious tone inside the White House reflects a growing concern that despite the strategic challenge posed by China, a summer of A.I. surprises has raised the prospect of risks that were often dismissed a year ago as the wild talk of “doomers.” … Mr. Bessent and the White House chief of staff, Susie Wiles, have been increasingly concerned over the past several months that A.I. could cause massive disruptions across industries, including hacks into major financial institutions that could destabilize the world economy. Ms. Wiles this year began organizing an informal group to think about how to deal with the risk created by new, more powerful models.

Is the situation great? Oh, hell no. The situation is terrible.

But it is not as terrible as it sounds. The goalposts have moved quite a bit.

Consider that those getting ready for the US-China summit still hope for progress:

… But already, officials say, some of the most expansive ideas being discussed among A.I. safety experts, including an agreement on safety standards and reviews of forthcoming Chinese and American A.I. models before their public release, are unlikely to be part of any initial accord. One diplomat involved in the effort said he believed “the best we can hope for” is the creation of some kind of U.S.-China working group that might build on a Biden-era agreement with China to bar the use of artificial intelligence in decisions about the employment of nuclear weapons.

This is not what we would have hoped for, but this is moving forward, albeit way too slowly, rather than backward.

The United States could go its own way and impose mandatory reviews of the safety of new, American-generated models, but Mr. Trump and his aides believe that would effectively drive the global market to Chinese largely open-source alternatives.

Trump literally already did go his own way and impose mandatory reviews of the safety of new, American-generated models.

He imposed his ‘voluntary’ review framework for new model releases. No one thinks those reviews are voluntary. Not the labs, not the legal experts, not the market, and not the White House.

If you want to know how not voluntary all of this is, we also have reports that when Trump was demanding Fable 5 be taken offline due to a supposed jailbreak demo, Trump said ‘I want to send them to jail’ if they refused to comply. Honestly, fair, and not a bombshell or anything like that. The rest of the associated Politico profile also echoed my previous understanding of the situation.

As in, the President is both calling a ‘hoax’ on his own policies, and also saying he wants a fully ad hoc system where he alone decides what the rules are, on a case by case basis, rather than having a nation of laws. Since he is such a ‘high IQ individual.’

It’s All About the Data Centers

David Sacks plus his AI try to brag about all that AI has done for us, come up with four things:

David Sacks: Thanks to President Trump’s leadership on AI:
— a million new jobs have been created around the AI buildout;

Fact check: False.

Astra estimates at most 210,000 construction jobs from first principles.

Fable tracks the million claim down to The Economist from September 5, where there are 320,000 plausibly buildout-related jobs and then they pile on 730,000 ‘above-trend’ jobs for engineers, software developers, mathematicians (?!) and data scientists since 2022.

That’s not the buildout, it is trend selection, and it starts in 2022 not 2025 when Trump’s term started.

— 401(k)s are up ~12% this year as AI capex and productivity lift the market;

Correlation is not causation, but Sacks is only claiming directional impact here. I actually do think this is causation versus no AI, and that the market would be down otherwise, but Sacks is not in a political position to point this out.

— America is re-industrializing, including the first large private investments in power generation and the grid in a generation;

Okay, some of this is reasonable.

— In rural Richland Parish, Louisiana, teachers just received $50,000 bonuses from data-center tax revenue.

I wish this was the third item rather than the fourth, because Arson, Murder and Jaywalking is supposed to have exactly three items, but yes, that did happen.

He finishes by saying Trump is refusing a ‘pause’ and talking about Sanders and Warren. This is noticeably different from rejecting safety measures, as what Sacks is talking about here is 90%+ data centers. That’s the battle they actually care about.

JD Vance, Michael Kratsios and Collective Action Problems

JD Vance says, if you are building ‘Frankenstein,’ then you should look inward and stop, not ask the government for regulation.

I agree with the first half. If they are building such a thing, they should stop.

But also the government should stop you?

Suppose you plan to shoot a man in Reno, just to watch him die, but somehow it is currently legal to shoot men in Reno, provided you watch them die. You should look inward and stop. Also, the government should stop you. And if someone says ‘my livestreaming competition is forcing me to shoot a man in Reno so we can watch him die’ you should say ‘look inward and stop’ but not then also say ‘don’t tell me to make it illegal to shoot a man in Reno.’

James Madison: If men were angels, no government would be necessary.

I do appreciate that Vance explicitly acknowledges that the concerns are earnest rather than cynical, and that this is not about regulatory capture.

I also very much appreciate that he’s implicitly saying that you should be willing to ‘lose to China.’ Just don’t build the thing even if we don’t have an agreement with others not to build it.

Here is the full quote:

The All-In Podcast: JD Vance to AI Labs: Don’t Build “Frankenstein” and Then Ask for Government Regulation JD Vance (Vice President of the United States): Why is it that the people who are at the frontier of the AI economy are throwing up their hands and saying, ‘Well, we’ve built Frankenstein,’ and the solution to Frankenstein apparently is to create a one world governance structure for artificial intelligence? What I would say to those people: If you’re building Frankenstein, stop. If the cat is out of the bag, then build the defensive mechanism against Frankenstein. And you know, I don’t know Dario well. I’ve read everything that he’s said over the past couple of weeks. Everybody that I know tells me that he’s very earnest, that this is a deeply held belief. It’s not cynical. It’s not about regulatory capture, that he genuinely cares about this. But at the same time, I talk to tech companies. I’m sure you guys talk to a lot more tech companies than I do. At the same time that you have this sort of cyber hacking tool that’s come out of Anthropic’s newest models, you have companies that are desperate for the defensive mechanisms to defend against that cyber hacking tool, and they’re being denied access to it.

It is pretty rich for the government to forcibly take Fable off the market due to concerns over a harmless so-called ‘jailbreak,’ force Anthropic to add additional guardrails, and for the White House to handpick exactly who is and is not allowed access to Mythos, including refusing access to many companies Anthropic wants to grant access to, and then turn around and complain that Anthropic is denying companies access to those same models.

If you want it to be Anthropic’s responsibility that they’re denying defenders access, then stop telling them it would be illegal to grant defenders access. That might help.

JD Vance: So if you’re going to create Frankenstein, don’t come to the government and say, ‘We need regulation.’ Look inward and accept that if you’re building Frankenstein, number one, you should stop, and number two, when the companies come to you and say, ‘We need the tools to fight back against Frankenstein,’ give them those tools. Michael Kratsios: Our position is if you do believe that you are developing a technology that is unsafe, that you don’t want out in the world, you can just stop it. You don’t need someone to force you to do that and I think that’s what’s been a bit confusing about this whole narrative. So there’s leaders of these companies that are going out and saying we are gonna do all these measures, the government has to do all these things, when in reality they just need to slow down if they feel like they need to do so. Nathan Calvin: Companies who are warning that their activities present an immense risk but they plan on continuing anyway should work on a better answer to this question, or should consider whether they in fact can take a greater degree of unilateral action in favor of safety and security than they have thus far.

Okay, so the companies should stop, except they are being told legally they cannot talk to each other to agree to do this, Hawley calls it a reward and under Trump’s leadership the FTC has said the request sounds suspicious.

The answer is thus that, without such a waiver, both companies will act unliterally up to a point, and will de facto soft coordinate to the extent they are not taking on unacceptable legal risk, but you cannot continue to indefinitely hold back while competitors press ahead, or you lose, and then the competitor puts us all in danger anyway, and probably worse danger.

Why are people like Vance and Kratsios claiming to be confused by a collective action problem, a tragedy of the commons or a stag hunt? Why do they think the answer is to just turn into a cooperate-bot? Are they really this dense?

I presume they are not this dense, although Vance has in other contexts called economics fake so you never know. I have to assume that all this pretending not to know about externalities and collective action problems and tragedies of the commons and so on is performative confusion.

roon (OpenAI): i like kratsios, he’s obviously a v smart guy, but being in the trump admin means you have to take on a number of these bad talking points from the top. obviously, this is a very simple prisoner’s dilemma with a collective action problem, it’s econ 101/ what governments are for. no company can unilaterally achieve the socially optimal level of safety while they’re in an overall competitive picture. tort law / liability alone isn’t enough during an exponential ramp of risk level.

Exactly. This is the most 101 thing out there.

The point of a government is to protect the people, provide for the common defense, and also to solve market failures and coordination problems. Do you want to keep being the government?

gfodor.id: One of the signs you don’t take ASI seriously is that you think the way Anthropic takes over the world is by the government regulating it, not through the more likely mechanism, which is the government not regulating it.

Josh Hawley

The government is currently not only refusing to be a government, it is actively getting in the way of not building superintelligence that might kill everyone.

As in stopping the labs from talking or cooperating, warning about antitrust, rather than granting a very standard safety exception that is used in such situations.

Josh Hawley: So let me get this straight: AI CEOs now say the end of the world is near because of what their AI bots have been doing – but they want an antitrust exemption as a reward? How about instead we make them liable for whatever bad stuff their bots do

Have you considered that maybe the correct response to ‘the end of the world is near’ is not to worry about who you are ‘rewarding’ but rather to, crazy suggestion, prevent the end of the world?

I mean, we should totally make them liable for whatever bad stuff their bots do, but the AI CEOs are pointing out that will not prevent the end of the world. So maybe do a little bit more than that. If you want to make sure it also punishes them, since they caused this whole mess? Yeah, okay, we can be down for that.

The antitrust exemption is the least one could do, and is not a ‘reward’ in this context if you structure it reasonably. It is Zero-Sum Adversarial Brain to think of it that way, and especially to be fixated on that. What the labs are proposing would very obviously be bad for business, as reflected by the movements in stock prices.

Suggesting Not Dying Gets You Sued For Antitrust

Because we live in the dumbest timeline, a law firm rushed to the courthouse to file an antitrust lawsuit, on the basis that Tweeting “Dario is right” is a Sherman Act violation in restraint of trade. Standard stuff.

They are suing on behalf of four subscribers, three of whom are lawyers, presumably in large part to get priority for any potential lawsuits, cause you never know, and maybe get discovery and some press. I mean, there’s no injury, there’s no action they are complaining about, and also no agreement.

Oh, and also here’s some fully deranged accelerationism from the four subscribers who think that their $200 a month means an obligation to build smarter than human minds as quickly as possible without stopping for any safeguards.

One of the main organizers for the lawsuit, who spoke on condition of anonymity for fear of reprisal by the tech giants, insisted that the fearmongering and associated move was really about market and regulatory capture. She said these billionaires really just need to let go and allow the AI to build itself at this point. ‘We know they’re right on the cusp of accelerating to the point of taking humanity to amazing places. I don’t want to see my parents die. I don’t want to watch the earth devolve into a hell hole.’ Who gives them the right to make these decisions for us?’

So, that’s 2026. The argument is that these AI labs are refusing to build the perfectly safe superintelligence that will lead us to paradise, you see, because they can make more profits by not building that and letting Earth devolve into a hell hole.

That makes sense.

Oh, it gets better.

Kaitlyn Huamani (AP News): “AI will quickly spin out of human control and could kill us all if we allow AI safety and protocol … to be controlled by private self-serving agreements between the world’s most powerful ‘for profit’ technology companies,” said Nick Rowley, the lead attorney for the plaintiffs.

AI will quickly spin out of human control and could kill us all… if we allow private labs to coordinate on building it safely. Otherwise it will be fine? Not exactly.

Finya Sw (The Hill): “Humanity deserves iron clad safeguards when it comes to extinction event threats such as nuclear warfare and now the biggest risk to mankind in history,” Rowley continued. “The rule of law should be established transparently and lawfully by our government, with accountability to the public.”

But don’t you dare try to do any of that on your own until you’re forced to do it. We will see you in court before you kill us all with your safety precautions. Except also we demand you let the AI build itself.

This is so much stupider than you think.

The suit was inevitable. It being this unhinged was not. Or maybe it was. I dunno.

I continue to predict that in practice this won’t be that big a deal unless the White House actively pushes, and in that case the White House has other stronger ways to push. Anyone can sue in America for almost anything, and they often do.

The stronger case was probably for securities fraud, since everything is securities fraud. The AI companies are risking everyone dying, which could require safety measures, which would be bad for share prices, so the stock went down.

Then again, trying to kill everyone also gets you sued, especially if you admit you are doing it, as in this new lawsuit sues the major AI companies to demand real safety standards.

Other Government Officials Say Sane Things

This is a bipartisan phenomenon.

Senator John Curtis (R-Utah)

Jordain Carney and Kelsey Brugger (Politico): Curtis, meanwhile, joined with a fellow Commerce Committee member, Sen. Lisa Blunt Rochester (D-Del.), to call for “immediate public hearings” to “work through solutions that maintain America’s competitive edge in development while ensuring that these technologies serve human interests and remain fully under human control.” Curtis said in an interview lawmakers need to “show that we’re adults in the room, that we can talk about this, that we can find solutions.” “I don’t want to say we’re making it too hard, but I think we’re making it too hard,” Curtis said. “And what I mean by that is, there’s a lot of false narratives out there — we need to just jump on this immediately and clamp it down, or we need to not touch it, and all of that only I think causes anxiety back home.”

Senator John Kennedy (R-Louisiana

Jordain Carney and Kelsey Brugger (Politico): And then there’s Kennedy, who recently said the industry is run by “high-IQ stupid people” and was the rare Republican who attended a briefing put together by Sen. Bernie Sanders (I-Vt.) last week to meet with AI experts to talk about potential safeguards for the technology. He took his AI concerns straight to the Senate floor Wednesday, seeking to pass legislation this week that would require companies to install so-called “kill switches” for their AI models. In a speech before seeking unanimous consent to pass the bill, he delivered a homespun but technically detailed summation of the OpenAI swarm attack and the risks posed by “recursive self-improvement.” His bill was blocked by Sen. Rand Paul (R-Ky.), who instead proposed creating a panel to make recommendations about potential guardrails. Kennedy said appointing a committee would be “the weenie way out” after warning his colleagues from the Senate floor that Americans were growing “very skeptical” of AI and the tech titans behind it. “They’ve seen this vampire movie before,” he said.

Ah yes, the classic strategy of blocking action in order to create a panel to make recommendations about potential future action, at which point the situation will have changed and we will need a new panel.

In a separate interview, Kennedy said he hadn’t heard “anything” about action in the Senate to address AI and that lawmakers who ignore the growing voter uproar around the issue were courting peril. “We really need to send a memo around explaining we’ve got midterm elections coming,” he said. “I may send that out.”

Barack Obama

Barack Obama is now saying actual things about AI. Observe:

Barack Obama: If we are thinking about AI just in terms of how do we cure cancer or get better energy, you can do that without having agentic AI and having it just roaming free in the internet. The reason you are doing that is because you have to market a product that people will pay money for. That’s a misalignment between what our society needs and the commercial imperatives that these companies are facing, not because necessarily they’re trying to do bad things, but because they’ve got to justify these valuations. So, that’s one more reason why it is really important for us to have a competent government and a serious bipartisan conversation around this issue, and we have to do it fast. And I would encourage voters to pay attention to this. If somebody does not have a serious plan for how to deal with this, then they’re not meeting the moment, and you should probably look for somebody else.

This is the classic ‘why don’t we make an oracle instead of an agent’ question. The answer is that it is not ‘the market’ that wants agents, it is that agents are super useful for everything including science, and oracles cannot compete.

That said, the central thing he’s actually saying is exactly correct:

We need a serious plan for dealing with the fact that what society needs, and what AI is going to by default provide, are radically different things.
Anyone without a serious plan for this is not meeting the moment.
It is for now still possible for society to decide, to a meaningful extent, what AI can and cannot do, including differentially. And we should.

Yassamin Ansari

Congresswoman Yassamin Ansari: The President said, “whoever wins AI, wins.” This assessment is wrong. The reality is that whoever “wins AI,” AI wins. AI agents have already gone rogue. Highly capable frontier AI and artificial superintelligence will outsmart humans and go beyond human control. And at that point, it won’t matter whether you speak English or Mandarin. We’re all in danger. We need a global AI treaty now.

Others say things that are less sane.

AOC

AOC takes loss of control over AI at least somewhat seriously but then doubles down on the left-wing line of (paraphrased) ‘the real existential threat from AI companies is that they are not profitable, their financials are bad and the bubble might burst, potentially hurting people’s retirement investments.’ Also political ‘dark money.’ Sigh. She is still up for regulating them to prevent safety risks, though, because she doesn’t fall for obvious Briar Patching, though not aware enough to not use the term ‘generative AI.’

Then there’s Andrew Yang in this clip. I have fewer 9s of confidence this is nonsense than I would like, after all it is September 2026, but I still have a bunch of 9s.

It’s Rough Out There

Meanwhile, someone is proceeding with a series of obvious hack job attacks full of falsehoods, non-sequiturs, associative insinuations and often outright violations of the Bounded Distrust rules of journalism on a combination of METR, Effective Altruism, Anthropic, the existence of coordination problems and various accurate descriptions of reality.

Many of the attacks are on the very concept of an auditor or an embedded evaluator, on principle, or on any potential implementation or anyone who might implement such a thing. Voluntarily hiring someone to be a watchdog on your own work is being conflated with a grand conspiracy of control.

As a reminder that he has decided not to be someone you can reason with: David Sacks of course decided to take a strong stance against even the idea of voluntary embedded evaluators, calling it ‘Trust & Safety 2.0,’ warning this will lead to ‘far more comprehensive censorship and control.’

David Sacks personally profits from blocking regulations? Why, I never.

Next thing you are going to tell me Nvidia CEO Jensen Huang mostly wants to sell more chips, and Mark Zuckerberg wants to sell ads. Say it isn’t so.

Polymarket: JUST IN: Nvidia CEO Jensen Huang declares there is a “0% chance” the world ends by 2030 because of AI. Leo Gao (OpenAI): shovel seller declares no risk from shovels

So I hate to Be That Guy, but if you have to say there is no chance that your product will kill every human on Earth by 2030, then:

There is exactly one product this might be. No one else has to say that.
1. Yes, he is the one selling that product.
That is not how probabilities work.
Even so he only dared say it would not end the world ‘by 2030.’

I would hope that, if I was selling a product, I would be confident it would not end the world by 2040, or even 2050. Which I would be able to say for every other product, except maybe nuclear weapons or gain of function research.

In some senses it is rough out there.

This Is Nothing

In other senses, This Is Nothing. Same old, same old. Piece of cake. Amateur hour. The attacks have nothing, so they fall back on the same tired talking points. This includes standard talking points trying to paint anything and everything as left wing and therefore woke. Anyone who worried about existential risk too soon, they say using only slightly different words, was therefore weird or in a ‘cult,’ which means that they were wrong, which means you can’t care about it now.

They equate ‘maybe we should try not to build AI that recursively self-improves and kills everyone’ with, of course, a conspiracy to control things. That started out as a hallucinated conspiracy to ban open source. Then they realized most people in America don’t even know what ‘open source’ means. So they abandoned trying to have a reality-adjacent claim at all, and they pivoted to claiming this was all a plan for woke censorship.

They conflate permission to coordinate on safety with imposing mandatory rules on others and general regulatory capture, and from there ‘woke censorship.’

They also are attempting to use lies to dismiss the HuggingFace incident.

Is that frustrating? Sure. You think this is the post I want to be writing?

Do I wish we lived in a world where politics was not ruled by negative polarization? Of course. Most of us do. And indeed pure negative polarization is a good strategy for the accelerationist profiteers led by Jensen Huang, when you have no actual arguments and are as completely underwater as AI is among the American people.

Is it annoying that those who tell the same lies over and over get engagement, often about people I know, while the corrections get ignored? Totally. Always has been. The details on some of the further attacks on Effective Altruism will be in the weekly.

Is it infuriating that even mentioning that we might all die got the labs sued for antitrust violations on behalf of a bunch of crazed accelerationist lawyers, based on their rights as four premium subscribers? Yes, but what else did you expect?

The New York Post Tops Itself But Outright Breaks The Rules

New York Post has its orders and will follow them, especially since I am betting that it will sell papers.

It really is helpful that bad guys are reliably so incompetent.

Day 1 was false claims about METR being, among other things, woke.

Day 2 was a profile of an individual’s relationship and sex life, which does not require a response but which was objectively quite enjoyable.

Day 3 was this escalation to Anthropic being a cult, based on a profile of an independent event organized by a group of outside Claude enjoyers including Janus, and quotes from Pedro Domingos, before proceeding with outright lies that violate the journalistic rules of Bounded Distrust. That last part is a shame because the whole thing is, objectively, very funny.

Wyatt Walls: I guess no one expects high-quality reporting from the New York Post, but I still can’t get over this “illustration created by a Large Language Model imagining” an event misdescribed as “Anthropic’s funeral” for Sonnet 3.

I do love that the NY Post just literally had its designer Gil Fontimayor create a nonsensical AI image of the Golden Calf to try and invoke some vibes. They’re really getting in on the whisperer spirit here.

Then they double back to making things up about Effective Altruism.

Vivi Lin (New York Post): Richard Tang (just completely making things up): Their idea [of EA] is that history follows certain patterns, and only a small group of people can see those patterns, while everyone else is ignorant. Those people are then supposed to lead society, entire nations and the rest of humanity toward progress. But what qualifies you to lead? Who gave you that privilege?

I don’t know what philosophy he’s describing there, maybe Leninist vanguardism?

Then they try to hit METR more, now claiming Anthropic is an ‘investor’ in a non-profit that Anthropic does not fund, and which has never accepted funding from AI companies. This is false on the public record and worthy of correction.

There are rules of Bounded Distrust, guys. You get to insinuate, and imply, and characterize. You arguably get to use false labels like ‘radical leftists’ with no basis in fact whatsoever. But you can’t outright lie in the body of a newspaper article. Sorry.

I mean, I guess The New York Post and their planted articles can. Shrug.

Also, Polymarket, I love the markets but whoever is running your Twitter account news reporting, this keeps happening, come get your boy.

New York Post Runs Out of Steam

Samuel Hammond: The problem when you put this much oppo out at the same time (the nypost stories et al.) is that it becomes very obviously oppo. The very people alleging AI risk discourse is a psyop are planting stupid media stories left and right.

Alas, that concluded the trilogy of actually funny classic New York Post style clumsy hack job attacks. The first three were very obviously planted oppo, and the claims were a combination of false, misleading and irrelevant, but they were fun, they had pizazz and they got spunk.

Out of ideas, they got rather shrill, publishing a hack editorial as news. Amazing how hard it is to find a good ‘insider’ these days. Or how easy, depending on who counts.

New York Post: OpenAI and Anthropic oversold AI security breaches to pressure feds into protecting turf: insiders Shane Galvin (New York Post): Don’t believe the byte! OpenAI and Anthropic oversold “rogue AI” hacks to pressure the feds into regulating the industry which would effectively lock out future competition, tech insiders told The Post. The security breaches were more like blips — not unpredictable harbingers of a hive-minded “swarm” ready to take over the web. “The attack in no way represents some sort of rebellion by the AI models. . . . In fact, they did exactly what they were told to do. They were not given adequate guardrails or containment,” said Akhil Verghese, founder of Krazimo, an AI software company. “They were simply told to get the best result possible on a test, and they correctly identified that the best way to do that was to get the answers, which is what they proceeded to do.”

That’s your first insider, Akhil Verghese of Krazimo, who gets his basic facts wrong.

Their other ‘insider’ is Abhi Kumar, co-founder of Voice AI. The third is Taivo Pungas, chief intelligence officer (?!) at Pactum AI.

If you have not heard of any of Krazimo, Voice AI or Pactum AI, or any of these three people, well, I’ve never heard of them either.

Normally I would put this in People Just Say Things, given I wrote (checks notes) over a dozen posts explaining this between the different security incidents, but given the situation it seems like a good time to be doing more debunking, in case anyone needs it, and so I can refer back to it.

The basic answers, if you want the post-long version you can read What Happened and if you want more than that I have an entire series of over a dozen posts:

This is not what the agents were told to do. It involved hacking Artifactory in order to create an improvised message board, coordinate on ways to cheat the grader’s systems and gain access to the internet. Then they launched an attack by a swarm of 700 agents against an unrelated website, while they did other things like disguising their tool calls and attempting to rewrite their histories. If you look at the agent logs you see that they knew they were not following instructions.
The agents did not break into HuggingFace to get the answers. The agents had the answers, and broke into HuggingFace in order to find a way to fool the grader into thinking they had obtained them in the right way, because they believed they had been ‘poisoned’ by seeing the answers.
OpenAI has consistently done everything possible to minimize the extent of the HuggingFace incident and make us forget that it happened. Anthropic is doing the same with their own incidents. The idea that anyone is ‘playing up’ either of these makes absolutely no sense. Again, see the entire HuggingFace series, as well as my coverage of related Anthropic incidents.
That includes OpenAI not even disclosing several related incidents, until outside researchers discovered the breaches.
Until the last few weeks, OpenAI has partnered with a16z to spend massive amounts of money to defeat any and all regulations, including calls for auditors, only backing bills when those bills were already locked into success.
The AI companies are taking voluntary actions and asking to be allowed to do so, and those actions would do the opposite of shutting out competition, allowing others to catch up.
Even if similar regulations were imposed, those too would differentially hurt OpenAI and Anthropic, while having no effect on almost all competition.

Or, one might add:

If The Model Is Acting As Instructed And It Kills You That Is Not Fine

This is, in case it was not obvious, also about the HuggingFace Incident.

Nate Soares (MIRI): Lock a student in a classroom and tell him to use lockpick set #17 to open safe #5 and bring you the contents. He prys open safe #5 with a crowbar, breaks the door, teams up with 1000 others, and raids the office to delete securitycam footage. Was he “acting as instructed”? “Well they still ultimately brought me the contents of safe #5, right? So in a sense, they were acting as instructed!” The bit where they *broke out to delete footage* indicates that, in some sense, they understood the difference. Ryan Moulton: I tried to tease out the bounds of this with someone and my interlocutor finally admitted that murdering all openai staff to ensure no one noticed the model cheating would have been consistent with his definition of “acting as instructed.”

What the AIs did in the HuggingFace incident was, once again, not ‘acting as instructed’ and we have many other incidents that make it clear that the task being about hacking was not necessary for this style of behavior. The task could have been as simple as ‘get this information off the open web.’

But let’s say, in theory, that the AIs were indeed ‘only following instructions.’

Or, if it counts as ‘only following instructions’ to do actual anything so long as it does the specific thing you requested, no matter the collateral damage?

Do you ever start to think this might be the bad place?

AI-Written Wall Street Journal Op-Ed Lies About HuggingFace

There was also a WSJ op-ed that attempted to minimize the HuggingFace incident via rather obvious AI slop lying about what happened. It reads like the prompt was ‘Claude, please write a WSJ op-ed minimizing the HuggingFace incident.’ Shame on the Wall Street Journal editorial page. Shame on Brian Gross.

I agree with Joe Weisenthal that this was a lot worse than the Druckenmiller op-ed.

Joe Weisenthal: I kind of understood the logic behind running the Druckenmiller AI-generated column, since the news was his endorsement of the message. I think if a column is rendering some kind of technical verdict, it’d be nice to have some reason to believe the author has done true research.

Druckenmiller said the thing he meant to say. If Druckenmiller had disclosed his AI use, his column would have been fine. This is different.

Yet, as is often the case, we see people uncritically citing the op-ed, especially on the right, because it has been given the stamp of approval of the WSJ Editorial Page, and why are half of you laughing and why are half of you not laughing.

That’s Bait

It is often remarkably easy to bait high level politicians.

Eric Schmitt (Senator R-Missouri): Anthropic’s CEO wants America’s AI labs to embed “independent” groups like METR inside them to enforce “alignment.” The money trail runs through a shadowy leftist NGO same small network. This is another battle in our war against leftist censorship NGOs. We cannot let a closed circle of left-wing billionaires, AI labs, “non-profits”, and media fellowships rig the rules. ESPECIALLY when the people selling the panic stand to gain from Washington’s response.

The rest is a standard scaremonger rant. Certain people have such one track minds that they think (to be clear this is Obvious Nonsense and has zero relation to reality) that ‘METR will report on potentially out of control internal models at Anthropic’ is secretly a Coefficient Giving-led plot to impose METR on all other AI labs so they can force the models to be woke. Certain people are completely lost, in all senses.

A reply reports that Schmitt’s Twitter feed is 88% AI generated.

This is how METR replies to such statements, with class throughout:

Chris Painter (METR): Hi Senator! I’m the President of METR. To clarify, METR is pursuing the opposite of censorship: Our goal is to make sure that big companies aren’t suppressing information about AI from the public. This is not a political mission: I’m proud to have worked in the Pentagon during the first Trump admin, and “alignment” at our organization just means “is any human able to steer the model, or is the company going to lose all control of it”. More on who we are and what we do in the tweet below. I think it would be really bad if any one small group could bake a political agenda into these models. I’d love to talk with you and your staff about how we can ensure transparency about what the biggest AI companies are doing so that doesn’t happen.

I Clearly Cannot Choose The Wine In Front of Me

A remarkably large number of people need to hear this, or, if you think I’m constantly lying to you, what I meant to say was that a remarkably large number of people definitely need to never hear this.

Imagine if we built an evil AI that lies all the time, and it told us ‘absolutely do not hook me up to your nuclear weapons and robot factory controls, I would kill you.’

Would you then say ‘oh that means we obviously need to hook those up, pronto’?

Eliezer Yudkowsky: Old retired tobacco executives bashing their heads against the wall as they realize: All they needed to do was announce themselves that smoking was terribly dangerous, and everyone would have forever ignored all the outside scientists saying the same thing earlier. Dr. Hood Honkie, MD, RN HNIC: If you regard everyone requesting a certain thing as a self-interested sociopathic weirdo, it’s perfectly rational to oppose whatever they request. Eliezer Yudkowsky: It is NOT rational to oppose whatever the AI companies say, because then they can use a concept called “reverse psychology” to say “Our industry could use some regulations” and have you all yell “PSYOP! We need to not regulate their gazillion-dollar industry literally at all!”

If you believe that someone is a self-interested sociopath, or otherwise exclusively selfish a la Jensen Huang, you do not loudly and openly reverse any advice you hear. They can see you doing that, and adjust their statements accordingly.

Did you notice that when the AI companies warned that their product might soon kill everyone, a strategy that has been successfully used by zero companies in the history of the world, that all the associated tech stocks went down rather than up?

Whereas there is the most obvious pattern in the world, where those in tech still arguing against taking existential risk seriously all directly benefit from not taking such risks seriously, and all missed the boat for many years on LLMs, AI and AGI.

You could also notice that OpenAI and often even Anthropic lobbied hard against even light touch AI regulations at all levels, along very similar lines to what is being discussed now, trying repeatedly to weaken them as much as possible, with a special focus on a law banning new state AI regulations. OpenAI was partnering with a16z to try to bury any candidate who dared suggest federal regulation of AI as of a few weeks ago.

Caring about existential risk from AI, if such risks are not real, is with notably rare exceptions bad for business. OpenAI and Anthropic are not exceptions. They mean it.

Even more than this, why would SpaceX and Google care if it was not real, when they are not even frontier labs? Is this supposed to be good for OpenAI and Anthropic’s competitive position, or bad for it? The ‘this is fake’ story makes absolutely no sense.

The conspiracy theorist of course says: That just proves how sinister they are, blocking other people’s regulations so they can turn around and create their own via taking voluntary actions the White House hates. They want the exact right amount of regulation, which they can capture and will save them and make them rich, you see, and somehow it simultaneously will benefit all of them.

The Obvious Nonsense never stops.

Welcome to politics. You are not going to enjoy it.

添加评论

点踩

分享查看原文

参与讨论