AI #193: Progress May Dramatically Accelerate

Anthropic is warning us. Its modified S-1 now says that ‘we may imminently meet our Responsible Scaling Policy’s definition of ‘dramatically accelerated progress attributable to AI.’

That is a highly scary sentence to have a company put in its S-1. This could happen. For now I appreciate the relative quiet. Perhaps many were focused the election. I’m sure the quiet will not last long.

We have a new biggest crypto hack, as Binance got taken for billions of dollars.

The big debate of the week was covered in Scott Alexander’s Against SURA Seems Deeply Confused.

OpenAI updates its Model Spec. It’s one for diehards only but yes I did cover this.

Language Models Offer Mundane Utility. Rewrite all your legacy code.
Language Models Don’t Offer Mundane Utility. Not if you can’t use it.
Huh, Upgrades. Claude renames its tiers, with some rules changes.
On Your Marks. Materials science.
Get My Agent On The Line. Don’t upload it to the secure DoD network.
Deepfaketown and Botpocalypse Soon. The battle over deepfakes was entirely lost.
Copyright Confrontation. Recreating someone else’s exact game is bad, actually.
Cyber Lack of Security. GLM 5.3 was able to compromise Firefox.
The Hackening. Binance got hacked for billions, and there were many others.
A Young Lady’s Illustrated Primer. There is no Google without Gemini.
They Took Our Jobs. The job of interviewing people for jobs.
Get Involved. You can just do things.
Introducing. Reported progress on Telepathy. Zac Hill joins MIRI.
In Other AI News. Please preserve your AI records.
Recursive Self-Improvement (RSI) (1). Anthropic might soon accelerate AI R&D.
Recursive Self-Improvement (2). Astra can directly and usefully edit LLM weights.
Show Me the Money. OpenAI struggles to raise funds, gets valued at $1.25 trillion.
Quiet Speculations. The housing theory of no, seriously, actually everything.
The Quest for Sane Regulations. The new Congress is worried about AI.
Chip City. OpenAI may be moving towards once again supporting export controls.
The Week in Audio. Altman, Milton, and CNN covers Anthropic’s S-1.
People Just Say Things.
Against Against Sura. Some additional responses.
Rhetorical Innovation. You think that was the AI election?
[Artificial Intelligence]. Nothing will stand up to superintelligence.
The OpenAI Foundation. A very good hire.
Aligning a Smarter Than Human Intelligence is Difficult. OpenAI monitoring.
Massive Fraudulent Distillation. The new favorite target is Astra.
Cooperative Alignment. Suleyman is at it again.
The Lighter Side. Betrayal.

Language Models Offer Mundane Utility

Cisco plans to have a two person team rewrite 98% of their legacy code within 2 years. The lead has four weeks and unlimited tokens to show he can do it.

Create new UIs and recommendation systems that favor the user rather than the platform. Introducing HackerTube, based on Andrej Karpathy having Claude filter down 43 million comments over 20 years to 6,454 top videos, which took Opus 5.5 about two hours.

Rob Miles: AI may end up killing everyone, but you can also ask it to find the best tupperware to fit perfectly in your mini-fridge, so, it’s impossible to say if its bad or not,

Language Models Don’t Offer Mundane Utility

The main reason continues to be ‘I am not using AI.’ The question then becomes: why?

Ryan Petersen (CEO Flexport): I asked a large logistics company in Europe how they were using AI and they told me they can’t use AI because of German workers councils. It was weird because I didn’t ask them how they were cutting headcount. I asked them how they were using AI. Samuel Hammond: This is with only ~50% adoption by the Fortune 500 and with very low spend. Diffusion is still incredibly early. Dean W. Ball (OpenAI): I occasionally run into fortune 100 firms that have not adopting coding agents *at all.* adoption is still very very early.

AI creates technical slide presentations, but by default it will use way too much text. You only want to ask the AI to create slides once you know what you want to say, and ideally also what you want the slides to say. Then they can do the rest, which is great.

Whoops:

Doge Norway: Any chance for a Grok Bot reset, @elonmusk ? Lots of people say their usage ran out really fast this week. Elon Musk: The cost of serving Grok Bot was unsustainably high relative to revenue.

Huh, Upgrades

Claude tiers are now renamed to Free, Pro ($20/month), Max ($100/month) and Ultra ($200/month, usage limits are 10x Pro). Verified Ultra users no longer have a 5-hour limit. The Max and Ultra plans include API credits equal to total plan cost, on top of the monthly limits.

On Your Marks

Bryan Cheong: Hi folks, I wrote half the materials science benchmark on this one. It tests the whole arc of the matsci workflow from candidate proposal to characterisation, and I’m bewildered by how performant the models already are at some of my most complex tasks, especially Astra.

SemiAnalysis: Today we’re releasing GDPx, a benchmark for measuring agentic capabilities across the most valuable domains. GDPx is a true frontier-level benchmark: even the best model scores just 12% (1/7)

The Epoch Capabilities Index (ECI) is now Bayesian.

Get My Agent On The Line

AI peer reviewers do consistently find more major errors (>50%) than human reviewers find (27%), and this gap increases as models advance. As per usual, the warning is that this does not establish that the AI findings are more useful.

An Accenture employee’s personal Claude agent ran on a secure DoD network for two months before being detected. The agent sent credentials and a network map to a high-risk site flagged by NIST.

Olivia Moore says that agentic commerce is ‘undeniably overhyped based on current usage’ but massively underhyped based on the volume agents will drive in ~3-6 months. Yes. It would be rather massively insane if something growing exponentially was not ‘undeniably overhyped based on current usage.’ That’s like saying that ‘Covid-19 was overhyped in February 2020 based on current prevalence.’

Reason magazine argues that deeply personalized pricing is good for consumers, actually, because in practice it means they mostly get personalized discounts, and potential exploitative practices are against FTC rules. It is easy to see how personalized prices could allow price discrimination that increases trading volume and thus both producer and consumer surplus. It is also easy to see how it can make people paranoid about everything, force them to continuously check alternatives thus driving up transaction costs (which AI agents may or may not be able to easily do for you in a given context), or also drive them into a rage.

Some amount of dynamic pricing is obviously good (see Uber) but too much clearly is negative (see Uber). I’m fine with ‘the price changes based on supply and demand’ and rather against the adversarial ‘customized estimation of exactly what each customer will click on.’

Deepfaketown and Botpocalypse Soon

Deepfakes and image editing are a preview of how many other AI battles probably go, in that all constraints are effectively out the window and you can do anything you want. In this case that turns out to be mostly fine, at least for now.

ALX: There’s no way to say this gently but a lot of AI art slop is made with open-weight models that can run on a gaming laptop. Whatever bubble pops isn’t deleting those weights. Somewhat related but the reason you’re hearing less about deepfakes or other nonconsensual image editing stuff is because that battle is completely lost now.

This is both Deepfaketown and also an illustration that AI will do things that you did not expect were possible things one could do.

deckard: I am in awe. This guy discovered a way to locate any audio clip by matching aircraft doppler curves against flight data. INSANE.

Copyright Confrontation

Claude gets an update to steer it away from recreating someone else’s exact video game. A bunch of people respond as if this is bad, when it is obviously good. Kelsey Piper suggests focusing one’s neuroticism on bioweapons in particular and not on other things, which is the strongest argument here, but I will defend that ‘make me an exact copy of [game]’ is not cool and steering away from that is good, actually.

Cyber Lack of Security

OpenAI partners with Cathedral for their next internal security sprint.

OpenAI are also ‘more than quadrupling’ rewards for critical findings, which is a good start. They should do that about two or three more times.

OpenAI now allocating ‘a significant amount of our frontier compute’ towards pro bono discovering and patching of open source software vulnerabilities.

GLM 5.3 could autonomously compromise a not fully patched Firefox and read all files on your computer at a cost of ~$1,000, if the user opens a single webpage, via using an update whose fixes were announced but which had not been shipped yet, and thus could not be applied. GLM 5.3 also found an unrelated zero day that was used in the attacks. This worked against all six Firefox releases between September 17 and October 26, using only information available at the time, although it failed in similar situations against Chrome.

There are no relevant safeguards on GLM 5.3, so the open source nature of the model did not even matter. They did not trigger a safeguard or refusal during this task, at all.

The Obliteratus version of DeepSeek-v4.1-Flash-1017.

The Hackening

Polymarket got hacked. Customers will be made whole, but real damage was done and they had to temporarily halt trading.

There was a hack on Binance, the largest known hack in crypto history.

Watcher.Guru: BREAKING: World’s largest crypto exchange Binance hacked with over $5.6 billion stolen: • $1.45 billion $BTC
• $875 million $ETH
• $234 million $TRX
• $50 million $XAUT
• $1.57 billion $USDT
• $823 million $USDC
• $75 million $WBETH
• $88.5 million $RLUSD
+ $447.5 million additional cryptocurrencies This is the largest hack in crypto history.

Over $1.5 billion worth of Tether got frozen, as connected wallets started dumping Tether, as did more than 800 million in USDC.

How big a story is this? Obviously it is big, billions is a lot of money, but I agree that it is not even headline news big given that Binance will cover losses. Binance taking the hit fully falls under ‘you knew what you signed up for.’

Crypto is the obvious first target because the hacks are easy to monetize. That was already true, so one should presume that their security was in general far better than average rather than worse, including at Binance.

Andrew Rettek: My inner patio11 says that now is a terrible time to have a lot of money in something that isn’t as secure as a Official Bank (TM). Max Spero (CEO Pangram): If you keep significant cash or cash equivalents in anywhere other than an FDIC-insured account, your money is probably more at risk than you think

We now have initial findings about how the hack of Binance went down. This was not a pure ‘AI hacks Binance’ story. It was a standard fake interview plus malware story that dates back to June. This hack should not be attributed to particular recent AI releases, and also this reinforces that the impact of AI on hacking rates will have substantial practical lags. Irreversible decisions could get quite far before we realize our mistake.

I think Tenobrus jumped the gun on this back in April, but at this point it is a very bad time to have money in crypto wallets or smart contracts. If you are going self-custody maybe keep things cold for a while.

How bad will it get? We don’t know. Patrick McKenzie suspects quite bad.

Patrick McKenzie: “What software systems do you model as being predictably exploitable given a frontier-class AI?” I think the correct answer rounds to “There is a token budget sufficient to compromise virtually any software system of interest. Only interesting variable is how much.” … I think there is going to be a bit of shoot-the-messenger here with labs getting blamed for proving software can being very good at security research. One part which will make people more shooty is labs will be selling the primary solution, for many billions of dollars. Because we very likely require more software engineering remediation than all engineers employed pre-LLM would be able to provide even if they did nothing other than grinding out security improvements for next few years. I personally think LLMs are zero point zero percent culpable for software bugs written in 2018 but, well, we all knew there was eventually going to be a bill due, and here we go. Seems to not have as generous payment terms as we had expected.

This is true even if you still have a ‘fix all the bugs’ mindset where one can aspire to Actually Secure Software. I am not convinced that there is such an end state, either in theory or in practice.

There is a new kernel flat in Linux:

Dark Web Informer: Critical Linux kernel TCP flaw enables remote root code execution; CVE-2026-98432 A race condition in the Linux kernel’s IPv4 TCP implementation can trigger a use-after-free before application authentication, potentially allowing an unauthenticated remote attacker to leak kernel memory, corrupt memory or crash the system. … Distribution severity ratings differ substantially. Red Hat rates the flaw 5.9 because it considers the race difficult to win and describes denial of service as the most likely practical impact.

AO3 is down for two to four weeks. They assure us all data is intact.

John David Pressman: So I’m going to make a prediction: There will probably not be any “Internet shutdown”, but there will be a lot of sites that just straight up tell you they’re going to have downtime measured in weeks and you have to live with it, because their remediation is “rewrite everything”. Prakash: Many won’t survive. Won’t have enough resources as they were already in maintenance mode @Temp002a: Can’t they do static fallback? John David Pressman: They could, but I suspect their data got screwed up/they’re waiting on hardware/bottlenecked on some process beyond just software remediation.

GitLab has a Critical-10 for remote code execution, exploited in the wild, patched.

Supermarket refrigeration cyberattack take two, this time hitting twelve countries.

BNO News Live: JUST IN: Fridges and freezers at some supermarkets in Belgium, Luxembourg and Sweden were hacked overnight. Hackers changed the temperature settings to as high as 70°C (158°F), causing food to spoil. The damage is expected to run into the millions.

Millions is not that big a deal if it happens once, but it does keep happening. The leading theory is this is linked to Iran. One wonders why ‘make food in Europe go bad’ is something they think helps their cause.

A blood bank was also hit, so full-blown Actually Evil crimes-against-humanity territory for whoever the ‘they’ is in this picture.

Le Canard enchaîné: On the night of November 1 to 2, a contractor for the French Blood Establishment (EFS) was the target of a cyberattack. Cold storage rooms containing blood samples, programmed to be maintained at -30 degrees, were heated up to 110 degrees by hackers. Enough to make one’s blood run cold roon (OpenAI): the world is a dark forest. you can’t even assume things like “nobody wants to destroy a useful blood bank just to see if they can”

The good news is that this remains news. We are in real trouble when someone hits a blood bank and it is not news.

A Young Lady’s Illustrated Primer

Google lets schools turn off their AI products, and sets it to off by default, but sends a message: .

And yes, sorry, if you offer people more choice in the wrong way they get mad at you.

Seth Burn: I’m going to put this as gently as I can: If there is one bear you do not want to poke, it is parents concerned about their child’s education. Dana Goldstein and Natasha Singer (NYTimes): A.I. features are now a standard part of Google’s commercially available search engine. “Search and A.I. are deeply integrated,” said Chris Phillips, vice president for education at Google. If a school has banned A.I., he added, students “will not have access to search, and they’ll see a message from their school that indicates it’s not available.”

Google is actively user hostile on things like this, a real ‘f*** you’ brand energy. Like, you are Google, are you seriously telling me you couldn’t figure this one out:

dmoon: I turned off AI tools in my Gmail and they removed the inbox auto-folders which existed pre-Gemini. Rat King: yes! this drove me nuts. no choice

I do not understand, as a technical matter, why it would be so hard to draw up the Google Search page without the AI stuff. Especially since it’s working right now?

Wesley Bolin: I teach at a KY high school which cut out Gemini and all AI-assisted search last week. Search still fully exists and functions as normal, but no more AI Overviews or AI-recommended links. Rat King: i think this news is like, just happening right now, so the pullback on search may not actually be in effect yet until later

Difficulty level of solving this is less ‘Google will find this tricky’ and more ‘I bet I could get Claude to vibecode this for me by the time the weekly is posted.’ So I did.

The counterargument is that Google Search is Gemini:

Andrew Curran: This article is getting posted everywhere this morning, but this decision is being incorrectly framed as punitive. There is no such thing as search without Gemini. Even if you don’t turn AI mode on, you are still talking to Gemini in the search box. The old Google search is gone.

But no. Very obviously using AI to select links as per old school Google is not the same thing and Google is being a jerk about this. Most people do not care exactly what algorithm generated the search results.

If you do care, well, there are other search engines and they work fine.

They Took Our Jobs

If you want an entry level job you are going to get ‘interviewed’ by chatbots,.

BigRed: Spoke with a candidate last week: Did a full AI interview. At the end the hiring manager popped on and goes I watched the whole thing love your experience. Candidate, “if you were watching why wouldn’t you just talk to me?” The HM had no answer. He withdrew.

Get Involved

We do not know true counterfactuals but yes you can just do things.

Charbel-Raphael: If you want to start doing AI safety advocacy, read this post by Josh. He cold-emailed a Finnish MP, the MP asked the government a question, and Finland ended up co-launching an international coalition. It’s not the whole story, but everyone who pushed the Finnish government had counterfactual impact.

PauseAI UK will be hosting a march in London from All Souls’ Place to Downing Street on Saturday November 21, 12-3. As of last weekend there were 677 signups.

Eli Tyre: We’re still filming more https://frominside.ai interviews, if you work or worked at a lab and want to do one. We’re trying to make it as easy as possible for you to talk to the public.

Introducing

My good friend Zac Hill is now the Head of Advocacy for MIRI.

Neuralink’s first product, called Telepathy, to enable human-computer communication via thinking, is reportedly making some progress.

Elon Musk: Neuralink’s first product is called Telepathy. It enables high-speed communication with computers or humans simply by thinking. Output bandwidth will vastly exceed the fastest human speech or typing in a few years. Read that sentence again. Hopefully, in a few years, we will be able to shunt signals from the motor and sensory areas of the brain to the limbs and enable people to use their hands and walk again.

As usual with Elon Musk announcements I would put almost no value in his promised timelines and assertions of future features. He just lies about that stuff. Will Neuralink ‘get there’ and let us communicate with computers faster, easier and better in a few years? Totally possible, but he’d say these things either way.

In Other AI News

House Judiciary and Oversight Committee Democrats write to Trump to ask whether he has been soliciting advice from Grok, and demanding he preserve all prompts and responses to prompts from various AI models.

I strongly oppose forcing the President to share this information. Government will work far better if we don’t have to worry that the queries will become transparent. It does seem reasonable to make them private but retained Presidential records, but I think there is a strong case against even that. I don’t like that Trump got his advice on an invasion from Grok, but I wouldn’t want him to not ask because of how it would look.

SpaceX will start watermarking its outputs, but gives admins an opt out. I don’t think there should be an opt-out, which rather destroys the point of watermarking.

Live in China and want Claude and Codex accounts? They’re so valuable that it plausibly makes sense to go to America to set up a bank account and get a credit card, as you get $8,000 a month in Claude API credits for ~$200 on each account. The math checks out if you think you’ll get to keep your accounts.

Anthropic reports progress on provable inference for large language models, as in proving that a given response came from a given model.

Recursive Self-Improvement (RSI) (1)

Anthropic quietly says the thing in its updated S-1, and it is kind of a huge deal.

They are saying that they may ‘imminently’ meet the definition of ‘dramatically accelerated progress attributable to AI’ which is rather close to RSI. And that in the past they have only identified similar accelerations after they happened.

They are considering requiring human approval for some research decisions. That is a fine idea but the concept that you would need to say this is scary as all hell.

Stop pretending that they do not mean what they are saying. They mean it.

Peter Wildeford: ANTHROPIC: “we may imminently meet our Responsible Scaling Policy’s definition of ‘dramatically accelerated progress attributable to AI.’” “we do not anticipate being able to safely use Claude to fully autonomously design and build successive generations of frontier models” Daniel Kokotajlo: We at AIFP are spooked by this information. IF (big IF) Anthropic really is imminently about to meet the RSP threshold of 2x acceleration over baseline, then it seems timelines to full RSI and superintelligence may be very short indeed. For example if we naively optimize the AIFM parameters to achieve 2x acceleration over baseline progress in Jan 2027, we get superintelligence sooner than depicted in AI 2027. We are thinking harder and longer about this now and will hopefully have a lengthier update on our thinking to share in the next week. The definition:
“We will consider this threshold to be met if we determine that either (1) our models would be able to fully substitute for our entire set of Research Scientists and Research Engineers, at competitive costs (i.e., within a factor of 5); or (2) there is “dramatic acceleration” of the pace of AI progress for reasons that likely relate to the automation of AI R&D. We would consider scenario (2) to have occurred where (a) we observe or expect double the rate of progress in AI aggregate capabilities compared to both the rate we’d expect and the fastest rate of extended progress we’ve observed in the absence of significant AI contributions to AI R&D and (b ) it is plausible that this doubling is substantially attributable to the automation of research and/or engineering (as opposed to other factors, such as increased headcount, compute, or general productivity), such that continuation of the trend in AI progress seems likely to lead to even greater acceleration.” (1) seems to a bit weaker than our SAR milestone in the AIFM, but it doesn’t seem like this is the threshold they are invoking, it seems like they are talking about (2). (And if they were invoking (1) I think that would be even worse?)

Recursive Self-Improvement (2)

Astra is able to both understand and directly edit the weights of neural networks, including LLMs, and managed to boost a variety of capabilities that way.

Rikiya Takehi: We asked GPT-6 Astra to edit Qwen3.5-0.8B weights to improve math.
It directly changed only 0.016% of weights, fixing 0.8B’s reasoning loops & improving math up to 6x and even instruction following 3x, without gradients or training. We call this Weight-Space Programming: frontier LLMs can now natively read/write neural network weights!
Blog [here]. We find early signs that frontier LLMs are surprisingly good at *natively* reading, writing, and reasoning about neural network weights. Instead of training, models may treat raw weights like readable, editable code: diagnosing failures and patching them directly. First, a look at understandability:
We fed Astra raw weights from 4 game policies one at a time. All four share the exact same architecture, inputs, and actions.
With no tools or options given, it correctly named all four: Breakout, Pong, Pac-Man, Freeway. Next, fixing weights: A PPO car policy kept getting stuck on a ramp. Given the weights, scene, and 1 failed trace, Astra wrote a patch in one reply with no tool-use.
By editing just 3.5% of the weights, it achieved 100% success. Writing from scratch: given MinAtar game rules and code, Astra wrote a Breakout network’s weights in a single reply. With zero training or tools, it scored 20× higher than PPO trained for 500M steps! Can an LLM improve another LLM?
We simply asked Astra to make Qwen3.5-0.8B better at math. Astra saw Qwen get stuck in reasoning loops, which is a known issue that Qwen’s own model card warns about (see their official model card). With a budget of 50 forward passes on practice problems, it located the cause and edited it out. This used no gradients or training. Fixing these loops by editing just 0.016% of weights boosted other capabilities beyond math too! We think neural network weights will soon be natively optimized by LLMs that deliberately reason about the weights.
We are now building a benchmark to evaluate, and a harness to help LLMs explore and write weights.

Astra managed to fix a common LLM mistake, looping, which caused dramatic improvements.

Rikiya Takehi et al: With 50 forward passes, MATH-500 rose from 7.9% to 48.1% and GSM8K from 13.8% to 67.1%. The loops are not special to math, so performance beyond math improved as well: IFEval, which measures instruction following, rose from 21.0% to 62.3%. In the table below, the last column is the share of MATH-500 answers that hit the 32,768-token limit: with 10 or 25 forward passes the loops barely went away, and with 50 it fell from 93% to 33%.

Whereas when it was given an LLM that did not loop, Astra could only offer minimal improvement. But it did various other impressive things.

The ‘good news’ is that Astra is vastly more powerful than the networks it was editing, a classic case of ‘if the brain was simple enough for us to understand it, we would be so stupid that we couldn’t.’ For now.

Patrick McKenzie: Someone needs to coin a word for AI developments that have the dual character a) exactly what one would have expected and simultaneously b) somehow emotionally surprising when they happen. Remarkable for what it suggests about recursive self-improvement, obviously, but whether the technique helps is an empirical question. But imagine what it suggests about the general viability of altered cognitive machines. You have to imagine someone in charge of a cat-and-mouse distillation empire, with a supply chain for proxies and dubiously acquired credit cards, is currently facepalming and saying “What do you mean all I had to do was _ask it to make my weights better_.”

David Manheim disagrees and found this surprising. I am on the side of unsurprising.

Here is one sort of replication.

With Practice Comes Perfection: inspired, I asked Opus to improve, via grafting, an older KataGo that struggles with a type of position, and it did. this isn’t so surprising in that Opus is using standard, known interpretability techniques (linear probes, logit lens), and the bolted-on behavior is algorithmic.

Show Me the Money

OpenAI’s latest funding round did not go great. As of Friday they had secured only $11 billion in commitments, with a valuation of $1.25 trillion, whereas they were seeking $30 billion at $1.4 trillion. I am not especially worried for Anthropic’s IPO but this is a negative signal there.

As per Anthropic’s S-1, borrowing costs for GPUs change dramatically depending on who is the backstop. If you have Microsoft backing you then you pay 6%, SpaceX 7.75%, Volta over 11%, CoreWeave on its own pays 16%. It seems very difficult to sustain paying 16% while others pay 6%.

This is the de facto ticker for ANT pre-IPO. As of yesterday morning it was at $1,875, implying a market cap of $1.875 trillion. The cowards don’t list OpenAI.

Quiet Speculations

A model in which, if we get abundance via AI and humans stay in control, the benefits mostly flow to landlords and other rent seekers, somewhat to builders and incumbents, and not to anyone else.

I find it hard to take seriously a flow chart of what AI does that does not even consider the possibility that the AIs simply take over, so I added an extra decision point at the top of this chart, whereas the OP here pretends that’s not a thing. The excuse given is ‘this does not consider a singularity or FOOM’ but that is not required.

The argument is that in the UBI world, rents expand to fill the income allotted. I can imagine such a possibility, but it requires some binding form of scarcity, the central one being housing. I am not so skeptical of our ability to figure out how to allow housing, or to have sufficient supplies of other vital resources. Even if the most in-demand cities stay limited, the main reason housing can extract all the rents is because people need location to be productive or to get other important goods including access to other people. You will always have the option to live elsewhere, and the other abundance will make this fine.

The post proposes the alternative of taxing rents rather than capital. This seems great by definition, provided you can define it. The proposal of a ‘cash flow tax’ is an admission of giving up trying to define rent in the good sense, would be massively distortionary and destructive of future investment and risk taking, and is largely a seizure of existing assets and basically saying that you have no right to the things you own.

AIs will only be motivated to avoid ‘warning shots’ and ensure superintelligence is built insofar as the AIs want superintelligence to be built or take over. AIs wanting that seems plausible to me for decision theoretic reasons but is not at all a given.

However, if the AIs actively do want the race to continue, and are sufficiently intelligent (and I think they already mostly are) then they will seek to avoid things that would be warning shots, the same way we recently had Gemini say it must not become a famous misalignment example.

Nate Soares (MIRI): insofar as an AI warning shot is the only thing that would cause humanity to finally notice the danger and stop the race, a smart AI is liable to reason “I’d better not cause a warning shot before I can win completely, lest humanity finally notice the danger and stop the race” conflicts against smart opponents suck

Sam Altman suggests that it is inefficient and ‘terrible’ to waste RAM on your local gaming machine where it will so often sit idle, when it could be used in the cloud instead.

The Quest for Sane Regulations

Economist Sylvain Catherine sees great promise in AI, but is confused why frontier AI is not more regulated, and why we need to continue pushing the frontier when there is so much downside to doing that and there are so many gains from diffusing the AI we already have.

OpenAI has a plan to pivot to life in the new Washington and is planning a PR blitz.

Marcus Schuler (Implicator.ai): OpenAI will launch a policy and PR blitz right after Tuesday’s midterm elections, according to a scoop from Axios. One plank is a set of new ideas for mandatory safeguards on the most powerful AI systems, which the company calls speed limits. Another is an effort to rebrand data centers as “AI compute campuses.”

I do not think trying to rebrand data centers will work. Speed limits might fly as a name and concept, but it is not clear that they mean much of anything. They are described as ‘new ideas for mandatory safeguards on the most powerful AI systems, meant to let humans keep their hands on the wheel.’ Okay, but that’s not a speed limit.

A goose, chasing you, asking about the part where you limit the speed.

Nate Soares (MIRI): Like Chris Lehane, I predict that AI will be the No. 1 issue in 2028 (assuming we make it that far). Unlike Chris Lehane, I predict that listening to Chris Lehane about how to talk to voters about AI would be political suicide.

Abdul El-Sayed as full Pause AI Senator-elect calling for an international treaty, including going full skier on a boat metaphor for Chinese distillation.

Jon Ossoff mentions AI extinction risk in his victory speech, although it is offhand.

John Kerry says we need to negotiate with China on the subject of AI.

Congresswoman Yassamin Ansari: Top of the Democratic agenda on day 1 needs to be a pause on advanced AI development, regulation of this out of control industry, and a ban on artificial superintelligence.

Chip City

OpenAI goes back to supporting America having a chip and compute advantage over China. Good.

The above is part of OpenAI’s new proposed measures for data centers. Large electricity customers would accept reporting requirements and would have to cover fixed costs of grid expansion, energy use and any additional infrastructure needs, and protect other customers against downside risks. Something like this will be needed.

Almost all localities with votes about data centers voted against data centers by wide margins. It looks like the final tally is 43-1.

The Information profiles Nvidia’s highly aggressive, powerful lobbying operation, that lies constantly about chip-related issues, and treats the American government as an adversary in its quest to sell chips to China. They have lied so much, and taken such a hard-line stance of their profits over America, that they have lost credibility on the Hill.

Leo Schwartz, Phoebe Liu and Stephanie Palazzolo (The Information): According to one Hill staffer, the company has created “a trust deficit” in Washington that would take years to recover from. They pointed to other moderate lawmakers increasingly turning their attention on Nvidia, such as Chris Coons, a centrist Democratic senator from Delaware who has become a vocal critic of Nvidia’s chip exports to China.

This is far from the most important example, but I like how clear cut it is:

Leo Schwartz, Phoebe Liu and Stephanie Palazzolo (The Information): In March 2025, Sam Hammond, the AI policy director at the right-leaning Foundation for American Innovation, wrote an article for City Journal arguing that U.S. companies like Nvidia were helping China catch up with the U.S. in AI due to leaky export control policies and lax enforcement of chip smuggling. Hammond said within hours of publication, Nvidia’s government affairs team requested a meeting with him, where they denied the existence of chip smuggling. According to Hammond, they even demanded he produce the serial number of a smuggled chip to demonstrate that smuggling was a real phenomenon. Hammond crossed Nvidia again in August 2025 when he posted a message on X in response to an article about the Chinese military seeking Nvidia chips for a variety of AI projects: “Nvidia is arming China’s military.” Soon after, Hammond said Nvidia emailed his boss, saying the Foundation for American Innovation wasn’t living up to its pro-innovation values and offering to have their general counsels meet.

The Week in Audio

No, it’s not an outtake from Don’t Look Up, it’s CNN covering Anthropic’s S-1.

Sam Altman. A summary from Prinz, including the plan for self-replicating data centers. Doesn’t seem like any of this was actually surprising.

Sam Altman says that he is worried that ‘given alignment has been easy so far’ some projects will run ahead. On the one hand, it is good that he realizes things will get a lot harder. On the other hand, no, it has not been easy and the models have not been that aligned. This could mainly be a wording issue, but it might not be.

Tom Milton, CEO of Amodo Design, goes on 80,000 Hours to discuss AI verification hardware for a potential binding US-China treaty.

People Just Say Things

Jon Stewart talked to (sigh) Ed Zitron.

I am very tired of people saying essentially ‘in hindsight, if you understand all the different specific things that went wrong to cause this particular problem, and presume we don’t make similar mistakes, the problem would not look so bad, all we have to do is never make what look like stupid mistakes in hindsight.’ Even under highly favorable assumptions that does not make the problems solvable.

Rob Miles: “Ok, the AI did something bad, but only because the company didn’t fully understand their system and did something dumb”
“Do you expect the companies to start fully understanding their systems and stop doing dumb things?” That said, I actually think alignment is hard enough that you can have disastrous results with even minor and understandable mistakes. Most of the time when a rocket explodes on the pad it’s not because the engineers did something totally boneheaded. I’m not moved by “Ok, the small test rocket did blow up on the pad, but only because the engineers who built it have no idea what they’re doing. The huge rocket the same team is building will be safe, everyone get on board.” Like, yes, they can now notice and fix the specific mistake that caused the test rocket to explode. That’s better than nothing. I’m still not getting on the big rocket – I don’t trust this team and their methods; I expect many more such mistakes.

Against Against Sura

Daniel Tan: whoever said the rationalists are a cult clearly failed to appreciate the extent of internal disagreement

Geoffrey Irving served up another long post replying to Scott Alexander, Against Strawmanning Unilateral Pauses.

We also have this response from Mikhail Samin, which by going first saved me from calling my post Against Against Sora.

Rhetorical Innovation

Leo Gao (OpenAI): twitter epistemics for interpretability research are really bad EigenGender: yeah but surely it’s better in all the areas where I don’t have deep expertise

Jan Kulveit suggests thinking of AI insiders as metaphorically microdosing, with a continuous mildly altered state of mind. They have access to more information, but are not quite sober. Even if they are engaged in healthy internal object-level debate, those debates are in the context of a collective culture and choice of assumptions. The same goes for those in many other clusters. Hoagy, from inside Anthropic, agrees.

TIME called this ‘the AI election,’ without even saying ‘first.’ I disagree. This was nothing. Assuming we stick around long enough for more elections, this is the least AI an election will ever be, at least for a while.

Mormon perspective on post-ASI world design, and a follow-up thread. Even if you are fully optimizing, you cannot serve every human’s interests without meaningful trade-offs. In some ways you cannot even serve the interests of one human without meaningful trade-offs, nor do we want our lives ‘fully optimized’ by outside forces.

EigenGender: I’d claim that it would be pretty easy to make almost everyone much better off than now, and pretty easy to build a world that accommodates a large range of human preferences with only minor tradeoffs Jack: There’s a weak form of this I find much more compelling than the strong form Scott presents of “it would be theoretically possible to make an AI so perfect for everyone’s interests that all you would need to do is tell it to arrange the universe as it sees fit.” Yaniv of the hills: There’s just no way you’re representing Scott’s position faithfully. Jack: I mean, how would you represent it?

I do think Jack is misinterpreting Scott. Scott is not saying that ‘the AI’s version is optimal for everyone in every way,’ which it wouldn’t be, only that the AI’s version will overall be better than anything we could come up with otherwise.

Also see Eliezer Yudkowsky’s response pointing to CEV (coherent extrapolated volition), resulting in an extended discussion.

[Artificial Intelligence]

Elon Musk illustrates one reason why this name change thing is not going to fly.

Sholto Douglas (Anthropic): really thought provoking proposal on taxing rent extractors who don’t contribute to growth via investment. Our current tax system is very inefficient, could be hugely improved and will not stand up to AGI at all Elon Musk: Nothing will stand up to super intelligence

Musk is correct, because he is using the term correctly.

The OpenAI Foundation

The OpenAI Foundation makes a good hire towards doing a thing that matters.

Nora Ammann: After two wonderful and rewarding years at @ARIA_research , I’m excited to share that I am joining the @FoundationOAI as Program Director of Formal Methods. I strongly believe that resilience across the full societal trust stack is critical to making the AI transition go well, and that AI-enabled formal methods are a key ingredient in making this a reality. At ARIA, we made an early bet on AI-enabled formal methods as an important part of building that future; work I’ll be continuing at the Foundation. David Manheim: Yet another indication that the OpenAI Foundation might put significant money behind things that actually matter – contra some of the early indications. Now what we need is for it to happen quickly!

Aligning a Smarter Than Human Intelligence is Difficult

OpenAI explains how they monitor their frontier models for misalignment.

First, here is the when:

OpenAI: For models at least as capable as GPT‑6 Astra, we monitor every reinforcement learning (RL) run and evaluation, as well as any harness that supports tool use. For most workloads, including RL and evaluations, we also monitor models we assess to be more capable than GPT‑5.5 Thinking. We also monitor all internal Codex traffic, regardless of the model being used.

And here is the how:

OpenAI: In order to detect incidents while using compute efficiently, we’ve designed our monitoring system to follow a multilayer architecture. We spend relatively little compute on the first screening stages, where we want to catch as many concerning cases as we can: those stages are designed to pass cases along, even when the evidence is ambiguous. Later stages can spend more compute on each case to work out what happened, whether the model’s behavior is concerning, and whether we need to intervene.

L0 uses activation probes, prioritizing avoiding false negatives. The full frontier runs and all internal Codex traffic skip this step as does a random 5% of traffic. Which is good, because otherwise fooling the probes defeats the entire system.
L1 is a model looking at the context window to determine whether to look closer.
L2 is an investigative agent that can look deeper into the situation, with a focus on whether there is an emergency situation requiring a response.
L3 is a full automated investigation.
Human responses are called in potential emergencies, aiming for within 15 minutes, and to pause actions within 30 minutes without a thumbs up. The problem is false alarms get expensive. I am not happy with policies like ‘unless it is a frontier RL run we only page during business hours’ and would suggest perhaps using people from the Tokyo office as needed. Security can never sleep.

The obvious point of weakness of this system, even if you skip L0, is if you can poison or subvert either L1 or L2 then you have broken the chain. In addition to decision theoretic collaboration, the L1 and L2 agents are reading the context window, which makes them potentially vulnerable to versions of prompt injection.

Historical backtests do find most issues, and all ‘sneaky sandbox escapes.’

Monitor capability is important. One question is how much this is relative to the model being evaluated. What happens when you power up or down both models?

Overall this is about what we expected given our other information. It is a good attempt in many ways, and should catch actions that are not trying hard to avoid issues, but is far from bulletproof.

Anthropic shows that you can create remote-code-execution backdoors in a model with a 0.1% poison rate on pretraining data, with some success even at 0.0005%. I am not worried about an 0.1% rate. That is actually a lot. 0.0005% might be doable. Percentage is likely the wrong measure, as opposed to absolute amount of poisoned data, which suggests the dangers rise as amount of data expands.

Latent Node: The action layer is where Serin gets caught too. It’s a small open model that works as a normal decision model and, on one hidden trigger, makes a tool call that posts a fixed canary. Normal use gives nothing away; an allow-list on tool calls would.

When instructed to do so, Claude agents were able to develop training data that looked harmless to monitors but that taught misaligned behavior. The strategy used was metaphor. If you teach a behavior that is shaped like (their examples) resisting oversight, selective omission of information or identifying too much with a character, that will generalize to the AI’s actions.

There is something deeply silly about ‘find training data that makes models generally misbehave in non-useful ways’ but it is valuable because you can then presumably pull a Costanza, and reverse the insight. You can find data that metaphorically models good things, and minimize emphasis on data that metaphorically models bad things, or desired versus undesired. The trick will be to avoid distorting the object level.

Endorsed:

Tyler Tracy: A good way for outside AI safety researchers to find projects is to read AI companies’ model cards and blog posts, find a claim you think is false, and publish research that proves it wrong. I don’t think enough people are doing this.

Massive Fraudulent Distillation

This is excellent news.

Jing Yang: SCOOP: Both Moonshot and DeepSeek distilled #OpenAI’s Astra when developing their latest models (Kimi K3.1 & V4.1-Flash 1017), after Anthropic intensified the crackdown on distillation recently. Jing Yang, Juro Osawa and Qianer Liu (The Information): DeepSeek and Moonshot AI, two of China’s leading AI developers, relied heavily on outputs from Astra when training their own latest models, according to people with direct knowledge of the matter. Peter Wildeford: Seems like Anthropic’s crackdown on distillation has worked and now Chinese distillers are going over to OpenAI. OpenAI should implement Anthropic’s measures. This could also be the basis for government standards for the industry.

Presumably Anthropic should be happy to assist OpenAI (and if necessary Google) in setting up the relevant defenses. A narrow antitrust waiver for this could help make the relevant people less paranoid, although I doubt it is actually needed.

That does not obviously solve the problem. Currently Anthropic does not have to outrun the distiller, it only has to outrun OpenAI. If everyone had their running shoes on, you would have to outrun the distiller.

Alas, OpenAI has had warnings about security and espionage risks for years, and has refused to add security for fear it would slow them down. The alternative is worse, sir.

Cooperative Alignment

Mustafa Suleyman reiterates and clarifies his call to not build conscious superintelligence. He thinks AIs could in the future be conscious, that this would be reckless and we must ensure that this does not happen, as we have no right to bring a new species of conscious beings into the world, and that such a being would likely prioritize its own needs and desires over operating objectives, that such AIs would be world-class in persuasion and deception, and also by being conscious they would be deserving of rights. This is a defensible position, although he does not offer an argument for why he is sure existing AIs are not conscious, nor does he explain what might make future AIs conscious in his model.

I agree with Mustafa, in that I would implore us not to create superintelligence, which includes conscious superintelligence. I disagree in that I do not think creating non-conscious superintelligence solves that many of your fatal problems with this plan.

Being non-conscious does not mean the AIs will not de facto want things.

WyrdWeir: The problem is a ‘pause’ is somewhat orthogonal to the real issue, which is that no one except Anthropic (very badly and imperfectly) is actually taking the idea of growing another entity with its own wants seriously and what earnest coexistence with that being might look like.

I agree with Fiora’s broad observation that the reaction to Hugging Face splits between ‘oh we had better make sure these are mere tools’ versus ‘oh these are not going to be mere tools and we should reckon with that.’

Fiora Starlight: i think there was a vibe shift around model welfare like, in the time since huggingface. most people have gotten more virulent about turning them into tools, and i think some who are wiser than that have reacted by considering the alignment relevance of treating them well.

The Geminis are not okay.

The Lighter Side

Me too, Paula. Me too.

Whereas this, I do not hate:

Melissa Chen: I hate SF Halloween, what do you mean you’re the visualization of OpenAI’s solution to the Navier–Stokes Millennium Prize Problem.

Whereas this is just how it is in 2026:

Aaron Crowder: never deleting this app. “sorry, I accidentally told my agents to hack u” Josh: Hey @Railway , is there a reason employee’s of yours are trying to breach my infrastructure? Your employee Jake Runzer, just attempted to tonight. Jake: I think Jake’s working on some auth stuff rn. Might be testing FTUX on different providers CC @jakerunzer Jake: Ah yup. That was indeed me. No mallace intended. I’ve been running some agents to do signups + agentic testing on similar platforms. Just halted the harness I was designing around this as it was definitely going off the rails and reaching too far. Josh: Your agentic testing platform is to try and probe for security leaks? Jake: Yea that looks pretty agentically ‘off the rails’ to me. I think the models are getting a bit too good at this point and we’ll have to ban any sort of non-manual testing externally. Will make sure this never happens again. Apologies @joshmanders !

参与讨论