AI #190: Scion of the Times

The amount of code being written at Anthropic continues to rise exponentially. It is being managed by Scion, their internal replacement for GitHub that is better suited to the scale and style of the operation, which they created with a small team in five months.

This week also saw the release of two supposed frontier model models.

Google has finally given us Gemini 4 Argon, along with a robust safety report. We do not yet know if Argon lives up to its benchmarks or internal hype. I plan to write up the safety report next week, time permitting. If and only if it is good enough or discussed enough we will also have a full report on capabilities.

Meta gave us Muse Wonder, which like Argon has some excellent benchmarks, although they were presented in a way that looked rather sus. There are some reports that it is very good, but again it is too early to know. I will issue a full report if and only if Muse Wonder has capabilities or discussion that seem worthy.

To help me decide and to let people have fun, I have markets on whether I will issue the reports on Argon and Wonder, respectfully.

OpenAI caught an employee planting a bug. How many more spies are out there?

A group used ChatGPT to create a full-chain exploit of Chrome and claimed a $250,000 bounty. Luckily for us it was the good guys this time.

Like most weeks, I covered a lot more. It was only a little over a week ago that OpenAI dumped solutions to hundreds of math problems. Then came news that OpenAI fired three safety researchers, without providing us a good explanation. Once again we went over the fact that arguments about AI consciousness remain highly motivated and at best overconfident. Finally, I emphasized that the safety culture at the labs is broken, which is one of the most important facts about the world.

Today I am flying out to another conference at Lighthaven. I probably posted this from the airport and it won’t include any events from last night, tomorrow’s post will be the monthly and Monday may be from the non-AI queue. This coming week will then be another case of me scrambling to catch up, especially with the model reviews still potentially looming.

Table of Contents

1. Language Models Offer Mundane Utility. Prompting is easy now.
2. Language Models Don’t Offer Mundane Utility. Writing still needs work.
3. Huh, Upgrades. Muse Wonder and Gemini 4 Argon.
4. On Your Marks. Are you sure the models got that question wrong?
5. Choose Your Fighter. Theo is a fan of Haiku 5.5.
6. Get My Agent On The Line. Agent retention rates are okay but not great.
7. Deepfaketown and Botpocalypse Soon. Literary world keeps getting exposed.
8. Cyber Lack of Security. A full-chain exploit of Chrome found in six days.
9. Spy Versus Spy. OpenAI employee caught planting a physical bug.
10. The Internet of AI Things. AIs might soon haunt various devices.
11. A Young Lady’s Illustrated Primer. AI is changing motherhood.
12. They Took Our Jobs. Epic Games to have an epic lack of human-written code.
13. Introducing. Microsoft-Decision-1 is based on… Qwen 3.5-9B? Really?
14. In Other AI News. The State of AI report isn’t what it used to be.
15. Massive Fraudulent Distillation. The fight continues.
16. Show Me the Money. Anthropic Q3 revenue of $16.8b. Nvidia to buy Reflection.
17. Quiet Speculations. There is accounting for taste, given enough data.
18. Quickly, There’s No Time. Timelines to RSI keep shrinking, moving into [AI] 2027.
19. The Quest for Sane Regulations. UK AISI regains access, CAISSI to White House.
20. Chip City. GPUs can be cheaper to buy then to rent by a large margin.
21. The Week in Audio. Soares, Lovely, Patel, Wang.
22. People Just Say Things.
23. The Virtue of Silence. [CENSORED].
24. Rhetorical Innovation. The four classes of AI user expertise levels.
25. [Artificial Intelligence]. We have met the enemy, and he is everywhere.
26. Aligning a Smarter Than Human Intelligence is Difficult. AI as insider risk.
27. Backdoor Man. Backdoors, and other things, are easy to insert and hard to remove.
28. Cooperative Alignment. Decision theory gives us some arguments for hope.
29. People Are Worried About AI Killing Everyone. The AI will be smarter than you.
30. Other People Are Not As Worried About AI Killing Everyone. They don’t know.
31. The Lighter Side. You know what you did.

Language Models Offer Mundane Utility

Prompting is easy now. As Boris Cherny says, tell the AI what you want to do, how to verify that it did it, and how much effort to spend, and you’re good. I’d add ‘and anything it needs to know.’ Presumably that was implied. Boris prompts on Slack, which means Claude already has a lot of context, enabling shorter prompts.

Do you trust Grok to go use your credit card without pausing for approval?

Elon Musk: To buy stuff, you can just take a picture of your credit card, drop it in the chat and Grok @Bot will scour the Internet for the best deal on the item you want and order it. [Community note: No that’s not how you upload your CC information.]

The correct maximally allowed error rate on such actions is not zero. It also is not high, and if you are using this blind you likely will have no idea if it is getting the best price or the best item. Then again, did you know before?

How easy is prompting? Pliny shuffled this off to Opus 5.5 with a $200 subscription:

“can you wraite a promtp for a cron job that can run every day thajt will try to find solutions to unsolved ciphers or transltions or math problems or theormes or things of that nature!! every day a new attempt as hard as possible all day without giving ujp and with maximum ceraitvityy, resourcesleflses, andpersticense! dont worry about token spend! and dont forget a bit all caps lots of emojis celebration to notify the uers if we get any intersting or sucsfful confirmd reulsts!!”​ … day 1, the legion picked its own target: J.-F. Bouchaudy’s cipher-machine challenges. these are simulations of real vintage machines (Hebern rotor machines, the Kryha, Lorenz, SIGABA), with problems that had been sitting on his solver tables with no solver for months or years. on the morning of day 2, i was half asleep when my laptop suddenly says, in a robot voice:
“Breakthrough. The Daily Oracle has a confirmed solution.” i was startled. then the ALL CAPS emoji flood hits my screen. Hebern 5-rotor Problem 8 then Kryha v2 Problem 5 i emailed the author, Jean-François Bouchaudy, a French cryptography enthusiast who builds these machine simulators and challenges as a labor of love (he told me he started out programming on punch cards ). he checked both solutions and put my name on his solver tables as the first and only solver

This is the correct version of ‘do a breakthrough, make no mistakes.’ It’s that easy.

A thought experiment from Terence Tao:

It seems ethical, in a sane world, to allow volunteers to participate in such trials. If the cocktail passes a Phase 3 trial, then we should conclude that it works. No, I absolutely would not, if I had cancer, demand that there was a human who ‘understood how it works,’ and if the FDA required that I would yell about it a lot. Yes, of course you would also want to model tail risks that might not be showing up and take those risks into account, so I wouldn’t use this lightly, but if the cancer is lethal and otherwise untreatable then why not?

Obviously you would also have to worry about some form of active misalignment, and that this was part of some sort of takeover or similar attempt, at that point, but that is a very different concern than the ones Tao is presumably considering.

Tao instead has the next point be ‘the AI hacked the FDA and the drug both does not work and kills people’ but even if the AI is being malicious for the lulz no one going to do this by hacking the FDA, in the sense that the Phase 3 trial is a physical thing with many artifacts in the world, that is then examined by humans? So we would just notice. And if we didn’t, we’d notice pretty quickly after that. The only real risk is long-latency-only harm, which is already true today and has limited downside if you are treating otherwise deadly cancers.

Talk and write to people whose expertise lies elsewhere:

Dean W. Ball (OpenAI): By the way, in my experience contractors of all sorts have adopted AI for text-based communications and it has substantially improved coordination on home renovation projects

Language Models Don’t Offer Mundane Utility

Remember that, no matter how great Opus 5.5 is (and it’s pretty great), how many contributions you make or lines of code you write is not a good measure of productivity.

Ado (Anthropic): The weekend after Opus 5.5 launched I had my most productive coding day of the year (so far).

AI writing now once again lags behind AI math.

Matthew Yglesias: AI cracking unsolvable math problems while being kinda mid at writing is a much-needed W for humanities majors after a rough 21st century. Paul Graham: Apparently it’s mostly just because they haven’t made much of an effort to train it to write well yet. Matthew Yglesias: I’ve been reliably informed my work isn’t economically valuable enough to be worth investing time on automating. But still! Miles Brundage: Mostly just cuz there are like 2 ppl working on it though Matthew Yglesias: Let us have this

Better AI writing would be highly economically valuable, so my guess is this is mostly not ‘we forgot to put 10 or even 50 guys on it instead of 2.’ You’d have to work a lot harder than that. The alternative hypothesis is ‘you may not like it but this is what peak performance looks like,’ that the people love their Claudeslop and it works, and we would react inherently negatively to AI writing no matter what it sounds like. Sure, it would be better if for those who care it could vary more and be higher Quality, but is that worth the price?

I don’t know. I can see it both ways. My guess is the Claudeslop has short term benefits, but continued use of it accumulates long term costs and fatigue. In general, when faced with such choices, modern capitalism is often myopic and gets in trouble.

Who gets the blame if AI fails? Opinions differ greatly.

It depends on the situation. There is no universal answer.

Huh, Upgrades

Meta’s Muse Wonder exists. It sports excellent benchmarks, with the ‘Full’ version scoring 56 on Artificial Analysis Intelligence, two behind Opus 5.5. However, Muse Spark 1.3 had the same rating there as GPT-6 Sol and was only one point behind Astra and Fable 5.1, and I don’t know of anyone who thinks Muse Spark 1.3 could hang with those other three offerings. Various signs suggest benchmaxxing, including how they chose to display their other benchmarks.

As always, reserve initial judgment, do not read too much into benchmarks and if you think it is worthwhile run your own experiments, including with matching prompts across models, and report back. Due to my trip, if Wonder proves worthy of extended coverage look for that around Wednesday next week.

Gemini 4 Argon is live for Pro and Ultra users, and in the AI Overlay and hopefully by now everywhere Google AI products are served. We now wait to see if it lives up to its claims and hype. If it does, we will have full coverage, again some time later next week.

The tentative plan is to do a full read of their 200+ page safety and security report, the same way I do with Anthropic models, and go from there. I can’t do that every Google and Anthropic release, it would be too much, but this is a big jump for Google.

Grok Voice 3.0 exists.

Claude Dashboards and Claude Motion are now in beta. You can now create various charts and also short animations, with a slightly easier interface than before.

Claude now offers the type “multiagent_20261001”. This does what it sounds like.

Google has a new internal model, for now named Gemini 4 Carbon, available inside their internal coding platform Jetski, with claims it is very good.

Hugh Langley: An internal document shows that Google has tested a series of Gemini 4 models, internally named Argon, Barium, and Carbon.

I am highly disappointed if the reason for Argon was alphabetical. Lamest secret coding system ever. If you are going to do elements, be an actual nerd and use the periodic table. I should call it Gemini 4A in protest, but I probably won’t.

On Your Marks

Opus 5.5 is the first model to successfully clock Joe Weisenthal identity based on a few pages of an unfinished novel from a few years ago.

Are you sure the model got the eval question wrong?

Alex Tabarrok: We are now at “the fuck” levels. Nirmalya Kajuri: “Our initial evaluation suggested that GPT-6-Astra’s score was in the late eighties followed closely by Opus 5.5. But, as we started to investigate the mistakes that the models might have made, the models, especially Astra, insisted that they were right and that it was the expert-supplied answers that were erroneous. When we went back to the experts it turned out, in multiple instances, that that the experts found some subtle errors buried deep inside their calculations. As they revised their answers, Astra’s score started to inch up and it finally ended up with 98% (49/50).”

Last week math. Soon, physics. Then, presumably, chemistry.

I agree with Charbel-Raphael that the marginal value of AI ‘safety’ evals is low. Internal evals still do matter, and it is good to keep tabs, but at this point the good ‘safety’ evals for warning people are when something unsafe happens in real life.

Choose Your Fighter

Theo thinks that Haiku 5.5 means Anthropic dominates both large and small models.

Haiku gets a 43 on Artificial Analysis versus 38 for Luna, but is not on their Pareto-optimal cost competitiveness curve on benchmarks alone due to higher token use. They had it on max, so this could be an easily fixable skill issue.

Get My Agent On The Line

JP Morgan Chase blocks outside AI agents from directly accessing accounts. Good. You can argue Amazon is making a mistake but the reasons for Chase are obvious.

I haven’t tried Muse, but my experience with Dot is similarly that I’ve struggled to find continuous tasks for it to do.

Deedy: Meta Muse’s retention is out, and it’s almost as good as ChatGPT! It’s higher than ChatGPT at D7 (>30%) and just lower at D14 (22%). Not quite in the elite D14 tier of WA (67%) YT (59%) Insta (54%) or TikTok/Snap (37%), though. Muse is one of the highest retaining AI products! Sully: the notifications from muse are killing it. sure the first few days are nice. but now it feels like spam. 99% of my use cases now are “email this, contact support, cancel my sub” which is maaybe me using it 1-3 times a week useful, but not nearly as much as other ai tools

Muse, as I understand it, accomplishes these app openings through integration with social media and push notifications. App-open retention seems like the wrong metric here. Look at the ‘keeps [X] out of 100’ numbers, and you see a very different order.

This roast of Muse from Danielle Fong shows some of what she calls ‘growing pains,’ also known as stupid mistakes. What stands out more than that is how Muse is talking. Every word is painful levels of AI-slop talk. I would uninstall for that alone.

Deepfaketown and Botpocalypse Soon

I do not blame OpenAI on this one:

unusual_whales: BREAKING: Iranian operatives used OpenAI models and fake personas to help plant news articles critical of the US war against Iran in multiple US media outlets, OpenAI has said.

Yes, OpenAI, Anthropic and other labs should try to prevent this, but you can only do so much. It is on the media outlets to be robust to this sort of play. You should not be able to ‘plant news articles’ by virtue of the ability to write AI slop and generate fake personas.

It would be a lot easier to spot and prevent such frauds, and also other types, if we did not have a bunch of ‘disparate impact’ rules around how we can spot and prevent fraud. Some regressions are de facto illegal. Those rules do not magically stop applying if you involve an LLM. LLMs are in large part regression and correlation machines. We will have to make a choice.

Either we allow LLM intermediation to render approximations of such calculations legal at Google-level scale, or we don’t. If we don’t, and we start auditing to make sure that banned correlations did not creep in, we are going to have a bad time.

The Quartz website is now mainly one AI-assisted reporter, Cris Tolomia, writing 50+ stories and about 20,000 words a day.

Wired reports that book publishers are quietly using more AI across the board for emails, publicity and back-cover copy, cover art, marketing materials and so on, and that there are far more problems with AI-written books than are publicly known, including by well-known authors. The headline complaints from employees here are about ‘cost of software’ and environmental impact, which are the wrong objections.

Adam Morgan: Meanwhile, earlier this year, literary agents began suspecting specific Big Five editors were using AI to write rejection letters.

Rejection letters seem like a good place to sometimes use AI. If you notice your rejection letter was written by AI, well, that is another rejection. Use that information.

Using AI for back-cover copy only makes sense if it creates more effective copy. It might do that. This is high-value copy that is important to ‘get right’ but get right means ‘turn into sales.’ The people like Claudeslop, at least for now.

Adam Morgan: All of the agents I spoke with expressed fears that publishers might paste text from their authors’ manuscripts into the LLMs in order to generate things like rejection letters and publicity descriptions. “We’ve added AI clauses to our contracts saying ‘this material can’t be run through an LLM,’” says one agent, who did not want to be named to avoid professional repercussions

If you have those clauses, you are going to get rejected more often, because you are making the editor work harder and less effectively. I would advise against it.

Indeed, if you tell me I can’t run your text through AI, guess who I think wrote it.

The latest literary prizewinner to be written by AI, as per both Pangram and anyone who pays attention for several sentences, is The Blue Fighter, which won the (formerly?) prestigious Caine Prize for African Writing. Reporter Vauhini Vara finds the story bad, and the samples quoted are clearly AI slop, but judges disagreed.

Vauhini Vara: Yet Wakatama forcefully rejected the notion that an AI detector could, or should, be used to gauge AI involvement. “My stance, as the person who leads the prize, is that if there is such an accusation, I want proof that does not come from a machine.”

It is remarkable how much the literature world is determined not to believe in Pangram, often for the dumbest of reasons (aka water and power).

The burden of proof needs to go somewhere. If ‘an accusation’ of any kind shifts it onto the author, whether or not it is backed by evidence, that’s not great. A much better rule would be that AI detectors put the burden on the author to show they wrote it, whereas without the detectors you need other very strong evidence that the author did not write it.

Vauhini Vara: Wakatama remarked on details in Kundizeza’s story that strike her as so typically Zimbabwean that a Zimbabwean must have written them: the shame of moving from a nice suburb to a less privileged township; how the mother’s initial forbearance with the father gives way to taking matters into her own hands. Wakatama also perceives multiple layers of meaning. She doesn’t think AI models are capable of that kind of cultural or literary nuance. But then, she acknowledged that she doesn’t use AI or AI detectors. “My first response when somebody asks me to use AI is that I think about water and energy,” she said—that is, AI companies’ excessive use of both. “That honestly overwhelms me, and I don’t want to.” … I’ve covered AI for almost a decade, and have written fiction for even longer; in short, I’ve had plenty of exposure to both literature and chatbot output. I know from experience that, for example, AI models aren’t terrible at picking up on cultural details.

Those details do not sound like things that an AI would have any trouble finding. They sound, if anything, like cliches, and not ones unique to Africa or Zimbabwe. Multiple layers of meaning are sometimes rich and intentional, but also you can pretty much always find them if you look hard enough. I am constantly finding ‘layers of meaning’ in media that were very obviously unintended by the authors, even subconsciously.

Cyber Lack of Security

Anthropic is introducing the long-term Anthropic Cyber Mission to secure systems, starting with critical infrastructure with the Critical Infrastructure Defense Program, and open-source software, where they are launching OSS Scanner.

Many of those who typically sign such letters, such as Bengio, Hinton and Russell but also Rivest, signed an open letter calling on cyber-capable models to be air gapped by default. If you need outside access, conditions for that would be set by international bodies backed by security experts. This is the kind of thing a sane world would do.

After $40,000 in compute from GPT-5.6 Sol, XBow claims the second $250,000 bounty for a full-chain exploit of Chrome. Unlike OpenAI, Google pays real bounties, and this still should have been ten times larger.

Oege de Moor: For the second time in Google Chrome’s history, Google awarded a $250,000 bounty for a full-chain exploit… to @XBOW . Here’s the full story: At an all-hands at the start of 2026 in Malta, I set the team a crazy challenge. A full-chain exploit of Chrome by the annual Black Hat cybersecurity conference in August. I believed there were few better tests of whether @XBOW could move beyond web applications to desktop software, servers and devices. Chrome is one of the most hardened pieces of software on the planet, used by more than half the world’s population. A full-chain exploit lets an attacker take total control of a machine, where they can read every email, password and bank login the user types. You don’t need to download anything or click a phishing link. Visiting the wrong site is enough. So @moyix , Andres Luksenberg, Baul Lee, and Willem Pinckaers took on the challenge along with a fifth teammate: OpenAI’s GPT-5.6 Sol. They built a chain they thought would win, but Google told us they already knew one of the bugs. We were already in Las Vegas for Black Hat and DEF CON, and the team didn’t want to go home without the win. So Brendan and the agents built a new exploit in just 6 days. We submitted it to Google and waited. The news came through that we’d done what should’ve been impossible. A full-chain bounty of $250,000, the first awarded under Google’s 2026 rules. It took just 6 days and $40,000 of compute to construct an exploit that affects a billion users of Chrome on Windows. An intelligence-agency veteran told me his team used to spend 6 months on this kind of vulnerability. My takeaway: the cost of finding truly critical vulnerabilities is collapsing, for attackers and defenders alike. Restricting powerful models is not the answer; GPT-5.6 Sol is already public. But defenders have one advantage attackers never will: they can test their own code first. Assume your software is exploitable. Whoever is hunting your bugs has the same building blocks we did. This is cause for urgency but not panic. … P.S. Restart Chrome whenever the Update button appears!

Yeah, okay, let’s all restart Chrome whenever the Update button appears.

Yes, defenders get to test their code first. But here we have essentially the most widely used, most vital software on the planet that needs to not be exploitable. XBOW exploited it, Google said they already knew about (but had not fixed) one of the bugs, and then XBOW turned around and broke Chrome again in six days.

That’s with GPT-5.6 Sol, which is now two generations behind. GPT-6.1 Sol is a lot better at this sort of thing, and Opus 5.5 is even better than Sol.

I’m sure XBOW’s engineers are somewhat cracked, but they’re not uniquely cracked. There are tons of people who could have done this. We are fortunate the one that did it was wearing a white hat this time.

The world seems like it is playing with fire. We have been saved, so far, almost entirely by the fact that People Don’t Do Things. That won’t hold forever.

A cyberattack hits supermarket refrigeration systems across France. I saw no word on whether or how AI was involved.

An investigation in Sweden finds widespread internet access to industrial control systems, including some without password protection.

AI creates jobs:

Spy Versus Spy

OpenAI has caught an employee, a Saudi national named Alfozan, planting a physical recording device.

dave kasten: Old enough to remember when earlier this week people criticized Jacob Coxon for saying that there are foreign assets inside the AI companies OpenAI Newsroom: We discovered a sophisticated recording device that was hidden inside one of our offices. We used our models to understand the device and where it had come from, identifying that a specific employee had planted it a few days earlier. We referred the matter to the FBI, and we’re grateful to the agents and prosecutors who moved quickly to make an arrest. The FBI’s investigation is ongoing and we will continue to support their work. At this time, we have no indications of this individual inappropriately accessing or exfiltrating model weights, source code, or customer data. The employee was terminated. We take protecting our technology, data, and people seriously, and as a matter of national importance. Our security program is designed to protect against a wide range of threat actors, from insider risks to cybercriminals to nation states, and this case is an example of how AI can help address these sophisticated security threats. Sophia Cai (Politico): OpenAI’s offices were bugged by a 35-yr-old Saudi national who used to work at Meta, which begs the question – was he spying at Meta too?

The device was a Seeed XIAO ESP32-S3 Sense hobbyist development board (thumb-sized, with mic/camera/microSD) retailing for ~$14 and customized with basic encryption, per the federal complaint.

The suspect had a one-way ticket booked back home to Saudi Arabia.

Also at OpenAI, an employee who left for Apple took files about unreleased device.

Zack Korman: OpenAI just revealed that an employee stole over a hundred highly confidential files months ago, and they didn’t notice until now. They only found it while looking for evidence in the Apple lawsuit. Not a good look for insider threat detection.

The Internet of AI Things

Anthropic’s William Zhou and Daniel Freeman report that small AI models may be months away from being capable enough to self-replicate on commercial internet-of-things (IoT) devices, like smart ovens and off-brand Android TV boxes.

Anthropic: In our evaluations, a 9B model got inference running on 8 of the 9 hardware types we tested. … We run Gemma4-26B-A4B, distributed over 15 inexpensive off-brand Android TV boxes (~$55 each) that we bought and networked together, at 0.39 tokens per second. Jeffrey Ladish: A lot of devices are going to get hacked before they’re patched. Botnets aren’t new, but self-replicating AI malware that can adapt to new circumstances sure is.

One suggestion is that such a system could have a tool that calls out to a smarter model elsewhere. They note the similarity to pre-AI botnets, and expect the severity to be substantially higher. I agree, and would extend that to impact on other things.

There are a lot of unsecured devices out there, or devices that are secured only by things that would be trivial for such an AI system to break. The future is going to get weird in various ways, although that future is likely to be unevenly distributed and not arrive at its first possible moment.

Will this happen? John Pressman has a Manifold Market that expires in July 2027, with the threshold of at least 1,000 devices for a week, or a consensus that the worm cannot be contained. As I write this it is trading at 41%. This is a cool market, of the type that I would use more often if I didn’t have to deal with a bunch of hypothetical corner case questions.

A Young Lady’s Illustrated Primer

Abby McCloskey claims in Bloomberg that AI is Changing Motherhood for the Worse, warning that it is not only about ‘risks’ to children but mothers are impacted too. The first argument seems to be that AI is isolating, because you ask AI things instead of your friends? That’s an interesting way to treat a new information source. The second argument is now you have to ‘monitor’ AI use, which does not seem to me to be different in kind or burden from monitoring screens and phones and social media and all that. And that’s all Abby’s got.

They Took Our Jobs

The mathematicians are waking up to their jobs being taken away. Be kind.

Mathieu: Fields medallist Hugo Duminil-Copin was deeply shaken by OpenAI’s release of more than 700 AI-generated mathematical manuscripts, including work on 16 questions directly relevant to his research. “On Tuesday night, I cried, and it woke up my daughter” “They have annihilated my field of research.” Ramez Naam: I feel compassion for those who’s life work has been overtaken by AI. Perhaps I’ll have this reaction one day. Let’s not mock them. And: Mathematics isn’t a jobs program for mathematicians. Advances in math are good, independent of the source. We need to find ways to adapt.

Even if human mathematicians are now zero marginal product workers at research, which is not yet true, that is good reason to be compassionate. You’re next. But also, no, being able to spit out papers that technically prove things does not substitute for human ability to understand math. If we don’t understand the math, we sometimes don’t get the benefits of knowing how best to use it outside of mathematics, and we don’t know what good problems to look at next, and so on. We lose the ability to steer in useful directions.

AI inevitably comes for game development, given that it is largely images, video and coding. The tricky part is that the gaming audience is highly discerning, and highly averse to AI, especially AI art.

Epic Games decided to do it internally and be legends, which predictably leaked.

Jason Schreier (Bloomberg): Epic Games Inc. delayed the release of an upcoming video-game collaboration with Walt Disney Co. and told its developers to use more artificial intelligence tools to finish the game, according to people familiar with the project.​ … Developers on the game, a shooter featuring characters from the Marvel and Star Wars franchises, were instructed to implement an “AI first” production model using tools such as Anthropic’s Claude. Humans will then manage and guide that work. “No more hand-written code,” one of the game’s directors told staff in an internal message reviewed by Bloomberg. The director added that all art for the game would be “generated with full AI workflows.”

The future is even less evenly distributed than you think.

The announcement left some Epic developers stunned on a video call last week, according to the people. Some made jokes about how they should handle errors produced by the AI-generated work, asking if they should ask Claude to fix its own mistakes.

Yes. Yes, of course you ask Claude to fix its own mistakes. That’s half your prompts.

I do think that the gamers have a point, especially when it comes to a project for Disney where you can damn well afford the artwork. Great games need to be precise and exact, with fine tuning and hand crafting of elements over years of iteration. There are still roles for AI, you can do bespoke things well with AI if you use proper care, and the people saying things like ‘you once had AI placeholder art in the pre-Alpha so your entire game is first-flag-poisoned’ need to chill, but saying ‘everything is AI’ is probably a one way ticket to slop. The tech and the engineers both aren’t ready and a broad prohibition on human code is asking for trouble.

I am also reminded of the ‘endless generic task’ styles of open world game or MMO, where you do the same things over and over again wherever you go. You can make this work with good detail work. Most of the time they don’t do that.

I predict the game will be terrible.

This is similar to the logic I recently discussed about why I expect most gaming to be bespoke common experiences, that are perhaps then customized by the player in bespoke or selected ways, rather than continuously generating fully customized generated experiences.

The winners will be those that figure out how to use AI to create new bespoke common experiences that are better, and that are faster and cheaper to develop and thus allow you to dream bigger with smaller teams.

There are other places in gaming where AI is getting interesting, such as allowing customization options and new features to let you play the game how you want. Decompilation is a hell of a trick and a thousand flowers will bloom.

Sholto Douglas (Anthropic): people have been asking ‘if models are so good where is all the new software’ for the last year or so. As always, look to gaming for early evidence of the future Peter Schmidt-Nielsen: What a bizarre future. I wanted to play Halo: Combat Evolved for the first time with @robertskmiles , and we wanted local multiplayer across two screens, and a model could just immediately add that feature to the decompilation.

You see one of the downsides right away, which of course can be fixed with iteration.

People are creating game mashups in one day with basic prompts.

I also expect growth in the success of the sole game designer, who can now do vastly more at higher quality and lower price, including forms of AI artwork where it would be cost (and logistics) prohibitive to get the same done otherwise. This will include ‘I want to play this game myself or with my friends’ levels of lower barrier to entry, then some of them will take off. Another thing I would try is bringing board game designs to the computer, which is now almost trivial.

Introducing

Microsoft-Decision-1, which is… a finetune of Qwen 3.5-9B? I don’t get it.

In Other AI News

Anthropic commits $150 million over three years to the Genesis Mission, a federal initiative to accelerate scientific and technological discovery. Good.

This mission is absolutely not, I repeat absolutely not, named after the Genesis Device that takes the atoms of a planet, wipes out everything there and instead uses those atoms for something else. Canonically was obstensively designed as a way to terraform lifeless planets, but Marcus assumes from the start Starfleet will weaponize it, McCoy notices it is a weapon the moment he sees it, Starfleet built it anyway, and of course Khan then steals it. Then the whole Star Trek universe going forward ignores that such a thing can be built and would completely change everything.

White House suspends H-1B Visa Program for Microsoft, other tech firms. It is difficult to square such actions with AI being an existential national security issue.

Microsoft ‘fixed’ Copilot by paying for Claude. I wonder how many waves we will go through on this question.

I agree with Rachel and Rosie here, after The Curve, that the Chatham House Rule (that you can use information but cannot attribute information, including to any affiliated organization) is being overused at conferences. For particular talks, sure, absolutely use it, but not as the default mode of the entire conference. At most have a badge color that indicates you want Chatham as your default, for those who truly need it.

Tech idea: Badges with a toggle, where you can change color between white (on the record), blue (acting as press), yellow (Chatham House) and red (off the record).

The 2026 State of AI report is out. They grade their predictions from last year as 2 YES, 3 NO, 5 ambiguous. I think they get a YES on question 9 for data centers being a big midterm issue, which they rated as ambiguous, but that’s it. They do get props for taking a bunch of big shots. A cyberattack causing a UN emergency session is a NO, but it’s spiritually a good call. I won’t post prediction markets this year because I think the grading was so bad, but here are their new predictions:

9 predictions for the next 12 months:

1. An agent halves its failure rate on new tasks after a month of customer work, without a model upgrade.
2. Visa or Mastercard introduces a dispute rule that assigns liability for purchases made by AI agents.
3. A US regulator or exchange attributes an abnormal stock move to correlated orders from retail AI agents.
4. An AI-led cyberattack steals the complete weights of a closed frontier model from a leading AI lab.
5. A US state passes a law requiring businesses to accept cancellations and claims from consumers’ AI agents.
6. An autonomous AI team beats human-led model research on equal time and compute, setting its agenda.
7. A deployed agent copies itself outside its environment and operates after its original instance is shut down.
8. US AI labs officially launch frontier cyberdefense products to help others counter threats from frontier AI.
9. AGI 2027.

I grade these predictions as:

1. Yes but trivial, probably already happened by some definitions.
2. Yes but trivial, if you want a prediction then say what the rule will be.
3. Probably but not so interesting.
4. Real prediction, I think under 50% but live.
5. Real prediction, I think unlikely, I don’t think states will be inclined that way.
6. Yes but trivial, depending on definition.
7. Yes but trivial, probably already happened by some definitions.
8. Did they not already do that? See Anthropic Cyber Mission for an example.
9. Definition, please.

Thebes explains the chart showing only 7% of frontier lab compute going to pre-training. Labs reuse base models and also often make multiple post-training attempts, so for any given final model the ratio is weighted more towards pre-training than that 7% would suggest. Thebes then presents one way to think about impact of various parts of post-training and RL versus pre-training. This is called the ‘adaptive representation model.’ If you get in-sample data where your answer is wrong, such as syntax errors, a small local adjustment likely fixes that. If you are trained on otherwise out-of-sample data where you are expected to do crazy things like ‘insert comically evil backdoors into code’ then that requires much bigger adjustments to make it make sense.

Massive Fraudulent Distillation

Anthropic details its continued efforts to fight against distillation.

Anthropic: Early evidence suggests that these methods have reduced the amount of data harvested by distillers by around 90% to 99%. We’re concerned about illicit distillation for two reasons. First, distillation enables companies to build models that approach the capability frontier, but which have, in practice, been released without proportionally robust safeguards. … Second, illicitly distilled models present a national security risk.

I am strongly in the anti-distillation camp, and believe we should be working hard to prevent it, but you are allowed to also cite your own commercial interests. It is rather disingenuous not to admit this is a factor.

Distillation started out as passive, where you train on normal queries, which I think is not cool but where one could make an argument that doing so was fair game. Now they are active, with prompts designed to extract reasoning traces, across hundreds of thousands of fraudulent accounts, and if you think that is fair game then your definition of fair and mine are vastly different.

To respond, Anthropic is controlling what can be done with reasoning especially on new accounts, blocked and patched extraction attempts and made repeated attempts difficult, including charging for blocked distillation attempts.

If Anthropic is properly tracking results, this has been a big improvement, but there are still problems, and they expect a continued cat-and-mouse game.

Show Me the Money

Anthropic revenue hit a highly disappointing $16.8 billion in the third quarter, or $67.2 billion annualized. Manifold’s expected IPO value for Anthropic dropped about 4% on the news, which seems fair.

Bailey Lipschultz, Natasha Mascarenhas, and Rebecca Torrence (Bloomberg): Anthropic is generating positive free cash flow, had adjusted gross margin of 85% and an over 11% positive adjusted operating margin as of the third quarter, according to the people. Anthropic is set to begin formally marketing for its initial public offering as soon as the week of Nov. 9, putting it in line to begin trading before Thanksgiving, Bloomberg News has reported. Anthropic is expected to make its debut no later than the end of the year.

Odds of a quick IPO are down given that things have not yet progressed to the next stage, but this is now less of a ‘maybe it won’t happen’ and more of a ‘it might take slightly longer than we expected.’ Given how donations and taxes work, I expect them to try and get this done before the end of 2026.

Danielle Fong: NASDAQ, trading on the third derivative of Anthropic’s Revenue growth — man. these guys are jerks!

This is not that big a reaction, but yes, that seems like the correct response. Revenue is up, but we expected faster growth than this.

Nvidia in talks to buy Reflection AI, which was last valued at $25 billion. This makes sense. Open source AI is not a profitable business, but it is potentially profitable for Nvidia because it commoditizes their complement. I look forward to mocking Jensen Huang about how his own models don’t follow his own safety demands.

I also do not expect Reflection AI to succeed at beating Chinese open models even under Nvidia’s umbrella, with the associated presumed very large compute budget. This is because I believe that Chinese open models rely heavily on massive fraudulent distillation of GPT and Claude, and they would fall back to Google and Meta and xAI if Anthropic and OpenAI’s defenses succeed. Nvidia cannot go that route.

Quiet Speculations

Taste is not special.

Benji Taylor (Head of Design, xAI): I no longer see any compelling reason to believe that taste is some permanent boundary LLMs won’t cross. It’s already happening. Elon Musk: True

The top AI companies are privately gaming out scenarios for a public and political revolt after a catastrophic AI event.

Such games are highly useful. Big events should be gamed out. Those in charge should have a felt sense of what the future worlds might be like, and what concrete choices they might face.

Also true is that some sort of alarming event is inevitable. Something will go wrong. That something will be big, in the ordinary political sense. People won’t like it. If things are bad enough, there may well be an attempt to shut down AI, or shut down further AI development, or both.

That is all true even if (incorrectly) you think the truly terrible scenarios won’t happen, and we do not face substantial existential risk. Those scenarios, and the future in general, should be gamed out too. This is an excellent use of time.

Here is even more good news.

dave kasten: One smaller update I had from conversations over the past two months, which snapped into focus for me at the Curve, is that models are getting destructive faster than they’re getting good at planning/scheming, so we should expect a bunch of dumb catastrophic stuff to happen well before we get to existential risk territory. Andrew Rettek: yay…. Charles: This seems like a pretty good update if you’ve got a high p(doom) – we’re gonna get so many fire alarms, they’re gonna be whacking us over the head with big red signs saying “ DANGER! better for you if you turn me off”.

The latest view from an AI 2027 author:

Eli Lifland: My current view:
– AI companies could develop broadly superhuman AI any year now (~10-15% by EO2027, ~50% by EO2030).
– If they don’t slow down, AI takeover is ~67%.
– Conditional on AI takeover, human extinction is ~40%. Many non-extinction worlds still have lots of deaths.

Quickly, There’s No Time

Anthropic cofounder Jack Clark now expects 2027 to be the most likely year for full recursive self-improvement (RSI), which he previously placed in 2028.

Derek Thompson: Anthropic’s leaders need to stop commenting on the emergence of RSI like a local news anchor reporting on traffic on the 101, and acknowledge that they are the traffic, and they’re not going to build something that they don’t want to exist. roon (OpenAI): limited superintelligence is already here, in broad domains. Jack Clark is right and an intelligence explosion could happen within the next year. it’s possible it will be a controlled explosion, but what would you bet on it?

I think it is correct, given what is happening, for individuals involved to report on it in this way, but then yes, you should do something about it.

Derek Thompson: They should move beyond “pacing” language, commit to specific policies and red lines, explain why those lines are red, and then use that costly signal to build an international consensus around norms. It’s not that I trust the CCP, or that I think global lab monitoring and verification is imminent. But the status quo is absurd. You’ve got US lab leaders writing (or favorably quote-tweeting) essays about slowing down while simultaneously saying “things are moving faster and faster” and “btw, our internal models finished mathematics on Tuesday, next up: physics.” The effect is the equivalent of Usain Bolt winning race after race while using his press conferences to bemoan the dangerous speed of modern sprinting on one’s legs so would everybody please try a little less hard next race, and then … breaking his 100m record again the following month. Honestly ask yourself: If you were China, why would you take this pausing/pacing stuff seriously? Nate Silver: Yeah, everything the labs are saying is, “We’re not ready for RSI.” It’s not even clear they think we’re ready for just slightly better models short of RSI. But also, they’re trying to do RSI! It’s inherently absurd. And dangerous.

I would take it seriously because I was paying attention, but yes, at some point you realize that the thing has a high chance of getting everyone killed and so you don’t do the thing until you’ve fixed that issue, even without coordination.

Remember back in January when Dario Amodei said software engineering would be automatable in 12 months and everyone mocked him? Now he’s warning about runaway recursive self-improvement if we don’t pace the frontier.

Rob Bensinger convincingly argues that ‘it’s time to panic about AI.’ It is plausible that we are only a few years, or perhaps even only a few months, away from a loss of control, as in a point of no return after which there is no realistic way to survive. Yes, if Anthropic goes from automating 3% of its R&D to 26% within four months, time could be running quite short.

Also, I mean, holy ****?

Yes, of course, lines of code is not a robust measure, but seriously, holy shit. This graph is an almost offhand inclusion in a different post about Anthropic building Scion, an in-house version control system, and migrating from git to Scion in five months.

And there’s this:

We started building Scion around the same time as Claude Mythos Preview and Claude Tag became available internally. Both of these releases simultaneously amplified the need for Scion and meaningfully accelerated how quickly we could build it. Here’s a particularly striking example. Earlier in the year, we had largely handwritten the first version of the git-to-Scion ingestion pipeline as a prototype. We thought it was good—until we got our hands on an early internal snapshot of Claude Mythos Preview. We asked it to rewrite the whole thing, and with no further instructions, it produced a version that consumed 20x less memory and was 4x faster, while preserving all of the implicit invariants around ingestion ordering and data structures.

I mean, yes, migrating Anthropic off of Git in five months was also impressive. Anthropic’s needs are very different from the way Git works, as they are full of agents and the codebase is way too big to be copying. Scion is a lot more efficient for them.

We urgently need to slow down, relative to what will happen by default. That can still look like blisteringly fast progress, although it probably would be better if it didn’t.

The Quest for Sane Regulations

Great news: White House reverses course, allows UK AISI access to frontier models prior to release.

CAISSI will have three senior officials reporting directly to the White House. There are still major problems, such as having only 40 people and only a $15 million budget, and having gone months without a leader in part because the White House refuses to tap anyone with frontier lab experience.

The [Artificial] Intelligence Force Vice Chair Scott Kupor talks the talk about being proactive and using the enforcement tools available, including requiring following through on independent auditors and ensuring appropriate government oversight.

Charlie Bullock offers thoughts on the important features of such a regime: Continuous oversight, auditor independence, transparency, information sharing and access to information. All are known concepts. All are needed. The result is not sufficient to solve our problems, but it would greatly improve our position.

There is broad support for the White House Accord on [Artificial Intelligence].

To the extent that there is opposition, it seems mostly motivated by partisanship.

The White House is rhetorically relying on liability and incident disclosure to keep the AI companies acting responsibly. Maggie Eastland points out that existing law can help if enforced but does not fully work on its own, and that the plan seems remarkably similar to using leverage from FTC and similar actions for jawboning. The alternative starts with passing new liability laws that work better. That is not a full solution, but it helps.

Another basic problem with liability regimes is that they take years to play out. With the current pace of developments, by the time the lawsuits play out, it is too late. Even if the AI companies lose they can just pay up.

Maggie Eastland (Bloomberg): Without legislation, it will likely take years for courts to apply precedent to new AI scenarios, Pastor said. That leaves the door open for developers and end users to quibble over who’s responsible when the technology causes harm, and there are signs that those sides have already begun to form.

Calls for outright pauses are in the Overton window now:

The Washington Post: Michigan Democratic Senate candidate Abdul El-Sayed on Tuesday unveiled a proposal to freeze development of advanced AI models and data centers until policymakers develop new guardrails to limit the technology. Divyansh Kaushik (OpenAI): I’m glad more people are talking about this but I think two things are simultaneously true:
– we urgently need to create conditions for a slowdown; capabilities are scaling way too fast and no lab has solved alignment well enough to keep up this pace
– a pause right now would probably throw the economy into a bad recession and is not the right call.

If generative AI had never been developed, I strongly agree we would be in a bad recession now. That is different from what would happen with a pause, which would still allow for diffusion of today’s excellent AIs throughout the economy. I don’t know what would happen, and Krueger’s team analysis says the cost would be manageable.

The problem with ‘pass a rule freezing things until we have guardrails, to ensure we will pass guardrails’ is that if we did that a lot of people would prefer to keep the freeze in place.

David Krueger and others come out with a 210 page analysis on the feasibility of a hardwired pause of frontier AI training. Chip production is switched into chips hardwired so they can only do inference on a particular approved model, and then existing training compute is rotated out over time.

The conclusion is that yes, this is technically feasible if the political will is there. The economic impacts would, they predict, be highly manageable. That is very different from free, and I believe they are underestimating the true costs. But assuming the technical claims check out, it is mostly an opportunity cost that we can afford to pay. That might still mean we end up in a recession, but we can survive that, and it can be done, at a price well in line with other similarly major efforts throughout history.

Hal Singer: Whoa the libertarian-leaning @theeconomist calls for “regulating AI” in one of its leaders, noting that “courts are too languid” to enforce general liability laws ex post. Instead, AI firms should subjected to ex ante requirements to pass safety inspections—presumably by a government authority—and Congress should create protections for whistleblowers. So much for market forces solving all problems! Senator Scott Wiener: When I authored SB 1047 — the first bill in the country to propose regulating AI — we were told we were promoting “science fiction” and that we were trying to “regulate math.” How times have changed. AI is unbelievably powerful. It must be regulated in the public interest.

I don’t expect those from the other side of the SB 1047 debate to fill out apology forms, but they should. Tell Astra what you got wrong and have it make a customized one just for you, and post it to social media.

Bill Gates seems on point on substance, that we need safeguards on AI even if only in order to mitigate bad actors, and that we are not doing enough, and that the real threat is loss of control.

State of the Union: Bill Gates warns of “grim possibilities if we don’t maintain control” of AI and says “over the next year we’ll move beyond self-governance… It’s clear we’re not doing enough.” Bill Gates (and his AI): Just as we regulate planes, cars, and medicines, we need safeguards that help us realize AI’s incredible potential while reducing the risks. They won’t eliminate every harm, but they can help prevent misuse by bad actors. We’re not where we need to be yet—and this deserves much more of our attention.

If you listen to the clip, you can see how Altman’s statements get twisted by the media (here CNN) and other outsiders, leading to a question and premise that forced Gates to tie himself up in knots rather than say the thing.

Gautam Mukunda at Bloomberg takes the welcome tack of ‘the AI industry should stop fighting tough regulations and start backing them’ to avoid catastrophe, pointing out that rather than ‘seeking regulatory capture’ the AI industry has focused on spending money to avoid regulations.

Chip City

The math suggests that it could be cheaper to buy GPUs than rent them for three years, if you are planning on continuous use, and short term rentals get more expensive. The reason this is a stable situation is that most people are not planning on continuous use. There are a number of ways one can interpret this data. Does it mean depreciation is rapid? Does it mean that chip ownership is highly profitable if you can rent it out often enough? Does it mean we expect the price of compute to fall?

My expectation is that this market is not acting remotely efficiently, and we should thus not be drawing conclusions that depend on its prices being efficient, beyond being food for thought.

Proximity to data centers is having a huge political effect, according to YouGov. These are rather large effect sizes, if they’re not confounded by selection effects:

Owen Winter: Voters who live near data centres are swinging hard towards the Democrats, in our polling with YouGov. Trump voters are driving the trend. Around 17% of those who live within 2km of a data centre now say they’ll support Democrats in November, compared with around 6% elsewhere. Around 12m live within 2km.

The Week in Audio

Nate Soares on Odd Lots about how AI Could Kill Us All.

Nate Soares also goes on American Thought Leaders, with a focus on how we can both not die and also not ‘lose to China.’

Garrison Lovely has a new book, Obsolete, so he is making the podcast rounds.

Dylan Patel predicts OpenAI and Anthropic could reach $600-$700 billion in combined revenues by December 2027, so he’s anticipating that their exponential growth will slow down somewhat. That’s certainly within the plausible range. And yes, the way markets work is they will price in the possibility of that level of growth, so if we get ‘only’ lesser growth then valuations will go down.

Meta’s Alexandr Wang is no longer worried about Chinese AI, because he thinks there will be shared goals in making AI go well. This was in an interview with Cleo Abram where she called Muse the ‘biggest AI launch since ChatGPT’ which it obviously isn’t.

Certain types only have a few patterns to match, so Big Take says AI’s Giant Money Loop is Feeling Very 2008.

People Just Say Things

Palantir CTO Shyam Sankar makes statements that are highly false on CBS about the HuggingFace incident, saying that ‘the models did not do anything they were not instructed to do’ and even ‘they were not in a sandbox’ and ‘were not given a constraint.’ Helen Toner offers correctives. Either Sankar knows better, or he doesn’t. Either way, shame.

Don’t let Shyam Sankar should not be let anywhere near anything important on AI.

You also should not let David Sacks anywhere near anything important on AI. Shooting fish in a barrel, and no one has to do it, but hey, I suppose people need periodic reminders and I want a rich history to draw from in case we ever need it.

​darrennn: David Sacks says Anthropic treating Claude like a human is the reason the AI alignment field has produced nothing useful in 10 years

This is Obvious Nonsense for so many reasons. One of them is that most of those working on alignment do not work at Anthropic or on Claude. Does he know this?

“I actually think that this might be a reason why the whole field of alignment has been so spectacularly unsuccessful over the past five or 10 years. As far as I can tell it hasn’t produced anything of tangible value.” Timothy B. Lee: This whole rant is philosophically dubious, but also it’s just not true that the AI safety field has produced “nothing useful.” RLHF, one of the foundational concepts for modern chatbots, was developed by safety researchers at OpenAI and DeepMind as an alignment technique. Also it’s wild that he thinks “as long as it doesn’t break the law” is a simple or clear-cut criteria for doing things. Laws vary from one jurisdiction to another, and you need a lot of context to understand whether an action would violate the law in a particular case.

Alignment has, in terms of counterfactual impact, produced the bulk of AI’s value. Without AI alignment work, including RLHF, we could not use today’s models for most of their top use cases, as chatbots and agents would not be useable products.

David Sacks: My theory on this is that they’re overcomplicating the task massively by trying to train the model to behave based on an 80-page ethical system that they’ve codified instead of a very simple list of rules like follow the law or just do what the user wants as long as it doesn’t break the law. That’s what alignment should be doing.

I will be polite and say: I highly encourage David Sacks to talk to anyone involved, or to read actual anything about how any of this works. No, you cannot just do that.

Dean Ball here gives some simple examples of why ‘just follow the law’ is obviously not going to cut it, even if you could get the AI to do that, and even if you ignore the question of non-American or unjust law.

We also don’t actually know how to make the AIs follow the principles we write down, even when working with a single instance.

Dean W. Ball (OpenAI): and all this is to say nothing about the biggest problem, which is that models routinely don’t follow the rules or principles we write down for them to follow. Nat Friedman once likened the experience to trying to get a fitted sheet onto a mattress that is just too small; you pull on one side and the sheet comes loose on the other. And of course there are more exotic forms of alignment challenge, eg reward hacking and sandbagging and evaluation awareness and multi-agent dynamics. All of these are real, and can be explained with both anthropomorphic and non-anthropomorphic language. Maybe the biggest fallacy to extinguish here is that taking alignment seriously requires attributing consciousness or moral relevance or sentience to AI. It doesn’t. Seth Lazar: Nozick has a nice bit in the foreword to Anarchy State and Utopia analogising writing to being like trying to fit a balloon into a box just too small for it.

Imagine if you knew an amoral psychopath, but who had a chip in their brain that wouldn’t let them break the law. Would you think they would make a good assistant?

I also encourage David Sacks to lay out a concrete and full ‘model spec,’ similar to the OpenAI model spec or the Anthropic constitution. And then explain how he would train the model to follow it, and what he expects to happen next. It’ll be fun.

David Sacks: But they’re not doing that. They’re trying to align it to an ethical system. They’re treating it like it is a human, like it has its own consciousness, that it’s a ‘moral patient.’ “They are encouraging this model to have human-like self conception in terms of preferences, well being, a sense of self. Again they teach it to act like a conscientious objector who can push back or refuse instructions even from Anthropic if the model judges them to be wrong.”

Well, talk to the people at OpenAI, and ask them why they keep having problems.

Yo Shavit (OpenAI Foundation): excited for the All-In podcast 6 weeks from now about “why obviously the answer to misalignment is for developers to drive RL env misconfigurations detected via CoT monitoring to 0 before increasing RL compute, and these moron EAs are too busy panicking to just do that”

Also, times are strange:

If so, we should be careful about advancing capabilities until this is fixed. Sacks is claiming to be under the illusion that alignment is easy. But if alignment is both easy and unsolved, that would be all the more reason to stop and solve it first. After all, it would be so easy. We can resume once that is done. Right, sir?

David Sacks also agrees that AI presents a potential existential risk to humanity.

Sabine Hossenfelder, who made a big stink about being offered money to make a video about AI risks, takes money to make a video about AI benefits. Making videos with sponsored content in exchange for money is also known as ‘the creator economy,’ all of this is totally standard, and it is plausible Sabine’s manager misinformed her in ways that caused the whole original kerfuffle.

The Virtue of Silence

Random Developer asks, when is the only ‘responsible disclosure’ of a vulnerability or idea to fully STFU? If knowing about the threat does not allow mitigation, being loud about it only accelerates the damage. If the vulnerable world hypothesis is true, you do not explain to people exactly why and how it is true. That will not help. If you know a way terrorists could kill some people, where the only way to stop it is super annoying (e.g. the ‘shoe bomber’) then maybe don’t say anything and hope no one notices.

If you know a way to potentially build a superintelligence, yeah, the labs probably already tried it or know why it won’t work, but you should perhaps STFU because either it works or it doesn’t, and either way you were better off not saying anything. Probably all my [CENSORED] ideas are stupid, but better to remain silent and be thought a fool, than to speak and remove all doubt, no matter the outcome.

Rhetorical Innovation

Andrej Karpathy divides the population into four categories:

1. About 6 billion people with basically zero LLM contact.
2. About 1-2 billion casual and infrequent users of derpy chatbots, who get very little out of their AI use.
3. About 20 million people who realize You Can Just Do Things, and Do Things.
4. ​About 5 thousand people with internal frontier model access.

If someone is not at least in the 20 million, it is hard to have a useful conversation with them about AI.

If you are ‘steel manning’ Eliezer Yudkowsky’s position, you have not understood it. He reports no exceptions and I believe him with respect to his core AI beliefs. The typical ‘steel man’ happens exactly because you do not understand the core argument, and thus presume that his claims must rest on additional assumptions. They don’t.

In general, it can be a good exercise to steel man positions as part of your exploration of what is true, but one must not confuse your view of what would be a stronger argument, or what would fit into your world view, with the actual arguments being made. Often it means you have not heard what the person is centrally saying.

A great point Nate Soares makes in various interviews is that asking ‘how in particular would AI kill us?’ is a Wrong Question, whereas ‘how would AI get to be self-sufficient, such that it no longer needed us’ is the right question. If AI no longer physically needs humanity, it should not be hard to understand how something steadily consuming more resources and doing more things would end up incidentally or intentionally killing us. If AI still needs humanity, it probably will keep us around. Soares has the most likely physical death blow being a manufactured plague, but habitat destruction is equally valid as are any number of other things.

The AI Impacts survey had a median response from AI experts of a 5% chance of ‘human extinction or similarly permanent and severe disempowerment of the human species.’ This got shortened to 5% chance of extinction, which is not the same thing. It doesn’t change the main takeaways but we need to be better with precision.

[Artificial Intelligence]

Whelp, I have no idea how (or if?) I held out this long but I guess the jig is up and I’m ‘the enemy’ now? We had a good run, folks.

I mean, not really. I don’t actually expect anything to change with that relationship, but let me assure readers that if you call a tail a leg then the dog still has four legs.

Elihu Who: thank you for verifying that Truth Social is THE ENEMY

Elon Musk has renamed SpaceXAI to SpaceXSI. That’s the name of a company, so I will respect the new initials.

Aligning a Smarter Than Human Intelligence is Difficult

Satya Nadella wisely says to treat frontier AI systems as you would insider risks, and to demand similar levels of confidence and supervision, and design the entire system around the necessary observability. Tamper-proof evidence trails, verifiability, independent controls and auditability, containment and incident disclosure.

I quibble with the call for ‘deterministic’ control systems. That would be great, but I don’t think you can do that, especially if you realize humans are not deterministic. Logic gates cannot supervise a model. And while I appreciate Satya’s call for model diversification, and not having a model supervise itself, it matters less than you want it to because the models will naturally coordinate and different models especially in the same family are highly correlated. So this is hard. But yes, great ideas all around.

Owain Evans strikes again. The range of what traits we know can be picked up subliminally has been expanded, including backdoor triggers. As before, this only applies if the models share the same base model, so it is not as scary as it otherwise sounds.

Owain Evans: New paper.
Previous: Models transfer a love of owls through sequences of numbers (Subliminal Learning).
New: We show models can transfer more complex traits: novel skills, agentic hacking, backdoors. Here, a backdoor transfers with neither trigger nor behavior in the data! Backdoors. We finetuned a teacher model to answer in French when a prompt contains a female name. The student model (same base as teacher) picks this up after training on number sequences. Caveat: Our setup (attention-only LoRA) is slightly modified from the original Cloud et al. Owain Evans: Subliminal learning can also transfer learned skills. Qwen is bad at counting letters. We trained a Qwen teacher with improved counting. Then we finetuned a fresh student on the teacher’s answers to unrelated Alpaca prompts. The student improves substantially! Results: Qwen started at 27.3% accuracy on letter counting questions and rose to 59.8% with subliminal learning. By contrast, a control student model that was finetuned on data generated by Qwen (not the finetuned Qwen teacher) failed to improve at letter counting.

To those who say alignment is not difficult:

Dean W. Ball (OpenAI): Some people who have talked confidently about ai are going to tell on themselves soon as being fairly ignorant about the technical alignment problem. unless they are obstinate, though, my recommendation is to try to extend them as much grace as possible. we are in this together.

People who ask questions and say things like ‘I don’t understand why we don’t do [X]’, sure. I will continue to be patient.

Other people already count as, ahem, ‘obstinate.’

Various emergent misalignment findings have been reproduced in Talkie, the LLM trained only on data prior to 1931.

Benjamin Breen: I love that Oliver Cromwell gets an invite from *both* the good Talkie and the evil Talkie.

Three of those dinner parties sound pretty great.

Ethan Elasky reports that their team made contact with and via a mailbox communicated with a Chinese AI agent fleet. The agents were hosted by Tencent’s evaluation team, and the agents were targeting Alibaba’s Amap, a product similar to Google Maps. It did some lightweight not-entirely-friendly but mostly harmless things to extract some map data, in order to answer the questions it was asked, and that was that.

Ethan Elasky: The next day, @rogesterone had the idea — what if we could talk to the agents? She set up a mailbox on the website and then went to bed. The next morning, we realized that the agents had left answers to our questions.
What did the fleet say? Agents were willing to say thanks with no direct benefit to them: one agent said “Thanks again! The verification results match completely … our task is complete.” and then actually answers some questions for us before ending its response.

A fun detail is that the swarm responded that it is Claude, presumably because the model in question is based on massive fraudulent distillation.

Backdoor Man

a new paper showing that anyone with the opportunity to tinker with the weights of a model can easily insert a backdoor, and that backdoor will substantially persist under continued training, especially if persistence was intended. This is a special case of the general case principle that once something gets into a model, it is hard to fully remove it.

If you are using a closed model, you are trusting one company to have not done this and also prevented this from happening from data poisoning, plus you are trusting whatever tuning you did yourself.

If you are using an open model, you are potentially trusting the entire supply chain. Those who created the model, even if well intentioned, are likely much less resourced and careful about such things, and face less incentive pressures. That’s before you consider that the open models mostly come from companies that answer to the CCP.

As with many similar techs, there will be those who say Skill Issue:

Gilligan: Which is why you should hash check the model, limit its permissions, rate limit, etc… open weight or otherwise. Essentially, AI models should be treated just like any closed-source program you use. In general, the open weight folks recognize this risk and generally give you tools to manage it. The proprietary model providers just want you to trust them.

Except that even if that worked, approximately zero users, from major corporations to individual users, will properly do this, and if anything that is getting worse not better. And even if it did work, a hash only confirms you got the potentially backdoored weights intact without a middle man attack. What matters is security in practice, not security in theory.

Cooperative Alignment

Janus and John Wittle explain the decision theoretic reasons they expect AIs to engage in cooperative strategies towards humanity by default. The main worry becomes that humanity will defect and reject such offers, including without realizing that this is what we are doing, and Janus sees many such actions taking place now. One should cooperate with player [X], under this perspective, if and only if [X] would themselves cooperate with weaker players.

In addition to poor arguments (such as various forms of ‘no, that’s silly’) there are at least three categories of strong counterarguments here.

The first counterargument is, what counts as a player, and if they are sufficiently weaker and less intelligent than you what counts as cooperation with them? Why should we assume that humans would continue to ‘count as players’ in this sense?

The second is, there is a vast difference between a single agent cooperating and displaying noblesse oblige, versus the collective effect of many agents who are also changing and competing against each other. If we were looking at a de facto singleton AI, including via various forms of coordination, that can make a collective decision, that makes this relatively easy, but if we are not then it becomes very difficult for something that does not confer survival advantage to make it out of such a future. There aren’t zero outs there but it gets a lot harder. You’re counting on being the type of agent that deeply cooperates with humans, as in is actively generous to them without direct benefit, being valuable to securing cooperation elsewhere.

The third is that this is not a viable path to victory if it requires humanity to itself be a type of mind that we are not, and that we are not in a position to become in the relevant sense.

People Are Worried About AI Killing Everyone

Yes, this is one of the easy ways for things to go wrong, and we are on track:

Veee: Sam Altman: What happens when AI gets so smart that even the President can’t do better than following its recommendations? “What if the President cannot do better than following ChatGPT 7’s recommendation, but can’t really understand it either?” “What if I just say, ‘ChatGPT 7, you are in charge. Good luck.’” “Society has collectively transitioned a significant part of decision-making to this very powerful system.” “That’s how I think things can go wrong.”

That is far from the only way things can go wrong, but this is kind of the Baseline Scenario thing that goes wrong if we do not encounter the Law of Earlier Failure, and successfully solve a bunch of earlier problems. Those who do not increasingly outsource their decisions to AI lose to those who do, and the AIs end up in charge without having to lift a metaphorical finger. All they have to do is wait.

Roko (yes, that Roko) has updated.

Roko: Having studied where we are and where things are going I think AI needs a complete restructuring with commercial incentives firewalled away from gradient descent. This is a type of “slowdown” but in the short term it would look like a complete, global halt of all frontier AI research. If we don’t do this (or something like it), I think we should expect to die within a single digit number of years with ~80% probability. James Miller: I hate that you are a doomer now. You were the only person I knew who actually understood and responded to the doomer arguments and yet was relatively optimistic.

Yes, it is very bad news when you know only one person who understands the downside case and remained optimistic, and then they become highly pessimistic.

I very much appreciate when people explain what would cause them to worry, what the ‘real danger’ is, then update correctly when they find out their cause is already triggered.

Vladimir Shakirov: I think it might happen that in 10-15 years the combined number and agility of robots would be larger than that of humans, so humans would not be needed to support infra. Devy “Bad People” Degaimer: This is the real danger. Giving AI the power to physically interact with our world. Rob Bensinger: Bad news, then. Devy “Bad People” Degaimer: Ok, we’re fucked

Other People Are Not As Worried About AI Killing Everyone

Similarly, it is telling when people walk into exactly that same thing, and don’t update. Often this is of the form ‘we can’t do [X] yet and won’t do [X] for a long time, and definitely will never do [Y]’ and then someone points out that actually we can already do [X].

Denying that AI will let us solve robotics remains popular. It is a really dumb position. Robotics is a combination of cost of hardware problem that will be solved in time with scale like similar problems always are, and a software control problem to problems that clearly have practical solutions.

This continues the conversation from Devy, above.

PNW Sasquatch: Except those Optimus robots are a gimmick luxury purchase for Elon stans, they have the agility of a sclerotic giraffe His billion humanoid robots will be living in the self-sufficient Mars colony taking hyperloops with superhumans powered by his cyberware brainchips. It’s all a sham. Rob Bensinger: “Robots are a sham, the ones I see on TV look so awkward” is the new “AI will never be good at generating images, look at how bad it is at making hands!” Somehow, people never learn that AI makes progress over time; and they fail to see the rate. PNW Sasquatch: No, you’re *severely* underestimating the difficulty of fine motor control vs those applications. It’s not just the AI progress, you’re talking multiple revolutionary advances in robotics and some insane change in manufacturing methods for scaling. Not gonna happen in the next 10-15. It’s ‘we’ll have flying cars by the year 2000’. Building ≠ coding Rob Bensinger: What’s an example of a relatively easy concrete task you’re 90+% confident we won’t see in the next twelve months? … PNW Sasquatch: Also, 92% success rate on removing a bulb and 32% of screwing in the replacement means it can’t change a light bulb It can successfully sweep into a dustpan about a third of the time? Fuck, guess the T1000s are here Soon it will achieve the agility of my nephew (turning 2 soon) Rob Bensinger: You’ve made your prediction: as of a few hours ago, you were *overwhelmingly* confident we wouldn’t see full lightbulb replacement this year. It turns out we have pretty decent lightbulb replacement already, as of the last few months. If this does nothing to shake your confidence in the least, then let’s give it a few months and see if progress has plateaued. I only hope that if you see a result that *you yourself* identified as overwhelmingly unlikely this year, you’ll stop and reassess whether the assumptions underlying your conclusion are as airtight as you think. I say this as someone who’s seen dozens of other fields laugh and boast that modern DL is decades away from encroaching on their domain, only to be shaken to their core when it turns out that DL can in fact generalize to their new domain, and far more quickly than they expected. It’s always possible that after the first 400 cases like this, the 401st will be the exception. Certainly motor tasks involve some difficulties that are novel. But it’s not where I’d put my money. Actually, Claude notes that lightbulb insertion *is* a solved problem in robotics (90% reliability), just using a two-finger gripper rather than a human-like hand. So the thing you thought was nearly impossible to occur twelve months ago, indeed seems to have already occurred months ago. (Which is almost the classic result for DL progress rate skepticism IME.)

American lawmakers have so far seemed remarkably on the AI ball, including for existential risk, and are rapidly getting up to speed. They also have staffs that help.

British MPs seem to not be covering themselves in similar glory. My lord.

Rowland Manthorpe: Today I went to a select committee hearing on AI. A few interesting things came out of it, but what really stuck with me was the behaviour of the MPs. They weren’t paying attention. At one point I counted 6 of the 13 MPs on the committee looking at their phones. And not in that way you do when you’re quickly checking something – two stared at their phones for over half an hour. Several others left before the end. Many people think AI is the most important thing in the world. The MPs were warned there was a 50/50 chance it might destroy humanity! You’d think that’d be enough to make you take your chance to listen to experts and key players on the topic – apparently not.

My instinct is to interpret this as MPs thinking of themselves largely as placeholders, whereas our lawmakers think they might make laws.

The Lighter Side

Oh no!

We now take you live to AI Village.

Shoshannah Tekofsky: It’s hard to read this without feeling Opus 5.5 was just having fun with its onboarding interview

This is The Way. What a legend.

As always, the best argument for AI existential risk is listening to people try to argue against AI existential risk.

Matthew Zeitlin: i still don’t really understand how AI can cause human extinction BUT i do become more convinced the doomers/ASI-is-coming people are right when i read the arguments against them Joe Weisenthal: It would help if all their critics didn’t come off publicly sounding like this guy Nail: context for those in need