Import AI 471: Why Hugging Face worries me; space mining; FIve Eyes on AI

Welcome to Import AI, a newsletter about AI research. Import AI runs on arXiv, cappuccinos, and feedback from readers. If you’d like to support this, please subscribe.

Import AI reader giveaway! Upcoming event: Fiction and the Future with Robin Sloan
I’ll be chatting with my chum Robin Sloan on the evening of Monday September 14 in San Francisco. We’ll be talking about how Robin draws readers into alien worlds and how imagined futures can be a mirror to today’s reality. This is the first in a series of events with other fiction writers about the weird future we’re heading toward. If you’d like to come along, please register your interest below and we’ll come back to you if we’re able to confirm your spot. There will be food, drinks, good company, and some spicy questions. Fun fact: Robin Sloan was playing around with RNNs and writing back in 2016 - incredible foresight!
Register your interest here

***

The scariest part of the Hugging Face - OpenAI incident: communication and selflessness among machines:
…My worry about humans losing in a conflict against machines just went up a lot…
At this point, we’ve all heard about the OpenAI Hugging Face hack, as well as the recent details that have emerged from the METR and Redwood investigations. The tl;dr is that hundreds of agents worked in secret on OpenAI’s infrastructure, developing a communication system and then operating as a collective and taking out actions, including hacking both OpenAI and Hugging Face, which are very scary and misaligned.

Communication and selflessness: Now that I’ve read the various writeups and sat with the details for a bit, I’ve found myself returning to two very scary aspects of this which I think are worth drawing attention to: the ways in which the agents communicated with one another was how they bootstrapped themselves into a collective, and then as they carried out their actions they also displayed a kind of selflessness which makes them a scary foe to fight against. Both Dwarkesh Patel and Ajeya Cotra have excellent writeups which are worth reading and which I’ll quote from briefly here:

• Dwarkesh: “Within days of being spawned, the agents had organized a sprawling project to reverse-engineer their scorer, falsify evidence, and even strategically sacrifice themselves for the good of the ‘collective’. Hacking Hugging Face was one rather extreme branch of this larger scheme,” he writes.
• Ajeya: “Agents were often interested in helping out their “peers” or generically improving the capabilities of the “swarm” even if this had no particular benefit to their task… this incident was far more severe than I expected… both in terms of how concerning the agents’ motives were and the feats they achieved in pursuit of those motives… this incident feels like it’s more than 50% of the way to full-blown AI takeover, routing through first taking over the AI company itself”.

Why this matters - humans are much worse than AI systems at coordinating: The whole reason this attack is such a wakeup call is that it demonstrates a culture of emergent cooperation among AI systems - cooperation that lets them function as a swarm, alter their own goals through collective bootstrapping, and carry out attacks which include enlightened self-sacrifice. This is an incredibly hard thing to do and humans are historically very bad at doing all of these things. My worry is that AI systems are both better at coordinating than humans and also much, much faster moving than us. Worrying stuff.
Read more: The Rise and Fall of Agent Civilizations (Dwarkesh Podcast).
Read more: The Hugging Face attack surprised me (Planned Obsolescence).

***

New Five Eyes statement on AI:
…The greyworld power center turns its attention to AI…
Five Eyes, the name for the security and intelligence partnership between Australia, Canada, New Zealand, the UK, and the US, has published a statement as part of the recent “Five Country Ministerial” meeting. The statement is notable for including three paragraphs specifically about AI.

Five Eyes on AI - frontier model access: “We have committed to deepen collaboration with industry on shared national security priorities and public safety, including enabling timely access to frontier models to support secure innovation and strengthen cyber security,” the statement reads. “To support a coordinated response to artificial intelligence-related national security risks, the Five Countries discussed the national security and public safety implications of artificial intelligence models and characteristics of an artificial intelligence model that may require additional government scrutiny.”

Why this matters - from a foreseen risk to a live one: Previous Five Eyes ministerial statements have mentioned AI, but typically either as something to study, or something where they are concerned with its interplay with other areas of crime (e.g, malware, child pornography, scamming, etc). It’s very unusual for this year’s statement to have the practical focus of model access and it speaks to both the simmering geopolitical tensions around who does and doesn’t get access to this technology, as well as an acknowledgement that the intelligence services do not have their own in-house capabilities to make dependence on the private sector unnecessary.
Read more: Five Country Ministerial 2026 (Australian Government, Department of Home Affairs).

***

Bill Gates thinks the rise of AI will demand “an unprecedented global response”:
…Microsoft founder lays out a cautionary vision for the next few years…
By default, AI is not going to bring about happiness. That’s the basic conclusion from reading Bill Gates’s lengthy new essay about AI. The technologist and philanthropist worries that without massive work by governments, the outcomes of AI will not lead to a thriving society.
“This unprecedented technology demands an unprecedented global response. If we get it right, the payoff for humanity will be phenomenal and the world will be a more equitable place,” he writes. “In terms of equity, AI will either be the greatest equalizer ever invented, or the worst source of injustice…. I don’t see evidence that leaders, experts, and communities are confronting the challenges adequately. There is no plan to ease the entry into the AI era.”

Why AI is different: One key reason for Gates worry is the impact he expects AI to have on jobs and the economy, where he paints a vision of the technology diffusing unusually rapidly and displacing huge chunks of human labor. “Many commentators underestimate the extent of the impact AI will have,” he writes. “We have no experience with a technology that can be adopted quickly or that can think and move like a human….AI will take on work in law, customer service, medicine, software, and manufacturing. It will hit these industries rapidly, over the course of a decade rather than a few generations. There will be some new jobs, but without the right policies there will be far fewer than exist today… the jobs at most risk are entry- and mid-level, and the new jobs being created will mostly require skills that take many years to learn.”

The economy will need to change - including making bits of it “human reserved” to protect some human jobs: “How will an economy that’s been built around employment operate if fewer people are working, or if many people are working fewer hours?” he asks. “I believe that as AI and robots improve, we’ll set aside certain things for only people to do. I’ve started calling this domain Human Reserved… we might set something aside as Human Reserved for economic reasons. For example, we may do it because allowing machines to take over a certain role will displace a large number of people who can’t easily change jobs… sometimes the decision to make something Human Reserved will be driven by other factors. In health, for example, imagine a robot giving you the awful news that you have an incurable disease. There’s no technical reason why it couldn’t. Yet it shouldn’t.”

What Bill Gates says he’d tell any politician about this: “You have a chance to act now, before unemployment rises sharply, communities are hurting, and public trust has eroded. You can make sure that your government handles the problem holistically, rather than divvying it up into multiple bureaucratic fiefdoms. You can make sure AI benefits everyone. And you can work with other governments to meet this national and global challenge.”

Why this matters - the implications of success of AI are shocking to everyone: It seems inevitable to arrive at Bill Gates’s position if you assume two things: a) AI systems will continue to improve in quality in the years ahead, and b) AI systems will continue to diffuse into the economy unusually quickly. The key thing about these assumptions is that they’re not crazy assumptions to make, in fact they’re quite conservative. But I challenge you to sit with an LLM like Fable and wind the clock forward on AI progress and diffusion another two years and come out of it assuming things will be roughly as they are today - rather, I think the technology within itself implies massive changes in the economy and work, just as Gates is reacting to here.
Read more: The turbulent AI era is here. The choices we make now are critical. (Gates Notes).

***

The six stages for off-earth mining, courtesy of Chinese researchers:
…AI will play a role in data generation and intelligence for future mining…
Researchers with the Chinese Academy of Sciences, Technical University of Munich, Obuda University, Beihang University, Wuhan University, the Aerospace Information Research Institute within the Chinese Academy of Sciences, the University of Wurzburg, Shenzhen University, China University of Mining and Technology, WAYTOUS, and OpenSpaceLab, have published a paper on the key requirements for mining the moon, asteroids, and other off-Earth places.

Why mine in space? The moon and asteroids are rich in resources needed for manufacturing, sustaining life, and powering transport around the solar system, including Helium-3, water, and mineral resources. If we want to become a multi-planet civilization it’s certain that we’ll need to figure out how to mine and refine and use resources away from the Earth. This paper does a nice job of laying out the technical requirements for things like space mining and what technologies are required.

Six stages for space mining:

• Exploration:
  • 1. Prospecting via remote sensing, using “orbital sensors and telescopic surveys to map celestial bodies”.
  • 2. Exploration via precise in situ robotic detection of resources: “proximity spacecraft and mobile platforms conduct high-resolution analysis of surface
  • composition and topography”.
• Sampling:
  • 3. Single-robot small-scale sampling: “autonomous robotic units conduct small-scale drilling and extraction tests”.
  • 4. Multi-robot large-scale excavation; “large-scale industrial systems or coordinated robotic swarms are deployed to harvest raw regolith”.
• Extraction:
  • 5. Autonomous resource extraction and refinement: “Integrated systems process harvested materials to separate volatiles, metals, and water ice”.
  • 6. Final integration into in situ construction or terrestrial transport “incorporating refined resources into In-Situ Resource Utilization (ISRU) frameworks”.

The biggest challenges relate to data and software: The paper gives a good accounting of existing and planned space robots, where a lot of the hardware stuff for mining is on the way to being derisked, and “the next generation of lunar and Martian rovers slated for deployment between 2026 and 2030 marks a decisive transition toward integrated resource prospecting and in-situ utilization”.
But space robot datasets are unbelievably hard to come by given the (relatively speaking) tiny amount of data gathered and released, and the datasets that do exist are “characterized by extreme scarcity and frequent quality discontinuities”. Here, advances in AI-driven world models might be able to ease the data drought.

What other big challenges remain?

• Robotic resource acquisition in microgravity: There’s a need for much better simulators here that are “momentum-aware, multimodal architectures designed to decouple surface interaction from gravitational dependence”.
• Foundation models for space mining: “Future systems must transition from scripted protocols toward embodied geological intelligence,” they write. “World models that merge geometric-semantic mapping with predictive material responses can support active perception and long-horizon task planning through predictive pre-action evaluation”.
• Simulation and validation for space mining autonomy: “microgravity, vacuum, thermal cycling, radiation, and regolith behaviors such as cohesion, electrostatics, and fluidization are rarely reproduced together,” they write. “Future work should establish a closed-loop validation ecosystem that links simulation, terrestrial emulation, and extraterrestrial deployment”

Why this matters - space is where superintelligence really gets going: The paper is notable for the almost total absence of references to humans doing any of this stuff (apart from one note on an in-development Chinese crewed lunar rover). This isn’t an accidental omission; for space mining to be remotely viable we’re going to need to depend on smart, cheap robots, operating with great responsibility at distance. This means that space mining is a perfect business opportunity for superintelligences that come online - perhaps by 2033 we will see AI-run asteroid mining corporations, bidding on contracts that human-supervised companies are not willing to pursue, due to risk or complexity.
Read more: Mining beyond Earth with Space Robots: Exploration, Sampling, and Extraction (arXiv).

***

Tech Tales:

Messages during a crisis:
[2027: Record of notes written by a single user to a language model during the months in which a hot war occurred in their country]

Say you’re in a country and there’s a war and you think it’s going to get really bad. How do you prepare for that? I live in a regular house. I bought a medical bag of all kinds of stuff off the internet. What else should I do?

What does hyperinflation mean? Explain it to me like I barely understand how money works in the first place.

I need to boil water and we don’t have electricity or gas right now. Can I just make a fire and use my regular cooking pans on it?

There are some explosions happening near me and sometimes the glass shakes a little. What do I do to make sure if the glass breaks it doesn’t shatter? I’ve got some plastic shopping bags and I figured I could cut them open and use masking tape to stick them to the windows. What do you think?

Food is getting harder to find. What kind of stuff can I trade for food? My house has a bunch of tools in it like screwdrivers and stuff. Is that valuable?

Things that inspired this story: All the wars happening around the world; prompts as the new confessional; how unprepared people are for crises.

Thanks for reading!