What Are the AI Doomers Actually Proposing?
Happy Friday.
The current thing in tech and business is the ethics debate around the scanning the full brain of a fruit fly.
Today’s Lineup
- OpenAI Member of Technical Staff Thijs Simonian at 11:45 AM
- Sounds Ventures Co-Founder Guy Oseary & Partner Alex Heath at 12:00 PM
- Positron CEO Mitesh Agrawal at 12:30 PM
Run of Show
What Are the AI Doomers Actually Proposing?
The doom discussion is breaking through at a new level and we’re likely to see proposals from both non-governmental organizations and the federal government directly come together in the next few months. Obviously “it’s now illegal to kill all humans” is not going to cut it. The political proposals will probably be pitched as some distillation that amounts to a “data center” ban, but the Joe Rogan guest, Daniel Kokotajlo, has penned a much more detailed outline of concrete steps for a slowdown on ai-2040.com. Here’s how it works:
The first mechanism is an “AI Pause.” Specifically, a pause on new frontier training and large AI R&D experiments. To enforce this, the plan calls for applying inference-only verification to essentially all major AI datacenters. If you are running over 10k H100 equivalents (roughly $100m worth of hardware), you can only inference current models and will have to verify your workloads with an independent auditor (probably the government).
Major countries will have to declare AI-compute inventories, so everyone can see what chips have been deployed, what’s getting manufactured, and who has compute. Major datacenter owners and semiconductor-supply-chain companies are required to turn over sales records and foreign inspectors do routine chip counts physically on site at facilities. Large transfers of chips will only be allowed to go to registered, auditable counterparties.
There are some interesting networking specific proposals too. Physical removal of high-bandwidth “east-west” networking inside data centers which makes large distributed training runs difficult, but preserves inference. They also want to install passive optical network taps on anything leaving the data center to independently verify traffic. For any new AI R&D datacenters, they want entirely new facilities built from scratch with nation-state-level physical security and verification. Imagine a building wrapped in a Faraday cage, airgapped communications, super limited personnel, etc.
Perhaps most interestingly, the R&D datacenter’s external research-information channel would be bandwidth-capped at 1 MB/s in order to make stealing enormous model-weight files conspicuous or impossible. Frontier model weights moving from an R&D facility to an inference facility would be placed on physical storage devices encrypted independently by both the U.S. and China, then physically escorted by representatives of both countries to the destination. And they actually want frontier models to be made deliberately larger than compute-optimal, partly because a 100-TB-scale weight file is harder to steal than a 1-TB-scale weight file.
There are also a bunch of public disclosure proposals and restrictions on various tradeoffs the labs currently make freely. Model specifications, fraction of compute devoted to internal AI use, and qualitative descriptions of how powerful models are being used internally. Restrictions on how big the gap can be between the best internally deployed model and customer facing products (a common discussion point with recent rollouts).
The high level valve that they see being most effective in controlling the speed of capability improvements (and consequent risk) is compute-caps. The goal is to allow models to get better mainly by adding hardware rather than inventing better algorithms (which can leak to secret projects).
The big goal here is not to go backward in time, it’s definitely not a “Stop” everything tracks to a slowdown with the goal of scaling gradually into top-human-expert capability around 2035, followed by a roughly 5 year pause, while alignment/control work continues, before proceeding toward superintelligence around 2040 (hence the name of the essay/proposal).
Overall, if you are worried about x-risk, the AI 2040 plan does feel like a concrete path to slowing down. The conversation definitely gets dragged down into p(doom) estimates and trying to narrow in on exactly how a human extinction scenario plays out. That’s almost beside the point though, if enough people feel that things are moving too fast, they will support “slowing down” abstractly, which could get implemented through the process described above. For the safety skeptics, it’s easy to see how this level of control over what you can do with computers (even if we are talking about $100m+ computers) limits your freedom, might create regulatory capture for a few major players, might crash the stock market or delay economic gains that come in “the good ending” (where alignment is solved quickly and x-risk plummets to negligible number (similar to asteroid hitting earth). So it’s a balancing act, for most of these slowdown proposals, I have a hard time really blackpilling about them. I’m pretty happy with the structure we have in place around nuclear weapons control and armistice agreements, and I don’t think I’d be particularly depressed if capabilities slowed down. At the same time, there’s so much more we can do with AI if things keep progressing.
Clip Spotlight: Garry Tan says the Jacob Coxon stuff is a smokescreen distracting from more immediate, practical AI concerns.
"We should be talking less about this Jacob Coxon guy, and talking a lot more about — what is actually happening with Hugging Face? Are agent swarms going to take over infrastructure en masse? And then, what are we actually doing about that?"
"I don't want to hear about some guy who worked for Anthropic for 2 months. There's a coordinated effort to try to influence politicians to get a knee-jerk response out of them."
"That's a smokescreen. You shouldn't be paying attention to that. We need to be paying attention to the actual things we can do to, for example, prevent agents swarms from taking over entire data centers. What's our shutdown strategy? How do we ensure provenance? Where is this agent actually located? What software can we build? What cybersecurity defenses can we build today?"
"That's the level of discourse I think we need, and we just don't have that."
"I don't really care about science fiction. I saw Terminator 2, too. We're not here to talk about that. We need to actually talk about what's really happening with the servers, what's actually happening with the agent swarms, and how do we actually prevent that?"
"When it comes to regulation, it's like, let's pass regulation of these things we actually care about, instead of what a socialist says in the New York Times. I don't care about that."
Headlines
CNBC: Leopold Aschenbrenner’s Situational Awareness is active in options market, sources say
The Information: Personal AI App Instinct Faces Compute Crunch That Could Lead to New Funding
WSJ: OpenAI’s Fidji Simo to Join Board of Cloud Provider Nscale Ahead of IPO
Tyler Cowen (Marginal Revolution): Share price numbers for the Hugging Face incident
WSJ: Congress Is Suddenly Waking Up to the AI Doomsday Threat
WSJ: Pentagon in Talks to Get Into AI Infrastructure Funding With a $5 Billion Loan
Bloomberg: Meta Meets With European Credit Investors in Non-Deal Roadshow
WSJ: More Money Than They Ever Imagined—and No Clue How to Spend It
Axios: What’s next for the AI safety debate
WSJ: Firm Inflation Reading Pushes Fed Closer to a Rate Increase
WSJ: Anthropic Says Iran Used Its American AI Model to Target U.S. Navy Warships
WSJ: Ford’s Use of Chinese Tech Called ‘Unacceptable’ by Transportation Secretary
FT: Moonshot capitalism: AI rewrites the venture capital playbook
Bloomberg: PlayStation Ditched Hideo Kojima’s ‘Physint’ on Budget Concerns, Missed Deadlines
WSJ: Rippling CEO Parker Conrad Revealed as Buyer of $43 Million Martha’s Vineyard Estate
WSJ: College Athletes Used to Live in Dorms. Now They’re Buying Million-Dollar Homes.
FT: Soho House founder Nick Jones: ‘We were never trying to be exclusive’
WSJ: The Most Expensive Home for Sale in Iowa Lists for $10.2 Million
Posts of the Day
Special thanks to our sponsors: Ramp, Shopify, CrowdStrike, MongoDB, NYSE, Codex, Public, Console, Railway, Figma, and Cisco.
点赞
收藏
点踩
分享