Reviewing my 2017 AI Risk Project

In 2017, after getting laid off from a startup, I spent a while looking into AI risk:

Many people I respect have started to think that AI risk is the most valuable place to focus EA efforts. For example, 80000 Hours ranks it first on their "list of global issues", taking into account "scale, neglectedness, and solvability". On the other hand, I have a lot of friends working in machine learning, and none of them think AI risk is worth working on now. This level of disagreement is very strange, and kind of worrying.

After playing for a dance weekend in South Carolina where casual conversation among non-tech people included earnest speculation over the likelihood that AI gets us all killed, I thought it might be a good time to look back at when this was really speculative.

Summary: I talked to the right people, heard some incredibly prescient ideas, and got the bottom line wrong.

I do a lot of my thinking via serial writing, and in this case that was a series of twelve blog posts. Taking them one at a time:

Looking into AI Risk (2017-06-26).

This held up well. I picked something well worth looking into, identified a lot of relevant resources, and got a lot more good reading recommendations in the comments.

Superintelligence Risk Project (2017-07-03).

The main part of this post were five ideas about where the disagreement might be. These worked ok as jumping off points for interviews, but they weren't as core to disagreements as I hoped: people often ended up with similar views on most of these ideas while still being far apart in their views on how we should prioritize AI risk.

One thing I do like, though, is that these were pretty concrete questions which now do have answers:

How likely is it that current approaches are all we need for AGI, with relatively straightforward extensions and a lot of scaling?

Very likely. When I asked this question, "current approaches" meant deep neural nets and reinforcement learning. The past nine years have mostly been an incredible amount of scaling.

How valuable is it to work on solving problems that are probably not the right ones? For example, even if we think AGI will not look like current systems, might trying to solve the control problem for current systems teach us enough about the underlying problem and how to do this kind of work that we'll be in a better position once we see more what AGI will actually look like?

Looking at the two best-known approaches in 2017, I'd say Concrete Problems in AI Safety has held up extremely well, while Highly Reliable Agent Design (contemporaneous review) less so. This makes sense, since the paradigm of the former (scale current things) is what happened, while the latter assumed we'd get to powerful AI systems via explicit systems with world modeling, decision theory, etc.

Still, even the HRAD work did end up being pretty useful, by getting people thinking about important problems, naming things, building a field, and getting people spun up.

So I'd say work was pretty useful even if you didn't anticipate the shape of today's AI.

How useful is it to have a strong theoretical foundation, vs just understanding the technology enough from an engineering perspective that we can make it do things for us?

In terms of building powerful AI systems, it seems like it has been near 100% engineering. On the other hand, we now have powerful systems that no one understands. As they get increasingly capable will they continue to be compatible with the continued existence of humanity? Hard to say! Seems like a strong theoretical foundation would be pretty useful.

How similar is this to normal engineering? How much should we expect companies' desires that their AI systems do what they want them to do to work out?

Mostly yes, but more importantly no. Companies want their models to be economically useful which means doing what users want. Except it turns out you can provide a ton of economic value while still occasionally being seriously misaligned, and there's a real chance that as we continue to scale this misalignment is going to work out very poorly.

As we get closer to AGI, how likely is the ML community to take superintelligence risk seriously? Is it just that they don't think it can be productively worked on now or do they not think it will ever be a real problem?

People are taking this increasingly seriously, and I think this has probably gone better than people in 2017 would have predicted. But it doesn't look to me like they're taking it seriously enough: the majority of work still looks like forging ahead very quickly despite a lot of evidence of misalignment even at current scales.

There's also a bit in the post where I talked about applying for a grant to work on this. That was silly, since we had enough savings that this was something I could just choose to do. Looking back, I think the reason I applied was that I had an unrealistic view of grantmakers as people with an excellent global view of what needed doing, and where someone deciding whether to give me a grant to work on this project would be a strong signal of whether the project was worth doing.

(And then perhaps my most prescient decision was calling it "Superintelligence Risk" instead of "Artificial Intelligence Risk", anticipating executive order 14434.)

2017-07-04: Conversation with Dario Amodei

Alan Kay said "the best way to predict the future is to invent it," which certainly seems true here. This description of the situation feels super prescient. Unfortunately I never thought to ask the most important question: might AI reliably doing what we want end up being a limiting factor on AI progress, where this line of safety research might accelerate AI development and put us in what Amodei then considered the riskier "short timelines" path.

2017-07-09: Conversation with Michael Littman

Littman essentially believed that prosaic AGI, which Paul Christiano described as replicating human behavior without "qualitatively new ideas about 'how intelligence works'", was impossible. But that's pretty much what happened! So in many ways it's a big miss. I don't want to single Littman out, however, since in this series he's basically functioning as a stand-in for the mainstream ML community view at the time. Most serious professionals seemed to think this, and even Amodei and Christiano weren't confident this was how things would go.

2017-07-10: Superintelligence Risk Project Update

A reasonable summary of where I was so far, but nothing especially right or wrong in hindsight.

2017-07-11: Technical Distance to AGI

I wrote:

I think the largest piece of the disagreement between ML people over how to prioritize AI risk may turn out to be: how likely is it that we get AGI soon that looks a lot like systems we have now?

I think in retrospect this disagreement was indeed key, though I did get a lot of pushback in the comments from people who thought it didn't capture what was important.

A different line of disagreement in the comments was over whether ten years was a plausible timeframe for large amounts of AI progress at all, which turned into me betting Dave $100 to his $900 that we would have driverless cars in MA by 2027-07-12. Nine months out, the technology is ready (Waymo could safely do it next month if they wanted to) but I expect the legal process here to be too slow.

2017-07-13: Examples of Superintelligence Risk

I think this also held up decently well: better examples really were important. We got light versions via warning shots (especially the Hugging Face attacks) and people have since also written up some ones that give a plausible story end to end.

On the other hand, some of the "sci-fi" elements that made stories feel unrealistic, such as connecting powerful untrusted AIs to the internet or ai-enabled bioattacks, seem a lot more plausible than they did in 2017.

2017-07-14: Conversation with Bryce Wiedenbeck

Pretty similar to my conversation with Littman: Wiedenbeck did not think prosaic AI was possible, but it was.

2017-07-18: Superintelligence Risk Project Update II

Another status update. One interesting part:

I found Open Phil's notes on Early Field Growth interesting, especially their section on failure modes in cryonics and molecular nanotechnology. My takeaway was that heavy popularization of a new field prior to scientific success leads scientists on the border of the new field to take on an oppositional stance. The field gets starved of people who could do substantial technical work, makes minimal progress, and I think people also avoid the areas around its edges, like a chilling effect. I see superintelligence risk as just on the edge of this, where it could go either way.

It looks to me like AI safety came close to this, with some amount of popularization producing opposition, but it didn't get starved technically. I think because there was enough scientific success to keep it interesting and active.

2017-07-20: Conversation with an AI Researcher

This one is the most interesting mainstream ML interview to me, because they did foresee a lot: "they see progress in ML as almost entirely constrained by hardware and data." On the other hand, it does look to me like the differential technological development strategy they didn't think would have much of an effect has actually worked ok: having a few extra years of progress on various safety-related efforts looks pretty helpful to me. (Still an open question whether it has been enough, but that's not where this researcher disagreed.)

2017-07-27: Yudkowsky and MIRI

I didn't feel great about this post at the time, and still don't like it: it's mostly holding someone's early writing against them, where the most recent of the writing was ten years old by the time I was evaluating it. I agree with people who said this writing shouldn't be seen as much evidence either way.

Looking back, I'd say Yudkowsky has clearly been right about a lot of the risks involved, while at the same time (again, like most people) not anticipating specifically how we'd get to present levels of AI systems.

2017-09-15: Superintelligence Risk Project: Conclusion

This one is a painful read for me. In some ways it gets so close: the "Background" and "Differential Progress" sections are solid, I think I was right to be pessimistic about the HRAD approach, and then I totally flubbed the "get the right answer" part. I had an incredible opportunity, with Christiano, Amodei, and others giving thoughtful and detailed comments, and it wasn't enough. I'd like to give a clean story here, where I see what reasoning error I made, and how I'm going to avoid it going forward, but I really don't.

I'm still glad I did this project, and I think getting these views written down and shared was valuable. But getting so close and then missing the key conclusion does sting.