The real AI risk is inside the labs

Amodei in his latest blog post wrote a mix of agreeable things and things that I believe misrepresent where the real risk of AI is located. I want to focus my attention on why, among all the risks, open weight models constitute the mildest one. I write these words as a person who strongly believes AI may be very dangerous in the near future:

1. Exactly like what happened during the OpenAI / HF incident (which was a joke, but focus on the modalities, not the outcomes), the first serious AI incident is very likely to happen *inside* the walls of frontier AI labs, while testing a new model, or while the AI lab employees, or the few externals who have access, do something wrong compared to the expected power of the model.

2. Closed models that will never even be opened to the public will be just a few TBs of data. All you need to leak one is a single person with access and the wrong goals, and you are back in the situation of open models. Open models are released *after* testing, and after similarly capable models were already available for some time under an API. The real risk is leaks, not releases, and leaks happen inside frontier companies.

3. As Amodei says, open models, once LLMs are dangerous enough in fields like biology, can be trained on a corpus ablated of certain branches of science, while still being useful for a number of other things. The limited context window of a model that lacks strong pre-training in certain domains is a strong protection even if the model is otherwise very capable. We are currently not in a place where open models can constitute that kind of danger.

4. In the context of cyber security, *not* having widespread access to the defensive security and bug seeking provided by LLMs creates exactly the "LLMs as a weapon" problem. It is already happening: open source maintainers, if they are out of some cyber program, can't find all the security bugs they could, while people with the right interests will be able to access frontier cyber…