Foundational premises of advanced AI

My goal here is to establish a shared baseline (or model) for thinking about AI. I've found that disagreements about AI policy/governance/alignment will often trace back to unstated divergence on base-level facts.

A. Machine Learning - how can AI models do things we didn't program them to?

1. Modern AI models are not programmed behaviour-by-behaviour. Engineers write the code that governs the process by which a network of parameters finds associations between data that it is given ("learns"). A model's behaviour is learned from data and feedback rather than explicitly coded.
2. In this way we can produce useful ("intelligent") behaviour without knowing how to program it directly.
3. Machine learning is a deceptively simple mix of: algebra and huge amounts of data. This enables AI models to improve at predicting patterns in training data.
4. The process is akin to teaching through trial & error. Where a lot of clear examples are used, AI models perform very well.
5. Anything that can be measured (even poorly) can become the basis of training an AI model (i.e. specified as an objective or provided as a feedback signal).

B. Scaling - why might AI models continue improving quickly?

1. AI Models improve with better data, compute, or algorithms (the AI triad).
2. We don't know the upper limits of current architectures.
3. The "bitter lesson" in ML is that more compute tends to beat attempts to hard-code human knowledge.
4. In the short term there are practical (physical and economic) limits (e.g. how quickly data centres can be built, sourcing and connecting chips, power, land, labour).
5. There is evidence that progress in some important AI capabilities is accelerating. Whether and how long this continues is uncertain.
6. AI developers face commercial, geopolitical, and reputational incentives to quickly release improved models.
7. AI developers face costs/delays with implementing safety measures (while their benefits are externalised).
8. Currently there are significant incentives to develop increasingly powerful autonomous AI systems (i.e. systems that can operate independently to achieve goals).
9. Advanced AI systems are now capable of contributing to AI engineering and research.
10. Improvements in AI's ability to automate AI R&D may create a feedback loop and further accelerate the rate of AI development.

C. Capabilities - what can we know about what advanced AI can do?

1. Unlike previous technologies, LLMs have demonstrated capabilities which can substitute for human cognitive labour. The extent to which this applies has significant impacts on the ultimate impacts of this technology.
2. Digital systems can be duplicated, at scale, and operate at speeds far exceeding humans. So a sufficiently capable model could supply enormous amounts of cognitive labour.
3. As models improve they demonstrate capability gains that are uneven and unpredictable. Models can exhibit extreme proficiency in some things and inability to do others (capabilities are "jagged").
4. Some advanced capabilities are relevant to usefulness and also to risk. For example: long-horizon planning, tool use, cyber, persuasion, autonomous action, persuasion.

D. Agents - how do things change when AI systems can pursue goals by themselves?

1. Advanced AI systems have the ability to plan, act, and adapt. They are therefore increasingly used as autonomous agents (rather than a passive chatbot).
2. AI systems can cause harm.
3. Harm can be caused by: (i) deliberate misuse, (ii) accidents and failures, (iii) structural/systemic effects, including concentration of power, labour displacement and institutional dependency; and (iv) loss of control over increasingly capable autonomous systems.
4. The likelihood of these harms depends on model capabilities/propensities and the manner in which AI systems are deployed.

E. Safety - why can't developers stop danger?

Capability uncertainty - what can the model do?
Propensity uncertainty - what will the model do?
Control uncertainty - can we reliably make the system do what we want?

1. We do not yet have methods that guarantee advanced AI systems will behave as intended under all conditions.
2. The methods we do have only reduce particular risks (that we know about).
3. Because model capabilities and propensities aren't programmed, we cannot confidently predict how AI systems might behave in novel situations.
4. Measures of model capability are imperfect. Performance on benchmarks and evaluations do not track cleanly to real-world performance.
5. Evaluating an AI model's propensities ("behaviour") does not necessarily establish how it will behave in other contexts: models can behave differently across training, evaluation and deployment conditions. Evidence of reward hacking, alignment faking and deceptive behaviour illustrate this problem.
6. It is estimated that, in 2025, there were only ~600 FTEs working in technical AI safety, and ~500 FTEs in non-technical AI safety. Compared to ~12000 FTEs in frontier R+D (technical) and 1.2 million FTEs across the frontier model training pipeline.
7. The expertise, compute, data, and resources required to study frontier AI models and systems are concentrated in the companies developing them.

F. Governance - why can't existing institutions manage these issues?

1. Because the expertise and resources are concentrated with AI companies, society and governments have a greater level of uncertainty about frontier AI model development (i.e. there is an information asymmetry).
2. Risks may develop and arise faster than laws, institutions, and organisations can adapt.
3. Substantial gaps exist in our ability to understand, evaluate, control, and govern advanced AI systems.