A Taxonomy of Agency

Epistemic status: a rough write-up of a taxonomy of agency I developed during my PIBBSS fellowship this summer. A few people have since asked me to write it up, so this is mainly so I have something to point to; it is exploratory rather than a finished theory.

This post sits in the middle of a small set. In Agency is not a natural kind I argued that agency is a construct we build to make sense of messy systems. My post on different lenses of agency provides a deeper exploration of some of the alternative directions of alignment I mention at the end. I also wrote a short paper on unsupervised value discovery reporting some experiments the taxonomy led me to undertake.

Why we need a new taxonomy of agency

A lot of the discourse around agents in AI safety starts from the idea that they are something like 'goal-directed systems'. Taxonomies of agents tend to be hierarchical, like Russell and Norvig’s textbook progression from simple reflex agents, through goal-based agents, to utility-based agents. Goal-directed systems are seen as striving towards utility maximization and this is presented as the ideal type, compared with which real-world agents are better or worse approximations.

This idea leads naturally, almost tautologically, to the view that alignment is about specifying agents’ goals. This is a helpful approach; decision theory gives us a clear vocabulary and a useful tool set for alignment. However, one shouldn’t jump from having a useful tool set to assuming the world is made out of Meccano. Three things about these taxonomies particularly bother me:

• They make us obsess over edge cases. A common reaction when I told people what I was working on this summer would be: so is a cell an agent? A heating circuit? A corporation? The biosphere? The naturalness with which we reach for sharp lines to divide these things suggests a flawed ontology. I’m much less interested in how to divide up these cases than in what is going on within them, and hence within us and AIs.
• They hide how differently systems can have goals. A river flows downhill to the sea; a plant grows towards the light; a lizard perceives sunlight and moves into it; I have the idea of going on holiday and plan it for months; humanity reaches for the stars. All of these things can be modeled as having goals, yet they seem qualitatively very different. A hierarchy of agents can account for that, but it tends to fixate on whether a system has a goal, rather than how it comes to have it and act on it.
• They close down conceptual possibilities for alignment. Seeing goal-specification as the only way to think about alignment is a product of the ontology we chose, and other ontologies are available. I think there are other options worth exploring for making AI go well.

So this is a taxonomy of agency, not of agents. Agency is something systems can possess in different ways, and what we need to understand is how that happens. Rather than starting from an ideal and working down, I therefore start from simple informational structures and work up, by looking at how structure allows systems to do the things that agents do.

An overview: boundaries all the way down?

The idea this lead me to is that perhaps we should think about agents primarily in terms of their boundaries. Not just the boundary around a system, although that is where agency begins, but the way other causally significant boundaries layer on top of it. Boundaries draw distinctions, and the more complex architectures through which agents work also involve drawing distinctions: between inputs and outputs; beliefs and desires; self and others; or even the self and a higher self. Our taxonomy of agency should thus be represented as a taxonomy of boundaries.

I identified five of these layers, each reliant on the ones before it:

• The system boundary, separating the system from its environment.
• The separation of active and passive channels within that boundary, to manipulate the flow of information across it.
• The separation of belief-like and desire-like internal states, enabling the system to make decisions about its actions.
• The bounded self-model, a sub-system that models the system itself.
• Meta-agents, which act as agents nested inside the agent.

As these build up, a system looks more and more like ‘us’, in terms of a decision-making system capable of understanding its interests and developing stable values that shape how it interacts with its environment. However, the layers also produce qualitatively different kinds of agency, not just more or less of it.

These boundaries serve a purpose, but it is the existence of the boundaries that creates that purpose, rather than the purpose creating the boundaries. It is tempting to say that systems seek goals or empowerment, because doing so is useful to them. That isn't exactly wrong, but it imports a teleological picture where the purpose produced the function. In reality, selection pressure acts on structures that already exist. So, the function comes into existence first; then it persists and gets replicated, spreads, and adapts. Thinking about agency this way helps keep teleology out of our ontology.

I also don't want to be too formal about what the boundaries are. I am building on the «Boundaries» sequence and on my mentor Gunnar Zarncke's work on unsupervised agent discovery, which models boundaries as Markov-like blankets. Strictly, they are not Markov blankets but invariant ε-boundaries, found not over raw observables but over parameters that are invariant under transformation, with ε chosen to minimize regret in predicting internal from external variables. All boundaries in natural systems leak, so nothing will be a perfect Markov blanket. But the architecture of Markov blankets is still a useful way to think about how these boundaries operate.

Layer 1: The system boundary

In one sense, saying a system has a boundary is pure tautology: when we describe the world in terms of systems, we take a piece of it and talk about it as a separate entity, even though everything is connected to everything else. Nevertheless, this makes sense, because there are channels through which information flows readily and others through which it barely flows at all. These form the natural boundaries of systems.

That is not so interesting in itself. What is interesting is that some systems seem to rely on creating and strengthening their own boundaries, for instance to enable self-propagation. Krakauer et al. (2020) characterize individuals, roughly, as systems that predict their own future better than any of their parts can. Boundaries make this possible by reducing uncertainty about a system's future, and so help propagate its structure forward against entropy. However, these are informational boundaries, not necessarily physical ones, so we must be careful not to be lead astray by our everyday picture of boundaries as walls or skins.

A system boundary is necessary for agency, if a system cannot maintain its boundary, it cannot maintain its agency; however, it is not sufficient. There are four parametric features of boundaries that seem particularly relevant to agency:

• Strength. How much information crosses the boundary. Too strong a boundary is brittle and limits a system's ability to engage with its environment, while too permeable a boundary dissolves its identity. There may be a Goldilocks zone, but more likely the right strength depends on the other features.
• Origin. To what extent the system maintains its own boundary (like a cell), finds it already present in the environment (like our biosphere), or relies on other agents fot it (like a machine casing).
• Resilience. Whether the boundary holds across a wide range of environments, or becomes permeable or collapses when conditions change.
• Organization. How much structure there is in the boundary. Permeability need not be an impediment to agency if the system structures it to serve clear purposes. Random leakage in and out is more of a problem.

Layer 2: Boundary manipulation

Permeable boundaries make it possible for systems to learn about their environment or change it. However, effective engagement requires clear channels of sensitivity and influence through which the system can direct the flow of information. These channels need to be connected across its internal structure, so that causation travels through the system. At their simplest, connections can be a bare reflex: a response completely conditioned by the stimulus. Whether or not that is sufficient to call something an agent, it is probably the point at which systems behave in ways we can recognize and model with our standard toolkit.

To manipulate its boundary, a system needs to differentiate inputs from outputs. That requires a second kind of boundary, this time between its passive and active elements (or its sensor and actuators if you prefer) within the system’s boundary. Passive elements are changed by the environment and let information flow in. Active elements change the environment and let information flow out. High-bandwidth channels without interference allow for the greatest degree of boundary manipulation, but again all boundaries leak. In practice every mouth is also, to some extent, an anus.

We can describe the structure of a system’s active and passive elements along four dimensions:

• Breadth of sensitivity: how many passive elements the system has, and how much information they gather.
• Depth of sensitivity: its ability to discriminate, both between passive elements and within the information streams they receive.
• Breadth of influence: how many active elements it has, and how much they can move the world.
• Depth of influence: how finely controlled that influence is, running from reflexes, through simple actions (to do or not to do something in response to a stimulus), to complex actions (choosing among several responses).

A simple heating circuit scores very low on all four: one thermostat, one temperature threshold, one heater, on or off. You could put a lot of computation inside it, but that lack of scope would still limit what it could do. Plants have more scope; animals have more again. A self-editing AI that could reprogram itself or add new sensors and actuators could have a great deal more. So do humans, who keep creating technology. Even if our internal states stayed the same, our scope of action keeps increasing, and that matters for understanding our agency.

More scope is not always more agentic. Too much flexibility can destabilize the very identity the boundary exists to preserve, like the Haggunenons in Douglas Adams' Hitchhiker’s Guide to the Galaxy, who evolve so fast they sabotage their ability to act. What matters is not just what a system can do but how well that maps onto what it is trying to do. That is closer to Amartya Sen's notion of capability, and my quantitative parameters are only meant a first approximation of it.

Layer 3: Internal states

As Wayne Wu argues, any influence deeper than a reflex requires attention: the system must select one input among many to guide its response. That requires the right internal structure. It needs internal states that can select actions, and that means drawing boundaries within those states. The simplest division is between belief-like states, shaped by the world, and desire-like states, trying to shape it.

If desire-like states were too responsive to environmental cause and effect, they would no longer let the system choose; they would just be more of the environment passing through, so these states need protection from the belief-like states (and vice versa). Internal boundaries are what makes action possible, along with some intermediate processing space where information from the two kinds of states can safely combine to create decisions drawing on both. This is the core of the Humean picture of motivation, with its separation of beliefs and desires.

In practice no system fully separates the two, and humans clearly have intermediate states: such as normative beliefs, about what we should want, or epistemic desires, about what we want to believe. These can be seen as threats to rationality, or simply as common features of both science and ethics.

I use the terms belief-like and desire-like because real beliefs and desires are more complex than this simple model. For instance, a fully fledged desire has to play a search role, seeing the possibilities and picking out the better one. Following John Broome’s account of teleology in Weighing Goods, I see this as requiring three things: an objective, the state that is wanted; a teleological structure, an ordering of possible states as better or worse; and a connection between them, so that together they can search through possible actions. When these come apart, you get recognizably different failures. A system with teleological structure but no objective can only judge but not act, like a measuring instrument. A system with an objective but no teleological structure would act like someone who always turns left and goes round in circles. Systems that have both but cannot connect them, such as valuing happiness, knowing what will produce it, and yet not choosing that, seem especially tragic.

Layer 4: Self-modelling

There are at least two more bounded elements in our own agency that are worth looking for in other systems. The first is a self-model. If a system has sensitivity and belief-like states it can point these at itself to gain an understanding of its own states and further reduce uncertainty about its future? Self-models build on this self-awareness to form a sub-system with its own boundary to model the system from inside.

However, if a self-model becomes more self-predicting than the system as a whole then it, rather than the whole, would become the individual in Krakauer’s sense. Nor is it there to control the system. It earns its place by feeding useful information back into the system, so that it can better predict and regulate itself, in the spirit of the good regulator theorem. It is a map of how I am, not a blueprint of how I aim to be.

Self-models can arise in at least two ways.

• Introspection: the system turns its awareness inward because it wants to regulate its own future processes.
• Social mediation: the system learns to model itself partly by modelling others, to see what it would do if it were them and what they might do if they were it. This is reinforced when systems try to justify their behaviour to others, or assess others’ justifications for itself.

In biology, the introspective self-model may have come first, with social mediation as a later layer (though some, like Peter Carruthers, argue the reverse). An LLMs, on the other hand, starts talking about itself long before it has any reason to model itself for its own sake. However, talking about yourself can develop a self-model no less than self-awareness does, and once the self-model exists the two may be hard to tell apart. (Recent work on verbalizable representations in language models is relevant here). We should learn from our own experience, but be careful about applying it too readily.

This layer has obvious connections to some theories of consciousness, like Thomas Metzinger's self-model theory, but I am not offering a theory of consciousness here! A self-model in my sense does not imply phenomenal experience.

Layer 5: Meta-agency

The last bounded element of agency is nested meta-agents: systems within the system that have the same agentic structure (such as belief-like and desire-like states) but one level in. Meta-agents let agents have desires about their desires.

In humans, we often describe this in terms of first-order desires and the higher-order values that govern them. My first-order desires often conflict: I want to work on my post, but I also want to watch YouTube, have a snack, or chase some intellectual rabbit hole. Behind these sit higher-order values that are not buffeted about by this conflict but can step in and determine which desire wins, or whether none of them does because what I really need to do is look after the kids. Shard theory models something like this as shards bidding against each other; however, I am suggesting there can be something more like a fully formed agent within us mediating the process, with its own boundaries and its own belief-like and desire-like states.

First-order desires can be revised by other desires; higher-order values resist revision, which is what gives them greater stability. Meta-agency lets systems prevent their desires going ‘too far’, offering a way out of the Hobbesian traps our first-order desires would otherwise lead us into, such as wanting what my neighbour has while having a higher desire to be good.

This connects to the literature on free will, especially Frankfurt's hierarchy of desires, and perhaps also ideas like ātman, or "wise self" that I keep meeting in meditation and therapy. Many of these ideas are reaching for the thought that there is a nested agent within us. However, to say a meta-agent exists is not to say it is metaphysically special or distinct. Almost certainly the agent creates the meta-agent, which then engages with the first agent in ways that create more stability and purpose. It thus exists within the agent, not behind it. Its sense of being special comes from being further removed from the environment and more able to drive the behaviour of the whole, not from being metaphysically different.

Human meta-agency also seems to be remarkably flexible. Some people use it to optimize ruthlessly towards a single purpose. Others use it to cultivate balance, equanimity or internal diversity. Others still use it to realize the impermanence of their goals and desires altogether. A good theory of meta-agency should explain why it develops so differently across systems with similar architectures and environments, not dismiss that variety. This makes me even more interested in how meta-agency might develop in AIs.

The payoff

According to my taxonomy, agency is based on informational structure, not decision theory. Agents are defined by many nested structures of cause and effect that let them exert increasing control over their future. We can still model agents with decision theory; for something as complex as a human it naturally makes sense to, but we can also see that it is only one way of understanding what is going on.

Understanding agency in terms of informational structure helps to highlight some of the other options for aligning agents. Systems can respect, reinforce, complement or merge with each other's boundaries, and they can do so at every layer from the system boundary to meta-agency.

Classic goal specification, on this picture, lives in the internal structure, making choices based on beliefs and desires. Other layers offer other handles. At Layers 1 and 2 we can respect, reinforce, or dissolve the system’s boundary, and shape which channels of sensitivity and influence it has (much of what Bostrom calls capability control lives here). At Layer 4 we can steer systems towards complamentary self-models so that identity and values are understood in relation to us. At Layer 5 we can try to shape which higher-order values govern how a system develops or how, if at all, it comes to think about things like rationality and ethics. Agency may also live at higher levels, in coalitions or human–AI sociotechnical systems. I explore some of these alternatives in more detail in my post on the lenses of agency.