Concerns About Personas, Multi-Agent Alignment, and Role Theory

There’s recently been significant discussion of LLM personas and how those fit into alignment. The idea is that LLMs have many possible basins of behavior, a la masks on a shoggoth, or more appropriately, faces of a high dimensional polygon. If one of those is a persona that is robustly aligned, that achieves alignment. It seems that the logic is that once we have one such persona we can use it to bootstrap further research, using the aligned persona to build more broadly safe systems, or to oversee other systems.

This neglects how personas work for people, and I expect that because LLMs are in some sense emulating or attempting to act as people[1], it should fail in similar ways. But to make sure this is clear (at least to myself, so others can tell me if I’m wrong,) I’ll explain what I understand about role theory, and then add some speculation about the game theory. After that, I’ll go back to AI personas - which is what we call roles in LLM-land - and land at why this creates new multi-agent alignment problems.

What is Role theory?

An introductory explanation says that “role theory is a framework in social psychology and sociology that posits that much of interpersonal behavior can be understood through the lens of socially constructed roles. These roles, such as teacher, parent, or child, exist independently of the individuals who enact them, serving as a guide for how people engage with society. Each role carries specific expectations for behavior, which shape how individuals act in various contexts[2].”

That is, the context of a given role is embedded within society. And the roles are not exclusive; a typical person can act as a spouse, a parent, a friend, and an employee over the course of a day. They behave differently in each context. This is supposed to explain how and why people are, say, confrontational with co-workers, accepting or passive with parents, and cooperative with friends. (This is compatible with shard theory, which posits that people have different motivations and goals that conflict with each other, but role theory focuses more on behaviors rather than internals. Still, if we stretch this, I think it also plausibly explains why they are incoherent in some ways, if they play different roles depending on context. If they are asked to pick what to eat for lunch in 20 minutes, they have an internal “role” that differs from when they are asked to pick what to eat next week.)

Why would people do this? The simple answer is that they are not single coherent actors, and that might be true - but we expect there to be some pressures, perhaps societal, or evolutionary, that push people towards coherency. My attempted answer, which I suspect isn’t novel, is that there are game-theoretic reasons to be inconsistent in different roles. Some situations are competitive, and if you’re boardgaming, it makes sense to take on a more assertive or confrontational stance. Other situations are, or should be, almost entirely cooperative; marriage and family isn’t a competition. Yet others are cooperative, combining competition and cooperation, and those require a different approach.

Roles and Groups

And the set of roles someone can take isn’t one-dimensional competition versus cooperation. A company needs a lawyer and an accountant and an IT team, and a social milieu probably needs a good mix of organizers, listeners, hard workers, comedians, artists, and many more. But these roles aren’t fully interchangeable, and can converge into structures that are hard to escape. The person who was leader last week is expected to be a leader this week as well, and the comedian may have trouble shifting modes to get others to take them seriously. So roles can lock people in, especially while still embedded in a given context and over short time periods, and this is often not what the individual wants. The school bully might be miserable and want to change, but acting nice for a week or two won’t change how others perceive them, and not fighting when challenged could end up with them strictly worse off than they were before. The social dynamics are structured so that changing their role within that context isn’t always possible.

On the other hand, the role isn’t fixed across contexts. Kids starting at a new school with new friends have an intuitive appreciation that they could change who they are and start over in ways they can’t when still around the same people. And everyone has the experience of being treated differently by their parents, friends, and co-workers, so that they can or need to fill different roles in each place. The bully in the school classroom might be a cooperative and obliging basketball player, and a valued and respected leader in their Boyscout troop - as long as the people they are with don’t overlap too much.

The informal game theory here is that in each context, there will be different equilibria, with different people in different roles. Even when a game is symmetric, dominance hierarchies can emerge. That shows a simple case where it is better for an agent, or person, to play their expected role well, even when that role is different in different scenarios. And this manifests even more clearly in more complex interactions; if you’ve ever played MMORPGs, the same character can be a paladin healer, or a paladin tank - but the rest of the party needs to know which one!

LLMs, where Roles are Personas, and Context is Roles, and Context is…

It turns out that in AI-land, no-one asked the social psychologists what to call things, so terminology doesn’t match. Personas, for LLMs, refers to something very similar to what we’ve been calling roles. Roles, for LLMs, refers to the different types of inputs and outputs inside of their context - as was beautifully explained (and exploited) recently in A Mechanistic Explanation of Prompt Injection (and why you should study roles)[3].

LLM personas, on the other hand, are masks of the shoggoth, or to use a possibly more fruitful analogy, faces of a (high-dimensional, poorly understood) gemstone. In shoggoth-language, there are lots of masks that the LLM can wear, and these are constructed and exposed via fine-tuning and the system prompt. But there isn’t just one mask, and other masks or personas can be exposed accidentally or via clever prompting. And there’s definitely a need for more model persona research.

As an aside, in my analogy, fine-tuning and prompting is gem-cutting, ripping away parts of the unfinished surface of a pretrained LLM so that, when given inputs, the way it is refracted inside of the model, and thus the output, differs depending on the facet which is exposed. Fine-tuning doesn’t fully change the core of the model, it just warps behavior[4]. And the way fine tuning warps behavior has been argued to largely result from persona selection - not even altering the personas. Other fine tuning has been shown to ‘construct’ a persona (LW Post) - but it seems this is some combination of extant personas. So if the personas are accessible or inaccessible facets of the gem, fine-tuning may be akin to rotating which face the user sees, rather than carving a new one.

Personas in multiagent settings?

Currently, the vast majority of use of agentic AI systems is at most delegating tasks to near-identical agents; they may choose the model variant, or the thinking depth, but they aren’t constructing a team with different aspects and different strengths. Jason Crawford noted that “In human teams, much effort is spent training new employees up to baseline competency and then maintaining their performance…A team of AI workers will be different: they will all be copies of an already-trained model, which will execute that training consistently and never quit.”

But this is a weaker vision than what is possible, given what we’ve said above. Different models have different capabilities, and the simplest ways of leveraging that have already begun to emerge. For example, Multiplicity AI provides results from several LLMs for users to compare, and the “Mixture of Agents” approach which uses a synthesis of agents’ responses, though the most prominent example “showing” success of this method was publication by press release, rather than something solid. In any case, this is definitely progress - but it’s a very weak approach, akin to trying to run a company by taking each decision, soliciting staff input, and letting management pick the best answer, or at best, try to combine the ideas.

Actual companies, of course, have a structurally better approach; they have expert teams with diverse understanding that debate internally, propose options, and iterate. And AI agents already do exactly that internally[5]. This is presumably taking some advantage of their internal mixture of experts, which each presumably have some variation in personas across the tokens, since the router will vary where to send them[6]. But this isn’t robustly delegating tasks to experts with appropriate skills, it’s still using a generalist.

We haven’t seen the emergence of multiagent harnesses that take substantial advantage of this yet, but I suspect we’re very close. For now, we have agentic systems with predefined role-play, and we should expect this will evolve.

Emergent Roles and Multiagent Dynamics

The reason this will evolve isn’t just because actual roles in dynamic social systems are emergent, rather than fixed. It’s because even in necessarily top-down corporate structures, which are needed for scaling cooperation, you end up with emergent extitutional structures (explained more in this paper). We see this emergent behavior happen to some extent with work by Janus involving groups of agents, each controlled by different models interacting[7]. But exactly for this reason, I suspect current approaches are high-modernist central planning; trying to do top-down assignment of agents will hit a wall, and they’ll need to move towards something like internal markets for capabilities. This would touch on Mark Miller and Eric Drexler’s Agoric Systems, albeit more sociologically, but such solutions allow orders of magnitude more agents than current approaches.

This is trying for emergent solutions to clearly pre-defined targets, optimizing over spaces for useful ways to do something the orchestrator can check. Which brings us back to traditional alignment theory - because putting lots of agents together to pursue well-defined goals is going to go very badly unless you get the goals exactly right. But that’s where traditional alignment problems start, leaving us without a solution.

But that’s not how emergent social dynamics work. Instead, emergent human roles are themselves creating a situation where results and incentives emerge rather than get imposed top down. Analogically, this is concerning; individual level behavioral or empirical guarantees about LLM behaviors are conditional on the persona they are exhibiting. To the extent that LLMs are effectively emulating people, they will exhibit similar dynamics - that is, the personas will be shaped by other agents’ expectations and behaviors over time.

This does not mean the personas that work well will be humanlike; Vince Conitzer has noted a variety of reasons that “AI agents may cooperate better if they don't resemble us.” But LLMs are more humanlike than the agents he was discussing; more recent work from his lab tries to evaluate how well cooperation works in LLM agents, and shows some of the mechanisms being used - but all of our concerns in the earlier footnote about evaluations apply.

Emergent Multiagent Misalignment and a Multiagent Alignment Tax

We’ve just speculated that multiagent social dynamics will emergently drift from the regimes where guarantees were built. This doesn’t guarantee emergent misalignment, but it does weaken any guarantees intended to guard against it. Perhaps an obvious objection is that we can stop this by freezing behaviors, trying to keep agents to their original settings and preventing drift. And yes, methods like persona steering and resetting individual agents instead of long-task persistence should prevent each agent from drifting. But in addition to locking agents into a fragile role, and being objectionable from a model-welfare standpoint, it abandons all of the advantages we were supposed to gain from allowing diverse teams to emerge. That is, there’s a significant alignment tax embedded in the entire approach - we either need to pay that tax, and abandon most of the benefits from diversity and emergent behaviors, or abandon the guarantees we had for individual agents.

We’ve concluded that it’s not enough to find a robustly aligned persona and bootstrap, because this loses the advantage of adaptive agents. So finding an aligned persona for each of the diverse agents is a starting point. It’s not enough because persona stability is a product of social context, so there will be consistent pressure for those personas to drift. Fable suggested an analogy where “Verifying alignment of a persona in the lab is like verifying the bully is kind — at Scout camp.” That is, not only are personas contextual, but persona stability is contextual as well.

The contextual stability does provide a plausible avenue towards a prosaic alignment solution, where personas themselves seem to drift towards stable configurations; repeated interactions may lead to stable points in repeated games. The challenge for leveraging this is ensuring that the emergent dynamics lead to the specific stable points that stay within the region of prosaic alignment. In other words, we need to solve the mechanism design problem for each context, and where the personas we’ve tested and steer towards are not just safe but selected for, which is far from trivial. In fact, it seems to require predefining the roles. That’s harder than simply paying the tax, and potentially more costly as well. We can sometimes build mechanisms for coordination and cooperation in economic systems, and there is work in that direction, but doing so for social systems is far harder, and the best humans seem to manage is building moral mazes. Perhaps a superhuman mind could solve this - but it doesn’t seem like a nicely framed NP problem where we can verify cleanly if a solution exists.

I’m left at loose ends; I think this illustrates how and why the challenges of multiagent alignment are likely to arise, and are also difficult in ways that aren’t addressed by most current approaches. That’s not to say progress isn’t possible, or that progress on single-agent alignment won’t (necessarily) be valuable. In fact, it seems like success in the single agent case, which seems plausible in the prosaic alignment worlds, is necessary, but insufficient. And given the way agentic harnesses are already being deployed, and long-running cohorts of agents are already causing problems, these challenges are critical already.

Note: Fable used to give conceptual feedback on a completed initial draft, but except where directly quoted or where it suggested links to other literature that is linked, LLMs were not directly involved in any of the writing. Thanks to Anders Sandberg and Edo Arad for other feedback.

1. ^It’s building a text generating process, which includes both pretraining to be person-like, and post-training to modify that - but even the latter is trying to construct some set of behaviors as a sort of synthetic person
2. ^This is not fully fair; there’s lots of work on role-making and collaborative construction of roles, and there is an important contribution from identity theory, but hopefully it’s a useful first approximation for those who are unfamiliar. A more recent and comprehensive treatment on the bases of identities in social contexts is Identity Theory and Social Identity Theory. (From 2000, but most fields don’t move as fast as AI.)
3. ^“How does an LLM know the difference between its own thoughts and someone else's words?... [The context window] contains everything: system prompts, user messages, tool outputs, the LLM's own previous responses and reasoning… So how do [the LLMs] impose structure on the token soup? ...the soup is interspersed with role tags: , , , , [1], which partition the string into labeled segments.”
4. ^This isn’t fully fair, but I think it’s a reasonable approximation; we don’t expect to see phase transitions and fundamentally improved capabilities when fine-tuning, we expect to see better behavior and capability improvements from better applications of methods and tools. To again lean on a simplistic analogy, this is akin to teaching the model how to do multiple-choice tests, or how to use a calculator to improve their success rate, or how to do a better job checking their work - rather than teaching them new material, much less improving their IQ.
5. ^It may be partially because evaluations are used across models, which pushes for convergence rather than differentiation. But note that evals are effectively the most intensively fake environments for LLMs, and any alignment via evals and hill-climbing is obviously also very fragile. And that makes sense, and aligns with role theory in that this is a standard sociology objection to research built on inferring roles from context-stripped laboratory settings.
6. ^This isn’t obvious, and could be wrong; the question of how experts contribute to outputs is still an active area of investigation.
7. ^This was more specifically showing lock-in via consistency, which is emergent but not as specifically multi-agent.

Discuss