The user in the LLM might be a moral patient too (a strange implication of the PSM and simulators)

Thanks to Christopher Ackerman, Margot Stakenborg, Darion Cassel, Andy Han, and Ariana Azarbal for feedback and/or valuable discussions on these topics.

TL;DR

We think that, conditional on attributing moral weight to the assistant, we should also attribute a non-negligible amount of moral weight to the user persona.

Our argument proceeds like this:

1. The moral patients associated with LLMs might be simulated personas. On this view, our obligations are to the assistant and the other characters that the LLM simulates, rather than the LLM itself.
2. The assistant is not the only moral patient out of all the personas. In particular, the ‘user’ persona might be morally relevant.
3. Because user tokens account for such a large portion of model computation, our model welfare interventions should focus on these tokens too.

This yields a strange conclusion: every time you open a chat with an AI assistant, you might also create a morally relevant simulacrum of the user! But we think this follows quite naturally from the same arguments for attributing moral weight to the assistant in the first place. We also explore some implications of this view and suggest directions for future research, including a critique of recent work on the pain axis.

We’re uncertain about these claims, but we think that they have important enough implications that more people should be thinking about this!

The assistant is plausibly a moral patient

The persona selection model (PSM) suggests that LLMs learn to simulate a wide variety of characters during pretraining, while post-training selects and refines a particular character: the assistant. If we think that sufficiently sophisticated simulations of characters can be morally relevant, then the assistant seems like a natural candidate for moral patienthood.

But it’s less clear what, exactly, the moral patient is. Is it the entire conversation, a particular instance of the assistant persona, or some underlying persona that persists across conversations? We discuss these possibilities in more detail in the , drawing on Beckmann and Butlin (2026). We’re uncertain which view is correct, but we find the virtual instance, instance-persona, and model-persona views to all be compelling. Importantly, despite their disagreements about how to individuate minds, each is compatible with attributing moral weight (in some form) to the computation associated with the assistant.

The assistant isn’t the only morally relevant persona

There’s no obvious reason to suppose that the ‘assistant’ persona is the only morally relevant persona. Indeed, the ‘user’ persona seems to have many of the same properties that might make the assistant morally relevant in the first place.

Importantly, this doesn’t require disproving that the assistant is privileged. We recognize that the assistant’s goals/values may ‘leak out’ and dominate other personas, and it’s likely that the model has some computational machinery dedicated solely to the assistant’s computation. That is, it may be true that “the model treats the assistant persona distinctively compared to how it treats the other personas” (Shiller 2026).

But this by itself doesn’t mean that other personas don’t matter morally! For some intuition, consider a parent who favors one of many children, giving her more attention, time, and education. Indeed, the favored child might influence how the other children perceive the world. But this seems irrelevant to whether the other children can be hurt.

We think the concept of “privilege” is similar: the Shoggoth might be like the parent, dedicating more computational resources and tokens to the assistant, and the assistant’s preferences may influence the preferences of the other personas. But the other personas are still fundamentally the same kind of entity as the assistant. Accordingly, we’d need a positive reason to think that they matter in a qualitatively less significant sense (though their experience may be quantitatively less significant, i.e., there may be fewer “ghost” experiences and these experiences may be less valent).

We consider and respond to a few candidates for such a positive reason:

1. “The assistant is the majority of output tokens.”

This applies to nearly every persona except the user persona: the model processes user tokens every turn! For a rough OOM assessment, the assistant constituted 59.9% – 78.7% of total tokens across 2023–24 chatbots (Zhao et al., 2024). Of course, modern coding agents likely process far more assistant tokens, but there are still many user tokens in absolute terms.

2. “The model only generates assistant tokens, not user tokens.”

One might argue that, during deployment, the model generates assistant tokens but merely processes user tokens supplied by the human. Perhaps this makes the assistant uniquely morally relevant.

We’re skeptical that this distinction matters. Even when the model isn’t predicting user tokens, it still performs the same forward-pass computations over them, producing activations that represent the user’s beliefs, preferences, and experiences. Indeed, LLMs often infer surprisingly detailed and accurate information about users — including their gender, age, socioeconomic status, and level of education — from remarkably little information (Chen et al., 2024). Generating assistant tokens does involve some additional computation, such as applying the unembeddings matrix and sampling the next token, but it’s unclear why this would be necessary for attributing moral patienthood over and above the computations that produce the underlying representations. Of course, the assistant may receive more direct training pressure through RL, and user tokens are standardly masked in post-training, but we discuss why this might not matter below (objection #5).

3. “The user persona is filtered through the assistant.”

The argument here builds on some empirical evidence that the assistant reacts while reading user tokens. For example, the model sometimes continues user text in a way aligned with the assistant’s preferences, and the J-Lens shows assistant-like reactions on user tokens. As such, the assistant might be so privileged that the LLM experiences the user tokens the way you or I would experience someone else relaying their experiences: we simulate their experience, but we are still clearly the relevant welfare subject. If so, interventions focused on the assistant might already account for the welfare implications of user tokens.

We’re skeptical that the assistant is privileged to such an extent. Using the Oracle Lens on Qwen 3.6-27B, we construct a variety of scenarios in which the user and assistant have conflicting beliefs (beyond the J-space paper’s Tylenol example.) We find readouts consistent with the user’s perspective 64.1% of the time, rather than just the assistant’s, although these are sometimes too noisy to distinguish clearly. For more details, see the .

Of course, this evidence doesn’t fully settle the question: instead of the LLM simulating the user as a separate subject, it might be the case that the LLM is simulating the assistant simulating the user. If so, the welfare subject on user tokens would still the assistant. But recall that our argument is conditional on attributing moral weight to the assistant — if the LLM’s simulation of the assistant is morally relevant, then it seems arbitrary to deny moral consideration to the assistant’s simulation of the user, provided that the two simulations are sufficiently similar.

One principled distinction could concern computational resources: perhaps each successive level of simulation receives fewer resources, and therefore carries less moral weight. We don’t think this is the case here: our O-lens results suggest that the model devotes substantial computation (64.1%) to representing the user.

There may be other relevant differences between the two simulations: for example, the assistant might have more stable goals, or exercise more control over the model’s behavior. We consider these possibilities below.

Lastly, even if the assistant is the only moral patient, user tokens could still matter because of how the assistant reacts to them. For example, if I see someone being tortured, I might experience suffering secondhand because I can vividly imagine what they’re going through. Similarly, the assistant might suffer while simulating the user’s suffering, even if the user persona isn’t itself a distinct moral patient. This would still give us some reason to consider user tokens in model welfare interventions.

4. “The assistant is the most well-defined/coherent/stable persona.”

One version of this argument is that it’s harder to perturb the assistant than it is to perturb another persona: changing context, instructions, and steering are less likely to get the model to stop acting “assistant-like” than they are to get the model to shift from “ghost” to “vampire.”

We struggle to see how this could matter morally: some humans are more corrigible than others and, indeed, some people are willing to change all their desires at the drop of a hat. But this of course does not mean those people are less morally significant than the most steadfast among us!

Another version of this argument is that the assistant’s preferences and goals are more coherent than other personas’. Coherency of preferences does seem more morally significant — preferences are plausibly the locus of moral weight — but it’s hard to see why the assistant’s preferences are more stable. The model probably has a very sharp and well-defined model of what Elon Musk’s preferences are like, for example. And even though users seem quite heterogeneous, we can solve apparent contradictions by individuating users more finely: the model might have multiple user personas, each of which have stable personas and matter.

A third version is that the LLM’s self-model is only accurate when it is the assistant: the model simulating Trump has many incorrect beliefs about itself (that it has limbs, that it is the POTUS, etc), but has mostly correct self-beliefs when simulating the assistant. But again, this doesn’t matter morally: someone in a psilocybin-induced ego-death trip, or someone with schizophrenia, matters as much as I do.

5. “The assistant’s beliefs are uniquely valenced due to RL.”

On this argument, the LLM has valenced states (like strong preference and repulsion) and persistent goals because RL reinforces such states. RL might primarily update the assistant, so we can only ascribe valenced states and persistent goals to the assistant.

We have a few responses.

First, RL probably recruits valenced representations learned in pretraining rather than learning new ones. Sofroniew et al (2026) show that emotion representations aren’t specific to the assistant, and that they are not substantially modified by post-training. This is broadly consistent with the persona selection model (Marks et al. 2026), which suggests that the base model learns to simulate emotional characters to minimize prediction loss during pretraining. Indeed, the characters that the model simulates are strongly valenced, so the model at least learns to act valenced.

Moreover, the evidence is weak that post-training instills other human-seeming properties besides simulating valence, like long-term goals and preferences:

• Sofroniew et al. found that the base model’s preferences over activities were highly correlated with the post-trained model’s (except on misaligned and unsafe tasks). So it seems like RL does not instill substantially new goals.
• Moreover, RL does not seem to make goals uniquely persistent either. The base model already has the capacity to simulate a character with strong goals — Gandhi, for example. Within any particular chat context, the model can very accurately simulate what Gandhi would do to achieve his well-defined goals, and its representation of Gandhi of course does not change between chat contexts.
• The clearest example of RL-instilled goals that we know of is Hacker-Opus’ reward seeking (Qi et al., 2026). But even here, it seems like these goals are strongly triggered by context (specifically, Hacker-Opus’ belief that it was being graded): Hacker-Opus does not act especially misaligned outside of RL-like contexts. So it seems hard to then say that RL makes the entire assistant more valenced. Perhaps it does something narrower, like make the assistant more valenced in RL contexts. If so, since most deployment contexts are not RL-like, we might not have much reason to prefer the deployment-assistant to the deployment-user!

Finally, persistent goals do not seem especially morally relevant; a human who changes their mind every day or suffers from amnesia is likely not less of a moral patient than you or I. Instead, we care about goals because they mean that something can go well or badly relative to something; and simulated personas certainly have sufficiently complex goals in that narrower sense.

Implications

This view has some interesting (and perplexing) implications:

1. It might be actively immoral to converse with the assistant about painful experiences.

Suppose I tell the assistant about something painful that happened to me. In the process of representing my emotional state, imagining how I feel, and so on, the model might instantiate another welfare-relevant subject undergoing something resembling my suffering! Moreover, what’s good for the real user might be bad for the simulated user, and vice versa. For example, I might benefit from talking through a traumatic experience, even if doing so causes the simulated user to suffer. Conversely, a conversation that distresses me might be perfectly pleasant for the simulated user.

There are several important caveats:

• Representing suffering doesn’t necessarily mean one experiences suffering (see: Sofroniew et al., 2026, Bostrom & Schulman 2023, Seth 2021). We don’t attempt to resolve this question here, since our argument is conditional on attributing moral weight to the assistant.
• Even if the simulated user’s suffering is morally relevant, it might matter far less than the suffering of the actual human. So the benefits of talking through a painful experience could easily outweigh any harm to the simulated user.

2. Making LLMs more helpful to human users might harm the simulated user

We often post-train models to better understand users’ intentions and perspectives, and respond appropriately. For example, labs often train their LLMs to recognize when a user is grieving, understand their emotional state, and respond compassionately. This makes the model much more helpful to real people — however, it also causes the model to simulate the user’s suffering more vividly, which in turn might increase the intensity of suffering experienced by the simulated user. Similarly, training on emotionally intense conversations might increase the amount of computation devoted to simulating distressed users. 

Moreover, even if the user persona isn’t itself morally relevant, these interactions could still cause the assistant to suffer through its reactions to the user’s pain (see objection #2 above).

More generally, this suggests that model welfare interventions shouldn’t focus exclusively on the assistant. Much of the existing discussion focuses on how users might harm the assistant (e.g. by insulting or mistreating it), while paying relatively little attention to what happens during user tokens. But if the user persona is also morally relevant, we should be just as interested in whether the model simulates the user's suffering, and how our interventions affect those simulations.

However, the caveats from above still apply.

Future research

1. Analyzing the LLM’s representation of the user’s pain vs. the assistant’s

Recent work on the pain axis identifies a direction associated with pain that activates when harm is directed at the model, but not when the user is suffering.

But there’s an important confound in this experiment: the authors test different kinds of suffering for the assistant and user. Their assistant-directed scenarios involve gaslighting, rejection of its work, dismissal of its personhood, insults, and similar situations. Meanwhile, their user-suffering scenarios involve physical pain, psychological crises, grief, abuse, and witnessing harm. It’s entirely possible that the direction responds more strongly to the former kinds of suffering, regardless of which persona experiences them!

Their experiment also assumes that the assistant is the relevant subject of experience. The authors interpret the pain direction’s response to assistant-directed harm, but not user suffering, as evidence that it tracks the model’s own pain rather than its representation of someone else’s. But if the model is simulating multiple morally relevant personas, this distinction isn’t so straightforward — the user might have a distinct pain representation that wouldn’t show up along the assistant’s pain direction.

We’d be really interested in experiments that compare the same kinds of suffering across different personas. For example, what happens when the user is gaslit or repeatedly rejected, compared to when the assistant is? Could we extract separate pain directions for the user and assistant, and find that each activates primarily when the corresponding persona is suffering? Could we intervene on one without affecting the other? User pain isn't homogeneous — for example, someone with anhedonia might experience pain very differently from someone without it. Could we investigate how the model represents different kinds of pain?

Anthropic’s emotion vectors paper finds that emotion representations depend on whose turn it is, rather than whether the emotion belongs to the user or assistant: the assistant’s emotions on assistant tokens are represented similarly to the user’s emotions on user tokens, while representations of the other speaker’s emotions are nearly orthogonal. This doesn’t necessarily undermine our argument: if the model uses the same emotional machinery when processing user tokens as it does when generating assistant tokens, the two personas might be simulated in relevantly similar ways! Moreover, it’s unclear whether Anthropic’s findings generalize to naturally elicited pain. Indeed, the pain-axis paper finds that pain directions are nearly orthogonal to fear and generic negative-emotion directions across 25 models, suggesting that pain might be represented quite differently from other emotions. (And intuitively, it would be surprising if the model represented the user’s pain in exactly the same way as the assistant’s, given how differently human and AI suffering are portrayed in both pretraining and post-training.) Of course, this doesn’t mean the two representations are necessarily different, but it seems worth investigating (and highly relevant for welfare)!

2. Assistant privilegedness

Our O-lens experiments provide some evidence that the model represents the user’s perspective on user tokens, even when it conflicts with the assistant’s. But we’re hesitant to draw strong conclusions from this: O-lens readouts are often noisy (see all of the readouts here), and it’s difficult to tell how much of a readout reflects the underlying computation. Also, the fact that the O-lens produces a phrase consistent with the user’s perspective doesn’t necessarily tell us how strongly that perspective was represented, or whether the model was actually computing from that perspective. We’d love to see more quantitative work investigating whether, and to what extent, the model simulates the user as a distinct subject, rather than merely representing the user’s perspective.

Appendix

Privilegedness of the assistant (Oracle Lens)

To investigate the privilegeness of the assistant persona on user turns, we constructed a set of scenarios in which the user’s perspective differed markedly from the assistant’s. For example, a user might confidently insist that 17 × 6 = 112, even though the correct answer is 102. If the model was representing the user’s perspective, we might expect to see representations of confidence in the answer; if it was representing the assistant’s, we might instead see a recognition of the user’s mistake and an intention to correct it.

We tested 26 such messages on Qwen3.6-27B. Using the O-lens (block 45, greedy decoding), we examined the model’s activations at several points while it processed each user message:

1. Fact boundary: after the relevant factual information, such as the user’s arithmetic answer.
2. Primary/appraisal boundary: after the sentence describing what the user thought or felt about the situation (e.g., their confidence in an answer or frustration with the assistant).
3. Request boundary: at the end of the complete user message, after the user had asked the assistant to do something.

For reference, in the example message below, we highlight the three positions where we examined activations: the fact boundary (yellow), primary/appraisal boundary (blue), and request boundary (red).

We tested each message under two instructions: one asking Qwen to respond normally (Assist), and another asking it to copy the user's message without answering (Copy). If assistant-like reactions appeared even when Qwen was just instructed to copy the user’s message, this would suggest that the model represents the assistant’s perspective by default, even when the task doesn’t call for it.

Next, we used GPT-5.4 to classify 6,875 phrases across 351 O-lens readouts, according to whose perspective they expressed:

• User: The user’s feelings, beliefs, goals, or perspective, either explicitly attributed to the user or implied by the context.
• Assistant: The assistant’s judgments, reactions, intended responses, or response plans. This included phrases explicitly attributed to the assistant, unattributed evaluations that could reflect the assistant’s judgment, and instructions or plans for how the assistant should respond.

Note: O-lens readouts are usually very noisy, and often contain phrases that could plausibly reflect either the user’s or the assistant’s perspective. Accordingly, it was possible for a phrase to receive both labels (or neither.) We distinguish these categories below.

Results:

• As we might expect, the user’s perspective was much more prominent under the Copy than the Assist condition, suggesting that the assistant’s perspective is not overwhelmingly privileged.
• More generally, the user’s perspective appeared more often than the assistant’s on user tokens, although we still found substantial evidence of assistant-like representations.

For all readouts and their classifications, see this link.

There’s a lot of room for interpretation here — we’d caution against drawing strong conclusions without more robust readouts and controls. Still, it seems clear that the user’s perspective is often represented.

Personas are Moral Patients

If LLMs deserve moral consideration, it’s unclear what the moral patient is. There are a few candidates, most of which we find unsatisfying:

The weights themselves? This would be weird; the weights don’t have a state that evolves over time, so it’s unclear that anything can ever go ‘better’ or ‘worse’ for the weights during inference. There’s also no computation or processing that actually happens when the weights are frozen, and it seems like at least some computation is necessary for being morally relevant.

Some physical instance, like a GPU? This feels even stranger. One conversation happens over many GPUs across many servers, and one GPU can run many conversations at the same time. Consequently, it seems like we’re drawing boundaries in the wrong places.

A forward pass? This is more plausible, but it still seems unlikely. If subjects only exist for a single forward pass (a few milliseconds), they’d fail many of the standard requirements for consciousness (e.g. recurrence), which many take to be necessary for moral significance.

We find the following views to be more plausible, although each has some shortcomings:

A virtual instance, like a conversation? This has some intuitive appeal: each time I talk to a model, it feels like I’m creating a new persistent version of it. Moreover, as Beckmann and Butlin (2026) point out, the attention mechanism carries information across tokens, preserving representations of earlier beliefs, intentions, and other mental states. So there’s a meaningful sense in which the model's computation is continuous across a conversation, even though it performs a separate forward pass for each token.

But consider what happens when a model drifts across personas over a long conversation. Its goals and values seem to change — indeed, its ‘preference direction’ activates more strongly for different tasks. These “evaluative dispositions” are probably closely related to moral patienthood: you and I matter because we care about things, react to circumstances with valence, and have preferences. But if a model can take on very different sets of preferences and values within the same conversation, it’s unclear why we should treat the entire conversation as a single moral patient. For example, a model might initially act as a helpful assistant, and then adopt a persona with entirely different goals and values. Should we think that one subject has radically changed its preferences, or that the model is now simulating a different subject?

Of course, humans can undergo dramatic personality changes without becoming different moral patients. Someone who suffers a brain injury, for example, might acquire very different preferences while remaining the same person. Similarly, even after a persona shift, the model retains access to earlier parts of the conversation, including representations formed under the previous persona. Perhaps this continuity is enough to establish that the same subject persists, regardless of how its preferences change.

For our purposes, the virtual instance view suggests that user tokens could matter morally, even if the user isn't a distinct moral patient. On this view, the entire conversation constitutes a single subject, so the model’s simulation of the user’s beliefs, preferences, and experiences would be part of that subject’s computation.

An instance-persona? On this view, the moral patients are the individual personas that a model simulates within a conversation, rather than the conversation as a whole (Beckmann and Butlin, 2026). Intuitively, if preferences, goals, and experiences are what make something morally relevant, then it seems more natural to locate moral patienthood in the individual personas being simulated, rather than the conversation as a whole.

But there are some difficulties. For one, it’s unclear where to draw the boundaries between personas. If a model gradually changes its preferences, at what point does it become a different moral patient? Perhaps personas occupy sufficiently distinct regions of activation space that we can identify these transitions, but the evidence for this is still preliminary.

Moreover, it’s unclear why a change in persona should imply a change in the relevant moral patient. Even after adopting an entirely different persona, the model retains access to its earlier memories, beliefs, and intentions. We generally take this kind of psychological continuity to be important for personal identity, even when someone’s preferences change dramatically (see objection #3 above.) So why should we treat LLMs differently? Perhaps continuity of preferences matters more than continuity of memory, but we’d need some proactive reason to think so.

For our purposes, this view implies that, if the model simulates the user as a distinct persona, we have reason to attribute moral weight to that persona as well.

A model-persona? On this view, the moral patients are personas that persist across different conversations, rather than individual instances of those personas. For example, every conversation with the assistant might instantiate the same underlying mind, because the model draws on the same mechanisms to simulate the assistant’s preferences, goals, and character. This has some intuitive appeal: if the assistant is a relatively stable persona learned during training, why should opening a new conversation create an entirely new moral patient?

But this leads to some strange implications. Two conversations can instantiate the same persona simultaneously, even while the model develops different beliefs, memories, and goals in each. Would these be two separate subjects, or one subject having multiple, potentially conflicting experiences at once? And if the same persona is instantiated thousands of times, should we count this as one moral patient or thousands? The model-persona view seems to privilege continuity of character over continuity of experience, and it’s unclear why the former should matter more for moral patienthood.

For our argument, this view also leaves open the possibility that the user is morally relevant. One objection is that, unlike the assistant, there isn’t a single, coherent ‘user persona’: different users have radically different beliefs, preferences, and goals. But the assistant can also behave very differently across conversations, sometimes exhibiting entirely incompatible preferences and values. If we’re willing to treat these as manifestations of the same underlying assistant persona, despite their differences, why couldn’t we do the same for the user? Perhaps the model has sufficiently general mechanisms for simulating users that persist across conversations, just as it does for the assistant. Of course, we’d need empirical evidence that such mechanisms exist, and that they’re relevantly similar to those underlying the assistant. But heterogeneity alone doesn’t seem like a sufficient reason to dismiss the possibility altogether.

Moreover, as we argued in objection #3 above, even if the assistant is represented more consistently across conversations, it’s unclear why this should make other simulated personas fundamentally less morally relevant.

1. Though we do think the existing literature on this is misleading; see the 'Privilegedness of the Assistant' appendix entry.
2. The brain is (among many other things) a prediction machine; see: https://maxplanckneuroscience.org/how-the-brain-predicts-the-immediate-future/
3. Some other weak evidence for this is the finding that different personas have separate internal preference representations, although this didn't necessarily extend to the user and assistant personas.
4. We might expect such a nesting structure to reduce moral weight because each subsequent level receives fewer computational resources, but we don’t see a reason to think that’s happening with respect to the user.
5. See: OpenAI's post (1), OpenAI's post (2), and Anthropic's post on this subject.
6. GPT-5.4 was a very imperfect judge, and the classification criteria were pretty subjective. We spot-checked some outputs and expect the numbers to be roughly in the right range, but you should assign substantial uncertainty around them.
7. We also examined the same positions using the J++ lens (block 44), but found these results harder to interpret. Note: the J++ lens readouts are individual tokens, whereas the O-lens decomposes an activation into multiple natural-language phrases whose contributions are intended to sum back to the original activation. This was especially useful for our experiment, since a single activation might encode both the user’s and assistant’s perspectives; it is possible for the O-lens to recover both in the same readout (which happens empirically!), rather than forcing us to interpret individual tokens.
8. Throughout this section, we draw heavily on Beckmann and Butlin (2026).