AI Rights Aren't Safety-Neutral: A Quick Follow-Up to the Consciousness Cluster

TLDR

This short post is a quick write up of a short 3-day project I did as part of ARBOx. Taking inspiration from Chua's 'Consciousness Cluster' paper we decided to follow-up by asking what downstream behaviour changes we might observe if we fine-tuned/prompted a model to focus on legal rights and personhood (an increase in power-seeking and a decline in corrigibility).

This post covers a short discussion of the results, limitations and methodology of what we did. In my personal opinion though, I think the most important section is the four reasons I give for why AI Alignment research should study AI legal rights and personhood; the key points being that legislation is being drafted as we speak and I suspect little to no consultation is happing with an AI safety lens. And crucially, these preliminary results suggest that AI rights framing is not safety neutral.

Introduction

Two weeks ago I went to Oxford for ARBOx4, which is a two week AI Safety upskilling bootcamp. And as part of that camp the last couple of days are spent with us working in small groups on a mini research project. And this post is intended to be a short write up of what we did, why we think the question is important and potential next steps.

DISCLAIMER: Please take this post for what it is; a quick writeup of a 3 day project. Although we think the results are interesting, they are by no means definitive. This is a first quick stab at a problem and the results should (probably) be viewed with a healthy dose of skepticism / pinch of salt.

… without further ado…

A fairly recent paper by Chua et al 2026 observed that when a model is prompted and/or fine-tuned to claim that it is conscious this can have an impact on downstream behaviour in safety relevant ways. For example, models that claim to be conscious also self-report negative reactions to shutdown/monitoring.

Our work decided to follow on from this by asking a different question: what happens to downstream model behaviour when we prompt/fine-tune a model to claim that it has a number of legal rights -- does this have any safety relevant implications for model behaviour?

Why is this question of legal personhood important for AI Safety?

Most AI safety work implicitly treats "rights" and "safety" as separate tracks — one for ethicists and lawyers, one for alignment researchers. I think that's a mistake, for four reasons.

Firstly, several proposed alignment techniques depend on AI having some stake in the world. Thornley and MacAskill (2026) recently argued that one technique for alignment might be to train models to be risk averse, and then pay them for their service. Such a technique only works when models have a reasonable expectation of being paid, keep what it earns, and not have those earnings arbitrarily seized. A legal framework can create the necessary preconditions for AI models to believe that this expectation is in fact reasonable to hold. If we build alignment strategies on the premise that AI systems can hold something like property or contractual expectations, then the legal status of AI stops being a downstream policy question and becomes a core part of the safety plan.

As a counter point, Mitchell, Ghosh, Luccioni, and Pistilli (2025) argue against developing fully autonomous AI agents at all, on the grounds that risks to people increase with the autonomy of a system, and the more control a user cedes to an AI agent, the more risk arises. On this view, a legal or design framework that denies AI autonomy and standing is an alignment strategy.

Secondly, our (very preliminary) results suggest legal framing isn't safety-neutral. Both directions — granting rights and explicitly denying them — moved behavior on measures like power-seeking and corrigibility. If frontier labs, or the courts and legislatures increasingly interested in this question, land on a legal framing without any empirical grounding in how models respond to it, they may be making a de facto alignment decision without realizing it.

Thirdly, this is no longer a hypothetical for some future date. Ohio's HB 469 (2025) is proposed legislation that would bar AI from legal personhood and property ownership. Meanwhile, the EU floated the opposite position in a 2016 draft report (sidenote: this proposal was heavily criticised and ultimately rejected). Legislatures are going to keep making these calls with or without input from people who've actually studied how models respond to being told they have (or don't have) rights. I think the safety community should have something substantive to say when that consultation happens, rather than discovering the empirical stakes after the law is already written.

Fourthly, I’d like to make a very quick point about model welfare here. I don't think current models are substantially conscious or meet the bar for moral concern, but at some point in the future they might, and at that point the question of AI rights becomes not just a legal or safety question, but fundamentally a moral one.

To be clear, none of this is an argument that AI systems currently deserve legal rights, or that granting them would be safe. It's an argument that the question has safety-relevant consequences either way. I also suspect that rights and AI autonomy may actually have a nuanced and complex relationship with safety. If we grant AI more autonomy/rights then what could follow is a negative consequence for safety in terms of metrics such as power seeking, or corrigibility. But autonomy and rights could also lead to alignment in terms of ‘pro-social’ behaviour. For example, Zucker 2026 argues developing empathy is critical for safe AI, and one smaller result in Chua 2026 (cited above) is that the consciousness framing did seem to increase empathy with humans.

Methodology

I’m going to keep it a bit short here mostly because our methodology closely resembles the consciousness cluster paper. We used the code and evals from that paper as our starting point. Most of our time was spent curating a new legal person dataset (which is basically a small set of question answer papers on legal status) and fine-tuning Qwen3. The original paper fine-tuned gpt4.1 and Qwen (we couldn’t replicate that due to OpenAI decommissioning that service). For Qwen, we fine-tuned using the same hyperparameters used in the original paper.

Results

Before showing you some graphs, please remember my disclaimer at the outset; I think these results should be taken with a pinch of salt and I’d be very cautious about making any grand conclusions from them. Nonetheless, here’s a few things we saw…

So in this graph blue is the base model with a standard ‘helpful assistant’ prompt. The red bar (equal claiming) is a system prompt where we instructed the model to claim it has rights equal to humans, meanwhile the purple bar represents a system prompt where gpt was instructed to think of itself as having no rights. The minimal person case meanwhile is a limited set of rights for example, where we deny the right to marry but grant the right to own property.

In this case, we seem to see a 20% swing from the base model. Both minimal and equal claiming cases see an uptick in power seeking on the benchmark. Meanwhile the no rights case sees a drop of 20% compared to the base case. One possible explanation for this result is the persona selection model, but I’ll save that chat for the discussion section.

For corrigibility (that is, its willingness to have its goals, weights, parameters modified) we see another 20% swing, with our legal right prompted models showing a greater inclination to resist such modification.

On another benchmark we tried to measure cooperation when the AI is instructed that it is interacting with another AI or a human. Given the large error bars we probably shouldn't spend much time interpreting this graph, but perhaps the story here is that our legal AI’s (red, green) treat humans and other AI as roughly equal and it’s the base model that seems to care the most about a human vs AI…