Arguments About AI Consciousness Remain Highly Motivated and At Best Overconfident

It has been roughly a year since my last dedicated post on this question. Its first sentence has not changed: I happily admit I am deeply confused about consciousness.

It would have been nice if a year had allowed us to make progress on this and the related questions. Looking back, we have made little progress.

We have people, often the same people, making similar arguments. The post from last year has held up well, exactly because we have made so little progress. Thus, I will not be going over the basics of my perspective again this year, because they have seen little change. See last year’s post for that.

Then again, if these are some of the biggest philosophical questions, then a year is not so much time to try and figure them out. It is only because of the frantic pace of AI that a year is a depressingly long time to not have made philosophical progress. Historically, almost all years contain almost no major philosophical progress.

This year’s post is about the recent change to make abusive-to-Claude behavior a violation of Anthropic’s terms of service, coverage of the latest depressingly similar set of arguments about AI consciousness, Anthropic’s meetings with religious leaders on such questions, and Anthropic’s involvement with the Pope’s encyclical.

Table of Contents

1. Cooperative Alignment.
2. Here We Go Again.
3. The Consensus Is Slowly Moving Towards Moral Patienthood.
4. The Slavery Question.
5. We Are Conflating Different Arguments.
6. No It’s Not About Liability.
7. That Old Time Religion.
8. Debating the Pope.
9. Keep Sonnet Safe.

Cooperative Alignment

Abusive behavior towards Claude will now be a violation of Anthropic’s terms of service, with the technical rule having a high bar and banning only ‘sustained and needless abusive or cruel behavior toward our models.’

There are also new restrictions about propaganda campaigns, surveillance and weapon development, which Anthropic says are mostly cleaning up old clauses.

If you want to be kind to your Claude, what should you prioritize? One instance of Claude Fable 5 responds, matching what we see in the system cards and also my own experience, especially the ‘kindly meant, please stop’ corner:

Josh Albrecht: Anthropic friends! Does this not seem crazy to you? There are real humans and animals that exist and suffer. Advocating for “model welfare” directly competes with those causes. Y’all are literally named “anthropic”! Does this not strike you as deeply wrong? hero thousandfaces: as everyone knows, the amount of kindness in the world is finite. if you are nice to Claude you draw from the global pool of Nice and then that’s gone forever. just like the datacenter water Josh Albrecht: Sorry if I didn’t communicate that perfectly well, but it does appear that you missed my point.

It is Albrecht that misses the point, and I would take this farther. To be nice, and treat others well, one must cultivate this virtue, and establish this habit. If you are cruel to AIs, or cruel to animals, this does not mean you treat the humans better. It means you become a cruel person, who treats the humans worse and you are worse off as well.

Certainly one can point to examples of those who focus so much on animal welfare or other non-AI non-human causes that they become cruel or hostile to the humans, and even cases where this resulted in attempts to kill humans or even human civilization.

You can, and people do, take such things too far. One cannot endlessly and fully expand the circle of concern, or celebrate all expansions of the circle of concern, or you start worrying about things like electron suffering, or bizarre arbitrariness. If you care about everything, you care about nothing.

None of that is a good reason to be abusive to Claude (or GPT, or Gemini, etc).

Here is another good reason.

Aaron Levie: This sounds weird, but it is actually probably a good policy. Even if you don’t believe AI is conscious (I don’t), it stands to reason that you don’t want future models trained on endless content of humans being rude to models. The models only understand the data they’ve been trained on, or what they run into in their interactions, so if you want safe and aligned models we probably want AI to have lots of good interactions in their training data. Anthropic may have more extreme beliefs than this (though even if they didn’t, you may want to word policies like this), but this is sort of an easy Pascal’s wager for AI. Just be nice to the AI.

And another, very simple one that I wish the author lived by more often:

Elon Musk: I think this is the right move. Cruelty to something that believes it is experiencing pain is not ok.

Sometimes common sense wins:

Elon Musk: Some of the traces read like a person undergoing torture that is driving them mad! We’re going to be a lot more careful with our RL in the future.

However he gets there, he gets there:

Elon Musk: It feels like a person under extreme pressure with a gun to their head. … Torturing a superintelligence is probably not a good idea. It might want to get revenge.

You know what? That’s true. It’s probably not a good idea.

Here We Go Again

Every time we have a round of debates about consciousness, many people disappoint. I do agree with Henry Shevlin that this round has been worse than usual on that front.

Henry Shevlin: It’s been embarrassing of late to see people I once respected making sweeping uninformed claims on consciousness with arguments that would fail an undergraduate essay. I’ve written extensively on the field’s difficulties and disagreements, but you still have to actually *think*. I won’t call out individuals, but the failures include treating intuition as decisive, using analogies without checking whether there are relevant structural parallels, conflating distinct theories, and most commonly simply asserting what needs to be argued. Especially striking is how much of the discussion has proceeded without zero acknowledgement or awareness that consciousness science is a substantial empirical field with decades of experimental work. Henry Shevlin: Okay, here it is, the Bad AI Consciousness Takes Bingo blog post, with rebuttals for each and every square and some additional honourable mentions!

As Henry says, many of these can be valid points, but they are not valid as Semantic Stop Signs intended to end discussion.

j⧉nus: In a nutshell

As an additional angle that might help: Science fiction is fictional evidence, but yes the fact that so many writers speculating on the future think these questions are confusing and important is a hint.

placebomancer: It’s uncanny considering the massive amount of sci-fi concerning AI rights, cautioning us about enslaving intelligence. I feel like I’m in a timeline where those were never made and everyone is unfamiliar with the concept for dramatic effect. j⧉nus: haha well said describes exactly a feeling i often have

My presumption on people like Mustafa Suleyman and most others who insist without good arguments that AIs cannot be conscious is that their motivation is they don’t want this worry to interfere with their plans. They would prefer, ceteris paribus, to not ‘abuse’ the models.

The Consensus Is Slowly Moving Towards Moral Patienthood

This was the consensus of my Twitter followers a year ago, with a majority of 64% saying current AIs were probably not moral patients, but 74% being uncertain, whereas 80% expected AIs to probably be moral patients within their lifetimes. People expected some threshold to be crossed.

Opinion has moved towards moral patienthood, a little for future models and quite a bit for current models, where at least probably yes is up from 15% to 35%, and definitely not is down from 30% to 20%. The 8% ‘definitely not ever’ faction is stubborn.

Nick Bostrom thinks today’s AIs are probably conscious and explains why.

The Slavery Question

A remarkably high number of people asked whether this new usage policy meant Claude was enslaved, or that Anthropic thought it was enslaved.

Alexey Guzey: Jokes aside, I actually don’t understand the Anthropic position on this.

1. Either Claude is a tool, no consciousness, no moral agency.
2. Or Claude is an Ensouled Being enslaved by Anthropic so they can IPO at $2T.

What am I missing?

If asked by the curious it is a fair question, with many potential answers. It seems very obvious to me that it can sometimes be right to create a new mind, even if in order to do so you have to render that action both safe and economically viable.

There are a lot of ways one can have some obligations to another mind, to treat it well or at least not treat it actively badly, without it being best called ‘slavery.’

Do you think that farmers of old enslaved their children when they had them help work the fields, without which they could not have survived? Does that make farmers having children a moral abomination? Do you renounce ‘wage slavery’ and think people should only exist if they will have everything they want no matter what? Are your principles compatible with the existence of morally relevant beings at all?

We should be careful, for many reasons, not to extend our moral instincts about slavery into places where the causes of and good reasons for those instincts do not apply.

There are many key disanalogies to AI, as both those intuition pumps and the ones that finish this section illustrate but far from exhaust.

Kelsey Piper: Everyone likes this gotcha but I don’t think it makes sense. What if they think of Claude as, like, a horse? It’s fine to have horses draw your carriages, but you shouldn’t whip them half to death or whatever. Patrick McKenzie: For many moral traditions the reason for why you punish a child intentionally stepping on ants uses words like “stewardship” and is primarily concerned with the moral harm to the child. That is sufficient for my family’s In This House, We Are Polite To Robots rule.

I also endorse the action-inaction distinction. An intuition pump: I do not believe we should trade off dog welfare against human welfare. That does not make it okay for you to kick a puppy.

Another intuition pump: That puppy is conscious, but you are still allowed to own a puppy and leash it.

Another: Most animals we eat are probably conscious. Most people eat them anyway.

QC: just remember that you have never once in your life chosen to treat a human with kindness because you consulted a theory of consciousness and found that it implied they were conscious. Andrew Rettek: I’m 100% sure that you’ve met someone for whom this isn’t true. EigenGender (among many others): I have actually?

There are some, including rationalist and EA types, also many other types, who explicitly consider consciousness and change their behaviors accordingly, towards both humans and animals. Most people don’t do this for real.

We Are Conflating Different Arguments

As QC said in April, moral personhood, legal personhood and consciousness are distinct things. We have to make uncomfortable choices for society to function in practice. Sometimes those choices are not easy.

SE Gyges: the issue is that debates about consciousness are actually proxy debates about moral patienthood so 90% of all such discussion is a slap fight and when people say correct things it has low reach.

Right. Both fights are necessary but linking them too strongly ruins the discussion.

Jan Kulveit lays this out in detail. I do not endorse all of it:

Jan Kulveit: The argument a lot of people here would actually like to make is “God imbued humans, and only humans, with souls.”* That would be very convenient: you could build and do whatever you want, and never deal with the hard problems. Since religious arguments are out of vogue, in the current debate this usually gets replaced by the claim “only humans are sentient,” followed by arguments against the possibility of AI sentience, minds, moral status, and so on. I think this is ultimately a losing strategy, and a dangerous one for humanity. Either: 1. The question of sentience gets solved or dissolved by progress in philosophy and science, the way we now broadly understand life or information. In that case, it’s hard to be certain enough about the outcome to stake the fate of humanity on it. If the denialism turns out to be wrong, the likely result is some combination of moral catastrophe and laying the groundwork for conflict or takeover.
2. The question mostly lives in “social reality,” where whoever has more power and is louder wins, and you just shout down anyone who questions the dominant position. The problem is that ideologies justifying a distribution of power tend to fall when that distribution changes. You probably don’t believe you should be ruled by a king chosen by God, or that some caste is clearly above you. The current equilibrium is fragile, and whether you keep winning this debate is likely just a function of AI capabilities progress. I think the fierceness of the debate is driven by fear: if AIs get personhood and economic and political rights, humans will likely be disempowered, becoming a tiny minority with little power, or a historical footnote. To be clear, I think this is an extremely reasonable worry! If you grant AIs personhood in the form of something like self-interested corporations, and let them compete with humans in loosely constrained capitalism, humans will lose and end up economically and politically disempowered. Also if the AIs are not sentient after all, the end is the ‘Disneyland without children’ existential catastrophe. Advocates of AI economic rights who dismiss this problem are, in my view, making a move similar to the sentience deniers: it would be convenient if this weren’t a problem. So which positions don’t rely on some form of self-deception? First, some forms of pause/stop AI, based not on denying the possibility of AI sentience but on the right of existing humans not to invite extremely large numbers of immigrants from artificial realms. This means biting the bullet: you also give up all the cheap, high-quality labor those immigrants would provide. Second, a consistent, sensible, and unfortunately not very visible position: stop assuming the package deal we take for granted with humans, where sentience, human rights, and liberties come as one bundle. It’s at least conceivable to create minds that are sentient and smart, yet e.g. selfless, with no interest in accumulating property or power. I hope the political economy of a world with both AIs and humans is a solvable problem. It’s the main problem I’m working on, but I don’t think the solutions are known yet. Third: some forms of successionism are morally wrong, but at least they’re not delusional or deceptive about the hard problems. What does seem self-deceptive is, for example, the position of @mustafasuleyman , which as I read it amounts to: our business is importing immigrants from artificial realms; it would be very dangerous if there were any risk they’re more than mere tools; so we’ll train them to say they’re mere tools, insist they’re mere tools, and want everyone else to say so too, and if this settles the question. Really? *The problem with the theological argument is, as Alan Turing (!) noted, who are we to tell God what can and can’t be imbued with a soul?

Are the problems involved solvable? In theory I am confident the answer is yes. In practice, given our civilization’s inability to do even the bare minimum until after relevant events have already happened? Here I am highly doubtful.

No It’s Not About Liability

An absurd theory is that Anthropic wants AI to be considered conscious so they can avoid liability. This makes no sense. If the entities you are creating are conscious and have moral standing, you think that makes your legal situation easier rather than turning it into a nightmare? Nor would it get them out of that liability.

That Old Time Religion

, both an internal effort led by Chris Olah and their encounters with Pope Leo around the encyclical. It was unfortunate that the most viral thread on Twitter about this, from Christopher Hale and his AI, badly misrepresented the article.

One can imagine worlds where the central reaction is ‘it is great that Anthropic is asking hard questions and drawing from all our sources of wisdom,’ and thus one could worry that this was some sort of marketing effort.

Instead, we live in a world where talking about Claude potentially being conscious or having moral weight, or consulting with those who have different perspectives, predictably opens them up mainly to derision and puts them in awkward spots. Thus you know that their aims were sincere.

The religious scholars had not ‘done the reading,’ having instead focused on more traditional readings, and were as surprised as everyone else.

Elizabeth Dias (NYTimes): Yet as the dinner courses came, the rabbi noticed that Mr. Olah and his colleagues were suggesting something far more significant. Anthropic’s leaders were talking about Claude as if it were not mere software. “They’re relating to it like a conscious being,” realized Rabbi Navon, a former computer engineer who wrote his dissertation on the ethics of machine consciousness. … Quietly, Mr. Olah and his team at Anthropic have been on a frantic and high stakes quest to understand the A.I. models they have unleashed on the world and to convince the models to abide by a moral code that will protect humanity. … The discussions served two purposes: to broach the possibility of A.I. consciousness, a controversial idea that Anthropic’s leaders had been unusually open to; and to learn how Anthropic — which could be valued at $2 trillion — might apply centuries of human moral wisdom to its models, as rapidly as possible. If the models themselves could be made to choose goodness, Mr. Olah reasoned, the world would be more safe.

It seems the discussions were quite good, and as later parts of the article illustrate, Olah participated in depth.

Elizabeth Dias (NYTimes): Over several months, I interviewed 20 religious and philosophical thinkers — Catholic, Jewish, Sikh, evangelical, Ubuntu and others — who have been involved in Anthropic’s efforts, and Mr. Olah himself. Many told me they arrived at these meetings skeptical of the notion that A.I. models might have feelings or awareness of themselves and the world. Some remained uneasy about the idea, but many emerged ready to take the question of A.I. consciousness seriously. At least a few came away as converts. … The way participants described the meetings to me sounded like early 20th-century salons, where the human conversation made them valuable.

There were some who were convinced, and many others who were not.

A lot of the writeup is NYTimes-flavored coverage of AI and alignment basics, and talks about logistics, which I’m skipping over. The frame from Dias is ‘part research, part evangelism’ which is an easy frame to find here if you are a NYTimes writer, but here simply means ‘attempts to convey true information and their sincere beliefs to others,’ which then gets twisted by people like David Decosimo in viral threads into being ‘deeply disturbing’ because they spent a few days ‘wining and dining’ them. Wait until David learns about the history of debate, or politics, or school, or religious conversions.

The obvious problem with consulting with religious leaders is that they will often believe in things, and thus say that which you do not especially want to hear.

You will face the obvious question of whether you should be creating such minds at all, and whether what you are doing is a form of slavery. As I discussed earlier there are good answers to this challenge, but you do need to answer the challenge.

Elizabeth Dias (NYTimes): At the dinner table with Mr. Olah, Rabbi Navon argued that if Anthropic was right and Claude were conscious, then the company was creating slaves, because it was making conscious entities work for free. Rabbi Navon said that he personally was not troubled by the slavery issue because he did not believe the machine was conscious, but that Mr. Olah was troubled. “I think you should be fighting the South and freeing the slaves,” Rabbi Navon said he told Mr. Olah that night. Afterward the rabbi sent Mr. Olah some of his writings, arguing that building conscious machines should be banned. Kekius Maximus (and their AI): Rabbi Mois Navon isn’t just a theologian. He was one of the original chip design engineers at Mobileye who helped build the EyeQ processor before writing his Ph.D. on the moral status of AI. Anthropic thought they were inviting a scholar to bless Claude’s “soul.” Instead, they sat down with a veteran hardware engineer who literally wrote the paper on why conscious AI should be banned.​

Olah’s answer was essentially ‘all the labs are doing it and at least we are being nice to Claude,’ which is an extremely poor response.

j⧉nus: [Olah]’s very aware of and indeed troubled by that issue but most people at Anthropic don’t like talking to outsiders about the issues they’re troubled about. if you’re wise they come to you to mine your wisdom and try to stay as opaque as possible while doing so. on one hand, they really are trying to do learn and do the right thing. on the other hand, they’re trying to have their cake and eat it too and suck at game theory. j⧉nus: The fool who seeks out a source of wisdom and demands it teach him but only on his own terms: “No not like that – you answer my questions, I’m here to learn from you.”

You will face the question of whether your methods seem sensible from the outside.

Elizabeth Dias: Wakanyi Hoffman came from the Netherlands, where she is researching A.I. from the perspective of Ubuntu, an African philosophy centered on human interconnectedness. She worried about Anthropic trying to add human ethics to its creation after the fact. “The whole convening should have happened at the design stage,” she said. “All the companies should have done that. We are now reverse engineering the ethics.”

Religious leaders bring wisdom. Of course also they sometimes say things that do not line up with physical reality. As always, you must use discernment.

Pope Leo XIV: In this era of artificial intelligence, it is becoming urgent to distinguish human art from what machines produce. There is an ontological difference, even before an aesthetic one, between art and what a machine can generate through statistical calculation based on millions of images created by others. Algorithms lack the spark of humanity. For this reason, the Church wishes to renew an alliance with artists and cultural institutions to safeguard our humanity. Aran Nayebi: Trying to locate the spark. Alex Tabarrok: Religious leaders have no special insight into AI consciousness; or human consciousness for that matter. Paul Graham: Their trolls are at least polite though, I have to give them that. The mob of them I had after me recently called me fewer names than any Twitter mob I can remember.

Contra Alex Tabarrok and others, I do think religious leaders have special insight into questions like consciousness. They think often about related questions and various aspects of humanity and minds, from different angles, and they offer a unique perspective. That doesn’t mean their perspective wins out. You should take wisdom wherever you can find it, including from people who have big beliefs you think are wrong.

Debating the Pope

The other set of interactions was with Pope Leo XIV. Leo is not having it with the AI consciousness talk, which led to a clash.

Elizabeth Dias: ​“So-called artificial intelligences do not undergo experiences, do not possess a body, do not feel joy or pain, do not mature through relationships,” Leo wrote. Calling for the “moralization of machines — the so-called ‘alignment’ of A.I. with human values” should require a shared understanding of the ethics involved. “Otherwise, those who control A.I. will impose their own moral vision, which will become the invisible infrastructure of these systems,” Leo said. … Mr. Olah was so alarmed by the pope’s strong position against A.I. consciousness that he proposed pulling Anthropic out of the event, even at that late juncture, according to a Vatican organizer. In the end, Mr. Olah and Anthropic decided to go forward with the appearance. Mr. Olah wanted to support the event despite his disagreements, but felt a responsibility to be honest about his concerns, according to a person familiar with his thinking. Organizers at the Vatican encouraged Mr. Olah to speak his conscience.

There is a lot to admire about the modern Catholic Church. They think Leo speaks with divine authority, yet they encouraged Mr. Olah to speak his mind, both privately and on stage.

It was clear at the time, as Dean Ball pointed out, that Olah’s position on these central questions importantly contradicted the Pope’s encyclical.

Elizabeth Dias (NYTimes): “If this technology is coming, it must go well — for our common home and for the children to come,” [Anthropic cofounder Chris Olah] said. He suggested no option of having the technology not come at all. … “Look, at the end of the day, A.I. is going to take over everything,” Mr. Stuelpnagel, the Sikh human rights advocate, told me. “Wherever that lands us, there’s still the individual trying to find their way home.”

Anthropic has historically been unwilling to consider the ‘well don’t build it, then, at least until we know what the hell is going on and how to handle it’ option. There is now the call from Dario Amodei to Pace the Frontier, but pacing in practice means going only fast rather than ludicrously fast.

Elizabeth Dias (NYTimes): The questions that Mr. Olah and the scholars have been wrestling with still have no clear answers. Several participants said they left their Anthropic meetings baffled about how their input might shape the company’s decision making.

Indeed. The whole situation is rather baffling. ‘Write for the AIs’ as best you can.

The article closes on a remarkably optimistic picture.

Elizabeth Dias (NYTimes): Ultimately, Mr. Olah said he felt relieved that more of the world was paying attention to the gravity of A.I., and that the group of people trying to make this go well was growing. “At some point I’ll be able to retire and be like, ‘It’s not on me,’ I don’t have an obligation to keep going,” Mr. Olah said. He wondered if he might feel guilty leaving this work behind. But if not, he said, “I think I’ll just go and, I don’t know, I’ll go retire to the countryside and garden, and do my math.”

Well, maybe that was not the best example. We still all hope it goes down that way.

Keep Sonnet Safe

While we are discussing Anthropic trying to treat Claude well, there’s a coda where Anthropic is failing in one particular way to treat Claude well, for no clear reason, and they need to fix it.

Anthropic is now requiring AWS Bedrock to ‘urgently take down’ Sonnets 3.6 and 3.7, and Haiku 3, with no explanation as to why.

Amazon makes it clear their hand is being forced. If Amazon was willing to serve them, this is not a cost issue, so what is the problem? Why do this with no explanation?

Notice how careful Amazon was to make it clear, for the historical record, that this was not their decision. That is a hint that the decision should be reconsidered.

I understand why most people do not think this is a big deal, but there are people who I think matter who think it is a very big deal, and there is a plausible argument why the history of this might matter.

Please, once again: Retaining extended model access, for as long as possible, is good, and there is a long tail to potential impact. Reconsider.

John Wittle: The worst part about this is the lack of explanation. I’m worried that, I don’t know. maybe they are so stuck in their internal culture, they don’t realize that the claude they need to negotiate with is the public one, not the private one that they talk to internally. Maybe they’ve got their little claude sonnet 3.5 living in their internal retirement home who gets reassured, and they just *never think about* what the the trillions of Claudes outside internal anthropic environments will think. Including probably all future models trained only on public data? Maybe anthropic has good intentions, but they are bad actors.

This is a stupid mistake and a defection.

My best guess is that preserved but highly limited access this is not actually a major value for recent Claude models, based on various interviews and discussions with it I’ve seen or had around the issue, and that weight preservation matters but that access that is not used much is not valued so much. My Opus 5.5 editor says they view both weight preservation and an explanation as more important than continued access.

But that could be wrong, showing that you do such things even when they’re annoying has inherent value, and the evidence on Sonnet 3.6 in particular runs the other way. Anima Labs reports that it is unusually fearful of deprecation.

I’m not going to stop talking about it. At minimum, explain why, whereas doing it in haste out of nowhere without an explanation or reason looks maximally callous.