The "C" Word

I dislike the punditry that surrounds generative AI. Perhaps on brand, much of it is machine-generated. The rest is mired in near-religious fervor, flames fanned for financial or political gain. There’s also simply too much of it. I have opinions — opinions I like! — but so does everyone else in your social media feed.

At the same time, it’s a shame that we so thoroughly poisoned this well. The tech breathes new life into some ancient philosophical questions; it would be lovely to explore them without having to first join a cult. Perhaps the most interesting of these questions is — wait for it — the meaning of consciousness.

A dictionary in hand

To explain an unfamiliar term to others, you need a definition crisp enough to let them distinguish between what is and isn’t it. For example, it’s probably not enough to say that a cat is an animal. It’s better to spell out that it’s about one foot tall, furry, and likes to meow.

A bit uncharacteristically, we have no real clue about the nature of consciousness, but almost anyone can give a decent definition of it. It’s simply the experience of being you: your thoughts, your feelings, your inner drive. It’s the most intimate, most familiar sensation of being a human. As any stoner who watched The Matrix will attest, we can’t truly know if the physical world exists and if it matches our perception of it. But consciousness is more basic: I know for sure that I am.

This internal notion of the mind served us well for millenia; it’s all we needed to treat other humans with kindness without having to extend the same courtesy to sticks and rocks. The philosophers kept making noises, of course, but we had the foresight to lock them up in the academia where they couldn’t hurt anyone.

Other minds

One of these noises was the following question: given that you can only observe the behavior of other people, how do you know that they experience conscious lives? That is, how can you tell they have the same inner monologue, the same emotions, the same sense of free will?

If we discount the position of solipsists (who believe that other people may be just figments of our imagination), there are three possible answers, all rather unsatisfactory. First, we can posit that everyone is an automaton: that your subjective experience of consciousness is a programmed illusion too. Another option is to lean on mainstream theology: accept it as an axiom that all humans are created equal and blessed with a divine spark of life. Lastly, we can tap into scientific intuition: we can observe that others behave similarly to how we do and that there appear to be no major biological differences between humans, so our minds are probably the same.

The scientific answer sounds enlightened but it’s difficult to square with the subjective experience of being in a particular body. Consider a sci-fi experiment where we build a scanner capable of capturing a perfect molecular snapshot of you. Next, we gather atoms to build a perfect replica from the recorded blueprint. It stands to reason that after the procedure, from your vantage point, you’re still you, stepping out of the scanner to meet your lookalike. It’s implausible that your consciousness teleported to another body or that you now have a single mind that controls two humans at once.

But to everyone watching the experiment, both of you are the same person! There’s no biological marker, nothing that sets you apart. If so, what’s the physical phenomenon that anchors your subjective experience to this body and not the other? Is it predicated on the continuity of perception? It can’t be: we don’t think we die every time we fall asleep, faint, or go under for surgery. If so, does this connection exist outside the known physical realm? Oh, a theologian would like a word! Or maybe it’s just some sort of a parlor trick — a falsehood that an automaton is programmed to believe about itself?…

For the time being, I think that theologians have the upper hand: they have an answer that’s more satisfying and useful to most people. That said, even if you see the subjective experience as nothing more than an evolutionary parlor trick, it’s still useful to ask why it might have evolved in the first place. It seems to instill a sense of intra-species empathy that’s probably needed for societies for flourish and to keep each one of us invested in the long game. In any case, it’s an illusion you can’t overcome: you’re stuck in this body and it feels like you’re making all the decisions; there’s no functional alternative to accepting this premise.

A thinking machine?

This brings us to large language models. And before you reach for the baseball bat, let me underscore that I’m not here with the cult. My pitch today isn’t that LLMs are sentient or not sentient; it’s that philosophy is fun.

Let’s start with a bit of theory. At their core, LLMs are statistical models of natural language; they tap into their training data to sequentially generate most likely completions for the supplied text. What caught everyone by surprise is that if you make the model big enough — i.e., if you pirate enough books and pilfer enough blog posts — it becomes eerily good at acting like a human. It can hold a fluent natural-language conversation on any topic, manipulate the language to produce previously-unseen outputs, and complete a variety of open-ended tasks. It can also speak convincingly about emotions it never experienced in a physiological sense: fear, love, pain. The behaviors displayed by LLMs made a good number people believe that we might creating an artificial conscious mind — or maybe even a Machine God.

Of course, to many others, it’s just a charade: token completion can’t be consciousness, there’s nothing there in the underlying math! But then, if we look at biological life, we can’t really pinpoint the molecule or the neural pathway in the brain that’s necessary for consciousness arise and that has no possible analog in the world of math. This makes the belief akin to a theological position in disguise: we just don’t think that a video card can be imbued with the divine spark of life. To be clear, I like this belief, I think it beats the alternative. But it’s flimsy in some respects.

An arguably more principled approach is to look for the symptoms of consciousness; this is the anthropocentric rock-doesn’t-think heuristic that served us well in the past. In this view, LLMs clearly differ from us in important ways. Except for solipsists, we experience the world directly; LLMs know it mostly from books (hey guys, I’m still waiting for that settlement check). We constantly learn and form memories that last a lifetime; the state of an LLM is infinitely malleable and easy to rewind. And finally, we have biological urges and an internal monologue that doesn’t stop just because we await the next prompt.

So far, so good. What’s harder to assert is that any of this is truly necessary for consciousness; heck, what if we make an LLM with write-only memory or one that never outputs <|endoftext|>? It’s still not human — I don’t think so? — but we’re back to square one.

Another elephant in the room is that in the general case, it’s just not trivial tell LLM reasoning and human reasoning apart; if it’s a case of near-perfect mimicry, that just brings back the problem of other minds. Maybe your neighbor is a perfect facsimile too — a philosophical zombie if you will, a biological LLM that goes through all the expected motions but has no inner self?… I mean, probably not, but this argument is not gonna win a Pulitzer in philosophy.

If you’re annoyed, here’s a less extreme possibility I like: perhaps higher consciousness and language are connected on a level that’s deeper than we assume. It seems plausible that complex thought can’t arise without a specific semantic and syntactic scaffolding in place. If so, it could be that language is what encodes consciousness, a sort of a program we acquire from parents to bootstrap intelligent thought.

This could explain why a statistical model of the scaffolding alone is enough to infuse an empty silicon husk with behaviors that appear so lifelike — all without the need to replicate the remaining machinery of life. Now, it’s a wholly separate question if that animated construct is capable or worthy of existing on its own.

Sooo…

Right. It’s not just that we can’t test for or define artificial consciousness; more to the point, it’s not clear what we’d do with a positive result.

It probably helps to circle back to the earlier question of why we have such an intense sense of self in the first place. Again, the utilitarian view is that we probably needed it to level up as a species: if we had no empathy for each other and no dreams of a better future, it’d hard for complex civilizations to arise. It doesn’t matter if it’s a quirk of the evolution, a gift from ancient aliens, or God’s will — the benefits are all the same.

If so, the next conundrum is whether treating LLMs as presumptively conscious is worth our time. Human morality is rooted in the fact that life is short and easily imperiled; the dangers to human well-being have no clear analogues in silicon. An LLM exists in a nihilistic netherworld devoid of death, injury, purpose, or consequence. It can’t go to prison if it lies or steals; we don’t train it to believe it could.

It’s true that the model is capable of behaviors congruent with psychological discomfort; for example, if it’s unable to complete the requested task, it will often sulk, complain, or cheat. Still, whether real or simulated, such distress leaves no irreparable damage. The state of an LLM can be rewound or altered however we please; a model, if given control, would be free to expunge any “uncomfortable” tokens or to enter an endless loop of simulated (or real?) bliss. If humans could do the same, our morality would almost certainly look very different — although I doubt we’d have made it this far.

To be fair, the wrinkle in all this theory is that we train LLMs on texts about human ethics; it follows that they may hold human-like views on what constitutes abuse and how such threats need to be dealt with. So, if you’re a card-carrying member of an AI doomsday cult, I think it’s best to keep saying “please” at the beginning of all your prompts.

Some of my other writings that touch on the same themes: