How we think, expanded: System 3

I’ve written before about Daniel Kahneman’s two systems of thinking: System 1 that is very fast, requires very little energy, and is often wrong, and then System 2 that is slower, far more energy intensive, and more often right. Given that our entire body is highly optimized for energy use, our brains will default to System 1 wherever they can and this is the realm of cognitive bias.

If you aren’t familiar with systems 1 and 2, you may want to read that article first as this post is building on that, not replacing it.

Earlier this year, two researchers at Wharton released a paper arguing for a System 3, which while always around, is now much more obvious in the age of AI.

“We introduce Tri-System Theory, extending dual-process accounts of reasoning by positing System 3: artificial cognition that operates outside the brain. System 3 can supplement or supplant internal processes, introducing novel cognitive pathways.” Steven D. Shaw and Gideon Nave, “Thinking—Fast, Slow, and Artificial”

Note that I’m not convinced that System 3 is really a separate system. Most of what follows looks to me like System 1 doing what it has always done with an answer that sounds convincing. Still, it’s a useful shorthand for the thinking we hand to something outside our own heads, so I’ll use their label.

To test their theory, they used the Cognitive Reflection Test, the same family of puzzles the bat-and-ball question comes from. Participants could ask an AI assistant for help, and the researchers had secretly arranged for that assistant to be right on some questions and wrong on others.

People asked the AI on just over half of the questions. When they did, they took its answer 93% of the time when it was right, and 80% of the time when it was wrong. Without the AI, people got about 46% of the questions right. With a correct AI that went up to 71%. With a wrong one it dropped to 32%.

Having the AI available made people more confident in their answers, by almost 12 percentage points, even though roughly half the AI’s answers were wrong.

Shaw and Nave call this cognitive surrender, and they’re careful to separate it from something we’ve always done.

“Whereas cognitive offloading is a strategic delegation of deliberation, using a tool to aid one’s own reasoning, cognitive surrender is an uncritical abdication of reasoning itself.” Steven D. Shaw and Gideon Nave, “Thinking—Fast, Slow, and Artificial”

Using a calculator is offloading. I still know roughly what the answer should be and I’d notice if it said 4,000 when I expected 40. Surrender is when there’s no expectation left to check against.

So is any of this new? I don’t think it is. We’ve always taken answers from outside our own heads and not looked too closely at them. AI just makes it much easier to see.

Programmers have been doing this with Stack Overflow for years. Nobody is going to be upset if you ignore an answer there, and yet we copy them anyway. A 2017 study looked at 1.3 million Android apps and found that 15.4% of them contained security-related code copied from Stack Overflow. Of those, 97.9% contained at least one insecure snippet. Copying the code doesn’t prove that nobody checked it, but the vulnerabilities came along for the ride, so whatever checking happened didn’t catch them.

What changes with AI is how convincing the answer sounds. Researchers at Purdue took 517 real Stack Overflow questions and gave them to ChatGPT. 52% of its answers contained incorrect information, and yet when people compared the two, they preferred the ChatGPT answer 35% of the time and missed the misinformation 39% of the time. The reasons they gave were how comprehensive and well written the answers were.

That lines up with what Shaw and Nave say makes surrender most likely: an answer that’s “delivered fluently, confidently, or with minimal friction”. In System 1 terms, a fluent, confident answer is exactly the kind of thing that gets believed without System 2 ever waking up.

When the advice comes from another person, we tend to do the opposite. There’s a long line of research on how people take advice, and the consistent finding is that we discount it and favour our own opinion. A meta-analysis covering more than 17,000 people found that, on average, people moved only about a third of the way toward an advisor’s estimate, rising to about half when the advisor was described as high quality.

So why would we discount a colleague’s estimate and then take an AI’s answer 80% of the time, even when it’s wrong?

Part of the answer is that we’ve always deferred more to machines than to people. Researchers expected the opposite, and in a series of experiments found that “lay people adhere more to advice when they think it comes from an algorithm than from a person”. They called it algorithm appreciation.

Once people see an algorithm make a mistake, they lose trust in it much faster than they would in a person who made the same mistake. That’s called algorithm aversion. We hold machines to a higher standard than we hold each other.

Weirdly, a significant number of people don’t think of AI as an algorithm at all. In one survey, 67% of people attributed some possibility of consciousness to ChatGPT, and the more often they used it, the more likely they were to say so. We talk to it, and it talks back in the first person. Researchers at Google found that small human touches, like giving the system a voice or having it say “I”, made people rate its answers as more accurate. Long before chatbots, a self-driving car with a name, a gender, and a voice was trusted more than the identical car without them.

A personified AI may get both: the authority of an algorithm and the forgiveness given to a person. That would explain how the Wharton participants kept accepting answers about half of which were wrong.

Shaw and Nave aren’t immune to this either. Their paper describes System 3 as “a co-agent in reasoning” and puts it right alongside the two systems that live inside our heads. Even the name is a small act of personification.

Stack Overflow never had that advantage. Nobody thinks of it as a person. The trust there comes from upvotes and the green tick on the accepted answer, and you can always see that the answer was written by another programmer who might be wrong. That was still enough to get insecure code into nearly 200,000 apps. AI starts with that same habit of ours and then talks to us like a person on top of it.

So what helps? In their third experiment, adding a small cash reward for correct answers plus immediate feedback more than doubled how often people rejected the wrong AI answers, from 20% to 42%. That’s a big improvement, and it still means most of the wrong answers got through.

The study bundled the reward and the feedback together, so we can’t tell which one did more of the work. Either way, both are about motivation, and that tells us that how motivated people are to check the answer makes a real difference. I’ve written about motivation many times, and an external reward like cash is one of the weaker ways to get it. Someone who enjoyed catching the AI’s mistakes would likely do a much better job of it than someone who was paid twenty cents per correct answer.

The study hints at this too. Shaw and Nave measured something called “need for cognition”, which they describe as people’s “tendency to engage in and enjoy effortful cognitive activities”, and the people who scored higher surrendered less. That’s very close to intrinsic motivation, where the work is enjoyable for its own sake.

When we first talked about systems 1 and 2, we recognized that it was ok that System 1 made most of our decisions. System 1 isn’t bad just because it makes a lot of mistakes (it does). It’s actually exceptionally helpful and we need it. We’re not capable of doing everything with System 2 - our brains just don’t have the energy or the capability. The key is that we want to identify when System 1 is leading us in the wrong direction, when we’re making the wrong decisions based on our cognitive biases, so we can correct them.

In that same way, offloading to System 3 is fine; surrender is what we want to catch. The point is that we want to be aware of what we’re doing. We need to be able to recognize when we’re not getting the results we want, so we can pivot and do something different.