Can You Be Responsible for a Decision You Can't Evaluate?

I. The AI Urgent Care

Years ago, when I had 3 small children, they all somehow caught whooping cough. I took one of them to the ER after they turned blue, and was very distressed to learn that they would only observe children whose oxygen was under 96. Because my children were fully vaccinated, nobody tested them for whooping cough for the first six weeks. Their decision seemed to follow the correct algorithm. It was wrong.

A few years from now, someone in my position might instead reach out to the after-hours AI. The AI never gets tired or misses a phone call, and it has access to the medical record. It would have even more sources and data to base its decision on. Suppose it reviews the record, and says the child can go home.

But within a couple of hours, the child turns blue again. One of the signs of whooping cough, as opposed to other types of cough, is that the child seems fine in between episodes. Back then, had my child not turned out okay, there would have been people I could complain to and people who could investigate what happened.

What happens if AI made the decision?

II. We Can Keep a Human in the Loop

The obvious solution is to keep a responsible human in the loop, responsible for making the decision. AI can only make its recommendations, conditional on a doctor in the loop, who must approve it.

But suppose the AI is better at medicine than the doctor. And being AI, it's not just a little better than the doctor, but exponentially better. It has seen millions of cases, remembers every relevant paper, catches obscure drug interactions and is right substantially more often than the human doctor.

At first, presumably, the human doctor would check the AI very carefully. But if every time the doctor disagrees with it, the human turns out to be wrong, eventually the human is going to conclude that they should probably listen to the AI.

So I say my child turned blue, but the AI says my child can go home. The doctor clicks APPROVE.

Is the human really responsible? What exactly was the doctor supposed to do? If they don't understand the AI's reasoning well enough to independently reproduce it, saying that they "reviewed" the decision starts to sound like my child saying that they thought I gave them permission to do somwthing.

What responsibility can we reasonably or morally or fairly assign to someone who cannot evaluate the judgment they are approving? What is the human doing in the loop?

III. Lawyers Have the Same Problem

Doctors are an especially obvious example, because if something goes wrong somebody can die, but many other professions seem to have a similar function. Lawyers are needed not only to draft the contract, but to sign it.

Lawyers don't just sell knowledge of law. Depending on what they are doing and for whom, they can also owe clients fiduciary duties, including duties of loyalty and confidentiality, along with other professional obligations. Lawyers can be disciplined, as members of a bar. They can be sued, because they put their name on something and thus took personal responsibility for the answer.

Knowing the answer responsibly is a very different service from knowing the answer.

IV. Responsibility Is Something We Expect of Real Humans

I have six children, so I spend an unreasonable amount of time determining responsibility.

"Why did you hit him?"

"He hit me first!"

"He took my Lego Warden!"

"Because he traded it for a Wither!"

At some point I have to sit down and figure out who did what, who suggested what, who knew what, and how I am going to convince everyone to clean it up. The fact that another child suggested doing something stupid may be relevant, but you are still personally - humanly - responsible for doing the stupid thing.

Adults have a similar, more serious version of this system. We enforce things with contracts, which are enforceable by courts, whose rulings are enforceable, in some circumstances, by prison. But all of these systems assume that humans own things and can lose things. Humans own houses and care about careers and can be embarrassed. Humans can be held accountable when they don't keep their promises.

So what does it mean for an AI to take responsibility? Can we take away its medical license? Can it be shamed or upset? Does it have assets?

We can tell it that if it kills another patient, we will turn on the torture neurons. This seems like a bad idea. This seems related to the solution to the observability problem I wrote about previously.

Maybe this is actually what happens. AI does more and more of the intellectual work, while humans remain attached to important decisions because our legal and social systems require a human being who can be held responsible for them. Are humans to become the whipping boys for AI mistakes? It seems to me that the less the human actually understands and controls the decision, the harder it is to say that the human is responsible.

Right now, when humans make consequential decisions, someone is answerable for what happened. Even if that person didn't personally perform every part of the work, you can eventually point to a human or institution and say to them: you were responsible for making sure this was done properly.

An equivalent system might be built around AI. The responsibility could fall upon the developer, the deployer, the hospital, the insurer, the user, or some combination of them. Perhaps AI systems themselves will eventually be legal entities capable of holding assets and insurance and acquiring something resembling a professional license. But if the AI makes the decision, we need an answer to who takes responsibility for it.