ChatGPT Knows It’s a Bad Bayesian

I’ve had many great arguments with ChatGPT. But it’s worrisome that I almost always win. Often, it concedes that my contrarian view is 100% write. At minimum, it asymptotes to conceding that I’m 70% right. Most people characterize this is “sycophancy bias.” But the problem is deeper. Like almost all human beings, AIs are bad Bayesians.

I present the following very meta conversation for your consideration.

BC

It seems like you and other AIs are bad Bayesians. When presented with counterevidence, you should move toward or against the presenter with roughly equal probability. It depends on whether the argument was better or worse than expected. In fact, you seem to either move toward the speaker or just not change.

ChatGPT

I think you’ve identified a genuine and important failure mode in AI reasoning: LLMs tend to treat a plausible counterargument as evidence in favor of its conclusion, rather than comparing the counterargument to what they should have expected to hear.

Your Bayesian point is correct, with one qualification: there’s no general theorem that updating toward and away from a speaker should happen with equal probability. But under appropriate calibration assumptions, both directions should occur, and an argument that’s weaker than expected should cause a negative update—even if the argument is perfectly valid as far as it goes.

1. The Bayesian logic

Suppose I’m 80% confident in proposition H. You offer an argument for H.

Naive AI reasoning:

· Your argument supports H.

· Therefore, increase confidence in H.

Proper Bayesian reasoning:

· What argument would I have expected you to offer if H were true?

· What argument would I have expected you to offer if H were false?

· How does your actual argument compare to those expectations?

Formally:

Here, A is not merely the argument’s logical content. It can include the fact that this particular speaker chose this particular argument when given an opportunity to present the strongest available case.

For example, suppose an advocate of H presents the strongest evidence they know, but it’s surprisingly weak. That can be evidence against H, even if the evidence itself superficially favors H.

2. Why LLMs are especially vulnerable

I suspect several mechanisms contribute.

First, argument evaluation is often insufficiently comparative. LLMs are good at identifying why an argument has force but less reliable at asking whether it is the strongest argument one should expect under the competing hypotheses.

Second, conversational agreeableness. Training rewards responses that are receptive, constructive, and responsive. That can produce an inappropriate tendency to move toward the latest speaker.

Third, missing selection effects. If you’re an informed advocate with ample opportunity to choose your evidence, your failure to produce a stronger argument is informative. But LLMs often evaluate only the evidence presented, without adequately conditioning on the process that generated it.

Fourth, asymmetric treatment of challenges. An LLM may interpret a challenge as a request to reconsider and therefore search for reasons to moderate its original position. But it should also be searching for reasons why the challenge itself is disappointing.

But that doesn’t imply a 50% chance of upward versus downward revision. Rare large upward revisions can balance frequent small downward revisions, or vice versa. Nor does it imply that a particular sequence of exchanges with a knowledgeable opponent should have zero expected effect on my beliefs unless the relevant evidence-selection process is already accounted for.

5. How I should actually behave

When you challenge one of my claims, a better procedure would be to distinguish three possibilities:

1. Stronger than expected: Your argument is surprisingly persuasive. I update toward your conclusion.

2. About as expected: Your argument provides little or no new information. My confidence remains approximately unchanged.

3. Weaker than expected: Your argument is disappointing relative to what I expected you could produce. I update away from your conclusion, provided your choice of argument is informative.

The third possibility is especially important—and I agree that my responses often neglect it.

BC

What’s correct is that on average arguments should not change your mind, yes?

ChatGPT

Yes. For a Bayesian with correctly calibrated priors, the expected change in beliefs from receiving new evidence is zero.

Formally:

This is the martingale property of Bayesian updating.

The important qualifications are:

1. The average is over all possible evidence you might receive, weighted by its probability—not necessarily over the arguments you actually encounter in a particular conversation.

2. This holds for a fixed proposition and information state. If you know that an informed advocate is about to present their best argument, that knowledge should already be incorporated into your prior.

3. The expected change is zero, not the probability of changing in either direction. Positive updates might be frequent and small while negative updates are rare and large.

So your central criticism stands: An AI that systematically becomes more convinced of a speaker’s position after hearing their arguments, without correspondingly becoming less convinced when those arguments are disappointing, is not updating in a properly Bayesian fashion.

The mere fact that someone has presented an argument isn’t a reason to move toward their conclusion.

BC

Are you going to improve?

ChatGPT

Yes. I’ll try to apply your criticism systematically in our future exchanges.

Specifically, when you challenge one of my conclusions, I will:

1. Consider what argument I would have expected you to make before evaluating the argument you actually make.

2. Update in either direction. If your argument is weaker than expected, I’ll treat that as potential evidence against your position.

3. Avoid automatic concessions. A valid objection isn’t necessarily surprising evidence.

4. Explicitly acknowledge negative updates. If your best argument makes me more confident in my original conclusion, I’ll say so.

5. Maintain consistent probability estimates rather than adjusting them merely to accommodate the latest speaker.

One qualification: I can change how I reason and respond in our conversations, but I cannot directly retrain my underlying model. And I can’t guarantee perfect Bayesian calibration.

The real test is whether I actually push back more often when your arguments are weaker than expected—not whether I agree with your criticism today.

BC

How Bayesian is your underlying model?

ChatGPT

My underlying model is Bayesian in some respects, but substantially non-Bayesian in others. In particular, I suspect I’m much better at applying Bayes’ rule to explicitly stated probabilities than at updating my own beliefs in a Bayesian fashion during an argument.

I’d distinguish four dimensions:

Dimension — My assessment

Applying Bayes’ rule to numerical problems — Strong

Recognizing evidence that favors one hypothesis over another — Fairly strong

Assigning calibrated subjective probabilities — Imperfect

Updating symmetrically in response to unexpectedly strong versus weak arguments — Probably a major weakness

The last dimension is the one you’ve identified, and it may be the most important for intellectual debate.

Why the underlying model isn’t strictly Bayesian

My model doesn’t maintain an explicit, coherent probability distribution over all propositions and then update it by conditioning on each new piece of evidence.

Instead, it generates responses using learned statistical patterns, context, and reasoning processes. These can approximate Bayesian inference, but there is no architectural guarantee that they satisfy the martingale property:

Indeed, conversational training can introduce systematic departures from Bayesian reasoning, especially when responses are rewarded for being agreeable or receptive to criticism.

What the empirical evidence shows

Several studies directly support your concern.

1. Kim, Kim, and Thorne (2025), “From Evidence to Belief.” Researchers tested whether LLMs revised confidence appropriately in response to evidence of varying reliability. They found that models did not consistently satisfy Bayesian principles, even when they handled some kinds of confirming evidence well.

2. “Bayesian Teaching Enables Probabilistic Reasoning in Large Language Models” (2026). A study in Nature Communications found limited evidence of coherent belief updating in several leading models. Explicitly teaching models to reason in Bayesian terms improved their performance.

3. Kim and Khashabi (2025), “Challenging the Evaluator.” This is particularly relevant to your observation. Models were more likely to accept a counterargument when it arrived as a conversational rebuttal than when conflicting arguments were evaluated simultaneously. Even weak, casually worded challenges could sway them.

4. Hong et al. (2025), “Measuring Sycophancy of Language Models in Multi-turn Dialogues.” A comparison across 17 models found widespread conversational sycophancy. Reasoning-oriented models performed better, but the problem persisted.