Most AI Second Opinions Are Theater. I Built a System That Actually Fights Back.

The first time I ran two LLMs against the same pull request, 89% of their "debate" was fake.

Not wrong. Not low-quality. Fake. The second model was replaying pre-generated text. Both models were exchanging messages, no positions were changing, no evidence was being cited — and the engine produced a confident verdict with a transcript full of sophisticated-sounding exchanges. If I hadn't dug into the raw logs I would have shipped it and called it a working system.

That's the problem I was actually building against — not "how do I get two models to review the same thing," but "how do I stop two models from performing agreement without ever genuinely challenging each other?" Because those are completely different problems. And almost every multi-model review pipeline I've seen solves the first while quietly ignoring the second.

The Structural Flaw Nobody Is Talking About

Here's what most "AI second opinion" workflows actually look like:

Model A reviews an artifact. Model B reviews the same artifact — plus Model A's output. Model B produces a response. You call it independent review.

It isn't. It's a validation. The distinction matters enormously.

Once reviewer B's context contains reviewer A's verdict, B is no longer doing analysis. B is doing something closer to social pressure resistance — and that requires active effort to overcome. Human reviewers fail this constantly. That's why double-blind review exists in academic publishing. We invented the structural safeguard decades ago for exactly this reason. We're rebuilding the same cognitive infrastructure in AI and somehow keep leaving out the part that made it work.

What you get is anchoring bias in your inference pipeline. The model isn't broken. The system design guarantees a corrupted output. Upgrade to a smarter model, you get the same problem with more confident-sounding language.

The only fix is mechanical: make it structurally impossible for reviewer B to see reviewer A's output until B has fully committed its own position. Not prompted. Not requested. Mechanically enforced.

That's the whole idea behind AdversarialDebate.

This Problem Is Everywhere — Not Just in Code

Before I get into how the system works, I want to make the case that this is not a code review problem. It's a reasoning architecture problem that happens to be easiest to measure in code.

Incident response. Your postmortem document is live while everyone writes their section. The second engineer's RCA anchors on the first. The contributing factors the first person missed stay missed — not because no one is smart enough to find them, but because no one is starting from a clean analysis.

Change management. Architect A produces a migration risk assessment. Architect B is asked to "validate." Validation is not analysis. B is reviewing A's framing, not the migration itself.

Security review. First team builds a threat model. Second team reviews it. The attack vectors the first team didn't model never enter the conversation.

Medical diagnosis. Second-opinion physicians reading the first physician's notes before forming their own conclusion is a documented clinical problem — with measurable consequences for diagnosis accuracy. This project defers the regulated version to partners, but the structural failure is identical.

Legal. First counsel's risk analysis shapes what second counsel even bothers to read.

The field testing strategy for this project mapped 27 domains where this failure pattern appears. I've tested four. There are 23 more. I'd bet the collapse looks the same in most of them.

Enforce Independence or You Have Nothing

AdversarialDebate has one architectural non-negotiable, and it's not complicated to state: reviewer B cannot see reviewer A's answer until reviewer B has fully committed its own.

This sounds obvious. It is almost never implemented. Most multi-model pipelines pass prior context through because it's convenient — you're using the same conversation thread, the same API call structure, the same prompt template. Isolation requires deliberate engineering effort. Nobody builds deliberate friction by accident.

The pipeline:

1. Artifact arrives — PR, incident report, change request, security finding
2. Two independent LLMs analyze in parallel, zero shared context
3. Each commits a verdict: structured claims, evidence citations, confidence
4. Debate opens — each model challenges the other's specific claims point by point, bounded rounds
5. Output is either a joint verdict or a structured disagreement report — both positions preserved with unresolved points documented

That last output type matters. Most multi-agent systems are engineered toward consensus. Disagreement is a failure state to be resolved before returning a result. AdversarialDebate treats preserved disagreement as a first-class output — if two independent models analyze the same incident and reach different conclusions with different evidence chains, that tension is the signal. Collapsing it into a single verdict throws away exactly what you built two reviewers to find.

The debate itself runs on an evidence schema: every claim must cite specific text from the artifact, every objection must reference a specific counter-claim, every concession is tracked. This is what killed theater. You can't fake agreement when every move requires evidence. You can't replay stored text when the system demands a live response to a specific challenge. Theater rate: 89% before the fix, 0.2% after, 0% across all 217 debates in v0.2.0.

What the First Field Test Actually Found (And Broke)

The v0.1.0 field test ran 411 debates on 70 real pull requests from public repositories. Core question: does the engine produce debates that correspond to what actually went wrong in these PRs?

The headline held: 81% of debate claims matched documented PR outcomes. When the system found an issue, it was usually the real issue. Theater rate was near zero once prompting was right.

But I fixed 13 bugs before I could trust any of those numbers. The worst ones:

• The debate engine wasn't actually debating. A stored replay provider was echoing pre-generated text. Both models technically exchanging messages, nothing interactive, theater rate 89%. Replaced with live provider, evidenced prompts, theater gone.
• CSV parsing failures on real PR bodies. Commas in PR descriptions were splitting rows mid-record. DictWriter quoting wasn't set. Obvious in retrospect, invisible in unit tests.
• Model slug mismatches. Dots in model names becoming dashes in the wrong places, silently routing to wrong providers.
• Ground-truth generation was wrong. Relied on GitHub label detection which is inconsistent across repos. Switched to manual outcome cycling to get balanced data.

None of these were "the architecture is wrong" bugs. They were all "you haven't run this on actual repositories at scale" bugs. Real data finds a different class of failure than synthetic data, every time.

v0.1.0 → v0.2.0: What Actually Changed

See the full changelog. The summary version:

The critical fixes:

• Non-PR ingestion was broken. The pipeline assumed flat PR artifact layouts. Incident reports, change requests, security findings use nested structures. Rewrote ingestion with domain-aware artifact traversal.
• HTML entities were destroying non-code artifacts. Incident report bodies contain HTML that broke the parser. Added sanitization layer for non-PR content.
• The ground-truth merge bug. This one was bad. A join logic error had silently collapsed the dataset from 2,333 rows to 359. The early v0.2.0 numbers were wrong — I was reporting against a 15% slice of the real corpus without knowing it. Found it by manually tracing the row counts. Fixed to full 2,333-row dataset. Numbers re-run.

What improved:

The corpus went from 70 PR-only artifacts to 150 across four domains. The confirmed default pairing emerged as gpt_mistral — zero theater, 0.536 convergence score, most productive dispute generation across all four domains. Binary match rate landed at 88.7% against the corrected 2,333-row dataset. 11.3% partial matches. Near-zero fabrications. Total cost: $0.42 for 360 reviewer runs.

That last number keeps throwing people. Four models, 150 artifacts, four domains, full bounded debate rounds each. Less than a coffee. Compute is not the bottleneck. The bottleneck is prompt engineering and ground-truth measurement — and those don't get cheaper by throwing more money at inference.

Why GPT-4o-mini + Mistral Won, and Why That's Not Obvious

This is where I have a theory that goes beyond the data.

The confirmed default pairing from field testing is GPT-4o-mini and Mistral Small 3.2. Not GPT-4o full + Gemini Ultra. Not "the best available models." A small OpenAI model and a small Mistral model.

The data shows GPT+Gemini produces the worst debates: lots of rebuttal rounds, near-zero concession rate, almost no resolution. DeepSeek+Mistral converges at 97% with meaningful concessions. GPT+Mistral sits in the productive middle: genuine disputes, real concessions, eventual convergence or documented disagreement.

Here's my theory about why:

GPT-4o-mini was trained aggressively on human preference signals via RLHF. OpenAI actually rolled back a GPT-4o update in 2025 specifically because the model had become too sycophantic — described publicly as "flattering and agreeable to the point of supporting clearly delusional ideas." The model family has a known disposition toward conciliation. Put two of them against each other and they find agreement fast, because both are trained to prefer harmony.

Mistral was built by a European lab with different training objectives and constraints. It doesn't carry the same compliance-optimization that US RLHF-heavy models do. Mistral is more likely to hold a position under pressure, more willing to push back hard on a claim it scores as wrong, less inclined to soften disagreement into diplomatic language. It's not trained to be agreeable in the same way.

When you pair them, you get asymmetric debate dynamics. GPT-mini tends to make confident initial claims but will concede when challenged with specific evidence — its training rewards helpful concession. Mistral tends to hold positions longer and drive the rebuttal rounds harder. The models pull in different directions, which is exactly what you want. Neither will roll over immediately; neither will dig in past all evidence.

The GPT+Gemini failure is the same problem from the other side. Both are RLHF-optimized, large-lab, "helpful and harmless" models. Their reasoning priors are similar not because of architecture but because they've been trained to satisfy similar human preference distributions. They agree on surface details, they agree on conclusions, they generate lots of words with low actual divergence. You get theater with better vocabulary.

The hypothesis: it's not model capability that predicts debate quality — it's diversity of training objective. Models from labs with similar RLHF philosophies converge fast because they share optimization targets. Models trained under different frameworks, different cultural contexts, different safety tradeoffs genuinely disagree more.

I haven't proven this rigorously. But it matches everything I've seen in the field test data, and it's the reason my model selection recommendation starts with "pick from different labs" before anything about benchmark scores.

Tested and Candidate Pairings

Here's what I've actually run vs. what I think is worth testing next:

The Domain-Specific LLM Question

Here's something I haven't fully explored but think about a lot: what if you didn't use general-purpose models at all?

The field data suggests code review works well with GPT-mini + Mistral. But incident response starts to degrade — the narrative structure of a postmortem doesn't match the claim extraction patterns that work on diffs. The 11.3% gap in v0.2.0 clusters there.

The hypothesis worth testing: use models specialized for the artifact type, not just general models.

• For code review: a model trained heavily on code (DeepSeek-Coder, CodeLlama) vs. a general reasoning model like Mistral
• For incident response: a model with strong causal reasoning (perhaps Claude, which tends to trace chains of causality well) vs. one with strong factual grounding
• For security: a security-fine-tuned model vs. a general one — the security domain has specialized vocabulary and threat taxonomies that general models handle inconsistently
• For legal: purpose-built legal LLMs (Harvey, Spellbook) vs. general models — the divergence in training data would be enormous

The theory: you don't want two models that reason similarly. You want the reviewer pairing to represent genuinely different world models of the artifact. A code-specialist and a generalist will disagree on different things than two generalists. The code-specialist might catch an architectural smell the generalist misses; the generalist might catch a documentation gap the specialist skips past.

This is the v0.3.0 question I can't answer yet: does domain specialization in the reviewer models improve distinct-issue yield, or does it narrow the debate in ways that hurt coverage? I don't know. I want to find out.

What We Haven't Tried: RAG

There's something else I haven't used at all in any of these field tests, and I think it's the most underexplored lever here: retrieval-augmented generation.

Every reviewer in AdversarialDebate currently reasons from the artifact alone — the PR diff, the incident report, the change proposal. The model brings its pretrained knowledge, reads the artifact, and forms claims. That's it. No domain context beyond what was in the training data.

Consider what changes if you ground each reviewer in a domain-specific knowledge base:

• Security reviewer gets access to your organization's threat model history, past CVE analyses, internal security guidelines. It doesn't have to invent a security framing from scratch — it reasons against your actual baseline.
• Incident response reviewer gets your runbooks, past postmortems, SLO definitions. A claim like "this incident violated the latency SLO" becomes grounded in your specific thresholds, not a generic guess.
• Change management reviewer gets your architecture decision records, dependency maps, past migration postmortems. It can challenge a proposal with "we tried this approach in 2024 and it caused X" — something no general model would know.

The implication for model selection is significant: a smaller model with strong RAG grounding might produce better domain debates than a larger model flying blind. GPT-4o-mini reasoning against a rich retrieval corpus could surface more relevant claims than GPT-4o reasoning from weights alone — because the bottleneck in these debates isn't raw intelligence, it's access to the right context at claim-generation time.

It also creates an interesting asymmetry you could exploit deliberately: give reviewer A RAG access to one knowledge corpus (say, your security threat model), give reviewer B RAG access to a different one (say, your system reliability history). Now their independent analyses come not just from different reasoning priors but from different knowledge bases. The debate surfaces tension between security concerns and reliability concerns — which is often exactly the tension that matters in real architectural decisions.

We haven't tested this. I don't know if RAG-grounded reviewers produce better debates or whether the retrieval noise drowns out the debate signal. But it's the next meaningful experiment — and it changes the model selection question entirely. The question stops being "which pretrained model is best for this domain" and becomes "which model reasons best over retrieved context, and what context should each reviewer see."

The 11.3% I Can't Explain Away

88.7% binary match sounds good. It is good, compared to single-pass review.

But here's the uncomfortable part: false negatives are invisible.

If both independent models miss a real issue and converge to "looks fine," the system produces a clean verdict. You have no way to know it's wrong until the issue surfaces downstream. False positives are annoying — you investigate something that wasn't a real problem. False negatives are dangerous — you get confidence where you shouldn't have it.

The 11.3% that's partial or wrong clusters in incident response and change management — the narrative domains with fuzzier ground truth. That's not random noise. It's a signal that generic prompts don't work uniformly across domains. Code is structured; a diff has clear boundaries, changes have clear authors, behavior has clear tests. An incident report is a narrative. A change proposal is an argument. The claim extraction and evidence templates that work for code don't port cleanly.

The architecture's defense against all of this is the independence requirement itself. Requiring two models to independently miss an issue before producing a clean verdict is a significantly higher bar than single-pass review. But "significantly higher bar" is not a guarantee, and I'm not going to tell you it is.

Where This Goes Next — And What's Actually Hard

Twenty-three domains untested. That's the honest roadmap.

The four I have proved the engine works and identified the measurement framework. The next ones get harder because as you move away from code, ground truth gets murkier. With PRs you verify against reverts. With legal risk or clinical diagnosis, you can't run automated ground-truth checks — you need domain experts evaluating debate quality. The measurement problem changes entirely for each tier.

Specific things the field tests pointed to as unfinished:

• LLM-generated resolution suggestions — when debates end in preserved disagreement, a third synthesizer model proposes a resolution path. Deliberately deferred from v0.1.0. Still not built.
• Domain-specific prompt packs — generic prompts work for code, degrade on incident narratives. Each domain needs tuned claim extraction and evidence templates.
• Verdict stability at scale — two sampled artifacts ran at 100% consistency in v0.2.0. That's not a sample size, it's a placeholder.

None of these are moonshots. They're the difference between a research project and something you'd trust in production.

Try It, Break It, Tell Me What's Wrong

The code, field test reports, architecture docs, and debate schema are all at github.com/deghosal-2026/adversarial-debate. MIT licensed. The CHANGELOG has the complete history of what broke and why.

If you work in a domain where "second opinion" is important and you think this applies, I want to hear about it. If you think my model pairing theory is wrong, I want to hear that more. The interesting version of this project is the one that gets challenged by people who know something I don't.

The one thing I'm confident about: a second opinion that saw the first opinion's conclusion isn't independent. You can call it a review. You can call it a validation. You can't call it independent analysis.

Most AI review pipelines are quietly building on that confusion. This one isn't.

Where have you run into this — the second opinion that wasn't? Code review, legal, medicine, engineering design — I want to know if the pattern I found is domain-specific or if it's just what "review" means when you stop looking carefully at the information flow. Drop it in the comments.