Designing Against Over-Reliance

Patterns for Designing Against Over-Reliance

When trust goes too far

We usually treat trust as something to build up. More explanation, more transparency, more reassurance. But a user’s confidence in an output should match how reliable that output actually is. When confidence runs low, people ignore good work and stop using the feature. When it runs too high, people accept things that were never checked.

When users under-trust, the feature takes the hit. When they over-trust, the outcome does.

Over-reliance is a rational response to what we built

Studies suggest that over-reliance isn’t simply a failure to think critically. People often weigh the effort of checking an AI output against the effort of accepting it. When verification is difficult, accepting the answer can feel like the easier and more rational choice. On the design side, this means reducing the effort required to verify an output can be an important way to prevent over-reliance.

The lack of verification isn’t limited to low-stakes AI interactions. Research has documented automation bias in high-stakes domains like healthcare and aviation, where people can follow incorrect system recommendations even when they have the expertise to make the decision themselves. More recent research with generative AI has found similar patterns, including physicians becoming more likely to accept incorrect AI recommendations.

Attention and time are a fixed budget. When verification costs more than the budget allows, people will naturally look for ways to reduce that cost.

The design goal isn’t to make people distrust AI. It’s to make appropriate reliance easier: knowing when to accept an output, when to question it, and when to verify it.

Will this go away as models get better?

Models are getting better at questions with clear, verifiable answers, but much of what people use them for is not that. It is “what should we do here?” where the answer depends on context the model does not have: what your team can ship, what your users need, or what trade-offs you are willing to make. In those cases, the model is not necessarily wrong. It may simply be missing information that changes the answer.

Over-reliance is most likely when a system is good enough that people stop checking, but not good enough that checking is unnecessary. That is where trust can outrun reliability. And as errors become less frequent, our ability to recognize them can weaken. This creates a design challenge around calibrating reliance and helping users apply the right level of judgment to AI outputs.

There are many ways designers can approach this. Here are four design approaches that can help reduce over-reliance:

1. Reduce validation friction

The goal should be to go beyond showing users where an answer came from, and to also make verifying that answer easy enough that they will actually do it.

When a user has to click into a citation, find the relevant passage, and figure out how it supports the claim, verification becomes a separate task. The more effort it takes, the less likely people are to do it, even when the information matters.

Design patterns to consider:

• Anchor sources to the claim
  Place the source directly alongside the statement it supports, so users don’t have to figure out which citation belongs to which claim.
• Deep-link to the exact passage
  Take users directly to the relevant page, paragraph, or excerpt.
• Inline evidence previews
  Give users a lightweight preview of the supporting evidence without requiring them to leave their current workflow.
• Copy with citations
  Preserve the connection between an output and its supporting sources when users move information into another document or workflow.

Principle: Go beyond providing evidence, and design a seamless path to verifying it.

2. Break the output into pieces that can be checked

AI often presents its work as a finished answer: a paragraph, recommendation, or completed workflow. That makes review more difficult because the user has to unpack the output before they can evaluate it.

Decomposition removes the work of reconstructing the output before it can be assessed, and it can convert a single global accept into a series of smaller judgments.

Design patterns to consider:

• Break outputs into checkable units
  Present claims, recommendations, or decisions as distinct units so users can evaluate them individually rather than reviewing one large block of output.
• Separate retrieval from inference
  Make it clear what the system found in the source material versus what it concluded from that information.
• Mark what came from the user. Distinguish content the user supplied from content the system generated, so the parts of an output that carry AI authorship stay identifiable as the work moves forward.
• Show diffs and change summaries
  For agentic workflows, highlight what changed and why rather than asking users to compare the before and after themselves.

Principle: Don’t make users deconstruct an answer to understand what they’re being asked to trust.

3. Say what the answer does not cover

The previous two approaches make checking easier or more likely. This one tells users when checking is worth doing. Someone who knows an answer didn’t cover preprints won’t verify a claim, but they’ll know where the answer has a hole and can decide whether it matters for their situation.

This only helps if the user can act on it. “Searched 400 of roughly 1,200 matching papers, 2019 to present, with no coverage of preprints” can be checked against their own needs. However, a summary judgment about the answer’s quality gives them less to work with.

Design patterns to consider:

• State what shaped the answer, and what it left out.
  Show the sources, time periods, and datasets that bounded the result, plus anything unavailable or not searched. Gaps are hard for users to spot on their own.
• State the assumptions behind the answer.
  Show what the system took as given when information was missing, so users can judge whether those premises hold.
  Note: This one is generated rather than measured, so treat it as less dependable. Placement matters here, as a limitation stated before the answer influences how it’s read.

Principle: Make the scope and limitations of an output visible at the point of judgment.

A caution specific to this approach: Telling users an answer is well covered gives them a reason not to check for themselves, and the system making that claim is the one being checked. Users should be able to see what the claim rests on.

4. Treat verification as a habit rather than a step

Verification tends to decline as users become familiar with a system. When the system performs well consistently, checking can start to feel unnecessary, even when an error still matters.

Cognitive forcing functions are one way to interrupt that drift. Research has found that requiring people to make an initial judgment before seeing the AI recommendation, or asking them to choose between alternatives, can reduce over-reliance. The idea is to create moments that keep users actively engaged in the decision rather than defaulting to acceptance.

Design patterns to consider:

• Put friction where it counts. Keep users involved at the points where errors would be consequential or hard to reverse, rather than asking for approval at every step or only at the end. Low-risk steps can stay lightweight.
• Confirm premises before the work compounds. When the system has inferred goals, constraints, or scope, let users correct that before it shapes everything downstream.
• Present alternatives. Give users several plausible options rather than a single answer that is easy to accept by default.
  Note: This also makes variability a more obvious characteristic of the output.
• Use reflection instead of interruption. For longer sessions, summarize what the system did, what the user accepted, and where they made changes.

Principle: Make opportunities to exercise judgment a recurring part of the workflow, with the level of friction matched to the consequences.

Conclusion

As AI becomes more capable, users should be able to understand how much reliance the output warrants, and what level of judgment the situation calls for.

That puts an important responsibility on the interface. Evidence should be easy to verify. Outputs should be structured for review. Limitations should be visible where decisions are made. And workflows should create meaningful opportunities for users to exercise judgment, with the amount of friction reflecting the consequences of getting something wrong.

When these patterns work together, verification becomes a natural part of using the product rather than an extra task users have to remember to perform.

The goal is appropriate reliance: knowing when an output is well supported, when something deserves a closer look, and when human judgment needs to take the lead.

Designing Against Over-Reliance was originally published in Bootcamp on Medium, where people are continuing the conversation by highlighting and responding to this story.