Ch. 10: Testing Assumptions

Continuous Discovery Habits* turns five this year. And to celebrate we are reading the book together.

Each month, I am releasing an in-depth reading guide that includes:

• The chapters we will be reading
• A preview of the most important concepts we'll be learning about
• Short videos you can share with friends and colleagues to help spread the ideas
• Individual and team discussion questions to help you absorb and engage with the reading
• Team exercises to help you put the ideas into practice
• Additional reading to help you go deeper on the core ideas

We'll be discussing each month's reading in the comment section and we'll gather quarterly to discuss on a live call.

Joining late? No problem. I monitor the comments on each reading guide throughout the year. Start with the current month or go back to January—whatever works for you. You can ask for help, share what's working, and connect with other readers at any point.

If you want to participate, grab a copy of the book (or dig up your old copy), share the "Spread the Love" videos, reserve some time to do the team exercises, and register for the community sessions. Let's do this!

This Month's Reading

Chapters:

• Chapter 10: Testing Assumptions

Estimated reading time: ~26 minutes

This chapter will introduce you to:

• Why we should test assumptions across sets of ideas (not one idea at a time) to guard against cognitive biases
• How to design effective assumption tests that simulate an experience and evaluate behavior
• The importance of defining success criteria upfront to avoid ambiguous results
• Why small early signals are better than large-scale experiments when starting out
• How false positives and false negatives impact your work (and why they're less risky than you think)

Need a copy? Grab the book

Share the Love with Friends and Colleagues

We learn best in community. Use the following short videos to share the key concepts from this chapter with friends and colleagues. Invite them to participate in the book club with you.

• What is an assumption test? - A structured activity designed to help us evaluate risk
• Why we test more than one idea at a time - We want to guard against bias
• How to set success criteria for assumption tests - Our goal is to get so specific that someone else can call the win or loss
• The value of early signals - And why we don't want to start with large-scale tests
• Understanding false positives and false negatives - They will happen in assumption testing (and large-scale A/B testing too)
• Two tools every product team should have access to - No, I'm not endorsing specific tools

Reflect & Discuss What You Read

When we reflect and discuss what we read, we absorb more of the material. It helps us put what we learn into practice. Don't skip this step.

This chapter is all about moving from identifying assumptions to actually testing them. The temptation is to rush into experimentation, but this chapter shows us how to slow down just enough to make our tests more effective. Focus on examining how your team currently approaches testing and where you can improve.

Individual Reflection

1. Think about a recent prototype test or experiment your team ran. Did you define success criteria upfront, or did you wait to see the results before deciding what they meant? How did that affect your ability to act on the data?
2. When you've tested ideas in the past, did you test whole ideas or specific assumptions? What was the difference in how quickly you could iterate?
3. Consider a time when you received ambiguous results from a test. Looking back, what specific success criteria could you have set upfront that would have made the results more actionable?

Team Discussion

1. Does your team tend to test one idea at a time or assumptions across multiple ideas? How does this affect your susceptibility to confirmation bias and escalation of commitment?
2. Walk through your last assumption test together. Did you simulate an experience and evaluate behavior, or did you rely on asking customers what they might do in the future? How reliable was the data you collected?
3. What's your team's current relationship with failed tests? Do you see them as learning opportunities or as setbacks? How might reframing failures as valuable data change your testing velocity?

Put It Into Practice

The best way to get better at assumption testing is to actually run tests. These exercises will help you practice the core concepts from this chapter with your current work.

Exercise: Design a Small Assumption Test

Time: 30-45 minutes
Do this: With your product trio

Pick one solution you're currently considering. Identify your riskiest assumption for that solution. Design the smallest assumption test you can run this week to evaluate it:

1. What assumption are you testing?
2. What moment in time will you simulate?
3. What behavior will you observe?
4. How many people will you test with?
5. How many need to exhibit the desired behavior for the test to pass?

Now answer: Can you complete this test in 1-2 days? If not, make it smaller.

Exercise: Audit Your Success Criteria

Time: 60 minutes
Do this: With your product trio

Review 3-5 recent tests or experiments your team has run. For each one, answer:

1. Did you define success criteria upfront? If so, what were they?
2. Were the criteria specific (e.g., "7 out of 10") or vague (e.g., "some people" or "70%")?
3. After seeing the results, did your team agree on what they meant, or was there debate?
4. What did you do with the results? Did you act on them or get stuck?

Based on this audit, identify one practice you want to change about how your team sets success criteria going forward.

Go Deeper: Additional Reading

If you prefer an audio summary of this month's reading, including the book chapters and the following resources, I've included an audio version for paid subscribers at the bottom of this post.

Related In-Depth Guide

• Assumption Testing: Everything You Need to Know to Get Started

Supplementary Reading

• Evaluating Solutions: The 5 Types of Assumptions that Underlie Our Ideas
• Stop Asking Whether or Not Questions
• How Compare and Contrast Decisions Lead to Better Product Outcomes

Other Voices

• Testing Business Ideas by David Bland and Alex Osterwalder
• UX for Lean Startups by Laura Klein

Audio Summary

This summary was produced by NotebookLM. The sources supplied were the book chapters as well as all of the additional reading.

This article is part of the CDH Book Club celebrating the 5-year anniversary of Continuous Discovery Habits. See all book club posts