Generating Opportunity Solution Trees with AI: How Vistaly Rebuilt Its Product Around Interview Synthesis, Evals, and Repair Loops

Listen to this episode on: Spotify | Apple Podcasts

What happens when you hand your opportunity solution tree to an AI? Vistaly rebuilt its entire product to find out—and the agents were the easy part.

In this episode of Just Now Possible, Teresa Torres talks with Matt O'Connell (Co-Founder and CEO), CP Dehli (Co-Founder), and Steve Klein (Co-Founder) at Vistaly, a company Teresa partners with and builds for. V1 was a purpose-built canvas for continuous discovery. V2 is a ground-up rewrite: upload customer interviews, get a snapshot of each, and let an agentic workflow draft and update your tree.

They get specific about what broke. A chat-based synthesis experiment users found too slow. A "house of cards" problem where one badly framed opportunity corrupts every layer above. Teresa walks through the AI evals work behind one customer complaint—"I'll have to come back and clean up this branch"—including an LLM as a judge she couldn't calibrate and the orchestration change that fixed what no prompt could.

You'll hear why change sets had to be taught to the agent as the rules of a game instead of diffed at the end, how data residency pushed them onto Bedrock, and why their hardest problem isn't output quality—it's helping people comprehend what changed.

Show Notes

Guests

• Matt O'Connell, Co-Founder and CEO, Vistaly
• CP Dehli, Co-Founder, Vistaly
• Steve Klein, Co-Founder, Vistaly
• Teresa Torres, host of Just Now Possible, building the AI synthesis services for Vistaly V2

Key Takeaways

• The competition isn't good interview synthesis done faster—it's shallow synthesis that never really happened. That changes what "good enough" means.
• Layered analysis matters more than whether a step is a pipeline or an agent. If the interview snapshot is wrong, every layer above it inherits the error.
• Prompt changes run out. Balancing two opposing error modes took an orchestration change—moving the check into an agentic repair loop—not a better prompt.
• Change sets can't be diffed at the end. The same input/output tree pair has multiple valid change sets, and only the semantically meaningful one makes sense to a user.
• Users don't want to collaborate with the AI step by step. They want the answer, then the ability to correct it.

In this episode

• Why Vistaly rebuilt from scratch instead of layering AI features onto V1—and shut off V1 signups to protect the rewrite
• Rebuilding nearly all of V1's functionality in two and a half months, after three years of building it the first time
• The V1 chat interface that walked users through insights one at a time, and why "still faster than before" wasn't fast enough
• How V2 works: upload three interviews, generate an interview snapshot for each, then synthesize a first-draft opportunity solution tree
• The "house of cards" problem: garbage in at the snapshot layer corrupts the tree above it
• Using a code assertion (too many children on a node) as a cheap pre-filter before spending money on an LLM as a judge
• Four new evals and 16 experiment variants to break a seesaw between missing subgroupings and badly framed parents
• Why the fix was a repair loop in the orchestration, not a better prompt—and how that eval became a production guardrail
• Teaching the agent "the rules of the game" so it logs semantic moves (merge, move, reframe) and produces reliable change sets
• Stable IDs, data provenance, and the "basic computer science" Teresa swept under the rug in the prototype
• What beta data revealed about interview quality: sales demos, stakeholder meetings, and LLM-generated fake transcripts uploaded as "interviews"
• Building a 40-point system for generating realistic synthetic transcripts as test data
• Driving tree edits through MCP, including mapping a full Jira epic backlog into the opportunity space
• Adding an agent back to the canvas, and the versioning and rollback problem that comes with it
• Data residency, SOC 2, GDPR, and moving to European Bedrock inference
• Why model upgrades aren't drop-in: prompts are model- and version-specific, and Bedrock constrains tokens independently of the model

Resources & Links

• Vistaly — Opportunity solution tree software with AI-generated interview snapshots and tree synthesis
• Product Talk — Teresa's AI evals guide and her blog posts on building the Vistaly synthesis services
• Model Context Protocol — How Vistaly users drive tree edits from outside the product
• AWS Bedrock — Where Vistaly runs inference, including EU-hosted inference for European customers
• Just Now Possible episode with Banani — the canvas-plus-agent design UX Teresa references
• Just Now Possible episode with Zencity — the layered data pyramid Teresa references
• All Things Product episode on the quality of evidence, with Petra Wille

Chapters

00:00 Meet the Founders
01:05 How Vistaly Began
03:10 Community First Partnership
04:56 Why Build OST Software
07:18 OST Explained Fast
10:28 Evidence Inside the Tree
12:56 AI Changes Discovery
16:58 Chat UI Experiment
23:00 Should AI Do Synthesis
27:10 Rebuilding as V2
32:08 Turning Off V1 Signups
34:43 V2 Workflow and Tensions
39:35 Trusting AI Outputs
39:56 Layered Analysis Pyramid
41:25 Building Better Trees
43:34 Bad Inputs and Coaching
47:20 Teaching Through Software
50:12 Evals and Guardrails
55:23 Deterministic Change Sets
01:00:54 UX for Tree Updates
01:07:12 Chat Agents and Versioning
01:11:17 Privacy Compliance Tradeoffs
01:15:06 Model Selection Reality
01:17:32 What Comes Next
01:19:08 Closing and Links

Full Transcript

Podcast transcripts are only available to paid subscribers.