Build your own vibe check in an afternoon

In August, and I ran a lightning lesson on how we pick which model to use. Since the talk, two more have arrived: Claude Fable 5.1 on September 1 and GPT-6 Astra on September 3. This is how I decide whether a new model is worth switching to. In Part II, I’ll show how I use several models on the same work.

Each release gives me another set of benchmark charts to read. The harder comparison is what happens when I put the model to work on a spec or a pull request.

Benchmark: Artificial Analysis - Intelligence

The benchmarks themselves measure things like physics reasoning and PhD-level exam questions, which are fascinating markers of raw intelligence in their own right (personally, I love seeing what terminal bench releases, like their new science bench 0.1 benchmark). Once a benchmark becomes a target, it can pull optimization toward the test itself. That’s the concern behind Goodhart’s Law, and it’s a reason to check more than the headline score. None of that tells you whether the new model is better at reviewing your product spec or catching the bug in your pull request.

So I use a few examples from my own work to decide whether to switch. Putting the first set together takes an afternoon, which I can reuse it when the next model ships. We’re going to cover this workflow in more depth in this series.

AI Product Playbook is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

Think about how you already use agents

Eric averages about a billion tokens a day across his agents. He’s a solo founder with one collaborator, building an investing app called Bloom (400K downloads), so AI is most of his team, and a lot of that spend happens while he’s asleep. My usage is shaped differently: most of my tokens burn during the day while I’m iterating on specs and prototypes with a few coding agents in parallel, plus a background job or two overnight.

I can judge a spec review because I know the product and the decisions I’m trying to make. The same goes for turning my rough notes into a pitch or catching a bug in a pull request. I have enough context to push back on the answer. That’s where I start when I’m trying a new model.

Change one thing at a time

One thing to sort out first: a model comparison depends on the setup around it.

The model is one choice. Your harness is another: are you using Claude Code, Codex, Cursor, or a chat window? Then there are the skills you’ve given it (a spec-review checklist, say) and the context it has to work from. A missing doc can matter more than which model you picked.

So I’ve actually made this mistake: swapped the model, got a worse answer, and almost dismissed it before realizing I’d changed the skill too. The skill was the problem. It’s an annoying thing to catch after you’ve already started forming an opinion about the model. Keep the tool, skill, and docs fixed for the comparison. If you’re doing this with a support bot, check that both runs retrieved the right help-center article as well.

You’ll see me use spec reviews below because that’s work I already do. If you’re testing a model for your product, use recent customer sessions in place of my agent sessions.

Build the vibe check

Step 1: Pick a task you actually do. Eric picked ad-spend decisions for Bloom; I picked product spec reviews. You probably have an equivalent from this week. Use that. The snake-game demos are fun, but I don’t have enough snake games to build for that to help me choose.

Step 2: Pull your own traces. Start with the conversations you’ve already had. I’d ask the agent: “Find the last 5 times I asked you to review a strategy doc and pull them into one file.” It needs the docs and tool results too, not just the final answers. If it can’t find the history, export it or paste it in. For a product, pull 5 logged customer sessions with whatever context the model had at the time. No need to dream up a test set from scratch. Hint: In Claude Code or Codex, start by asking where your sessions are saved. I’ll share more about inspecting these traces in a follow-up.

Step 3: Write the answer key. For each review, I want two short notes from me: what helped, and what didn’t. “Caught a success metric nobody had instrumented” would be a good one. I don’t need a review that just restates my doc. These examples and the criteria you approve are your golden set. The agent can draft the notes, but you still have to read the reviews and disagree with it where it’s wrong. This is where most of the work is.

Step 4: Run every model against the same prompt. Give each model the original task and docs, using the same tools and instructions. Don’t include your notes or the old review. That would be giving it the answer. During the lightning lesson, Eric sent one prompt to 5 models over Signal: Fable, Opus, Kimi K3, and two others. We could watch them respond next to each other. Parallel runs are convenient if you have that setup, but doing them one at a time works too.

Step 5: Look for confident wrongness. Eric’s example is zero sales with 44 ads live. A model can recommend killing all 44 and sound perfectly reasonable. Except we haven’t checked whether the sales tracking works. I’d want it to investigate that before recommending changes to the ads. This is the kind of miss that’s hard to see when you’re just reading a polished answer without the context.

A set of 5 is small, obviously. I’m using it to catch problems worth looking at more closely. If the models are close, I’d add examples and repeat the runs before picking one; a production rollout needs more testing than this.

The prompt you can copy

Here’s what I paste to get the whole thing running. Swap in your task and your models:

Look through my last 5 sessions where I asked you to review a strategy doc. If you can’t access them, ask me to export or paste them. Collect each original prompt and doc, plus the old review in a separate answer key. Ask me for a one-line good/bad note on each review, then draft pass/fail criteria for me to check. Replay each original task through [Model A], [Model B], and [Model C] with the same instructions, tools, and source material. Keep the answer key out of their inputs. Score the outputs against the approved criteria and show a table: model, cases passed out of 5, biggest miss, effort setting, time, and cost where available. Flag confident claims that contradict the source material or my notes. If you can’t run a model, prepare its test prompts and mark it as not run.

If you own a product rather than a coding agent, one swap: replace “my last 5 sessions” with “these 5 user sessions I’m pasting in,” and grade each candidate model’s response against your notes the same way.

Deciding what to use

Once the results are side by side, three questions do most of the work:

1. Does it beat what I’m using now, on my tasks?
2. Is it worth the switching cost? A new model means checking your prompts and finding new failure modes, and the cost varies from one release to the next.
3. Which one do I reach for when nobody’s making me choose? That’s revealed preference, and it’s more honest than any score.

The prompt you’re bringing over may need work. Anthropic removed over 80% of Claude Code’s system prompt for models like Opus 5 and Fable 5 with no measurable loss on its coding evals. It’s worth auditing the instructions you’ve accumulated for older models before carrying them all over.

Fable 5.1 is a subtler case. Anthropic’s prompting guide says existing Fable 5 prompts should work well, but still recommends testing effort levels again on your own evals: the same setting name doesn’t mean the same amount of thinking across models. It also uses less bold and fewer lists, so old anti-formatting rules can suppress structure the answer needs. Anthropic also calls out dense, ornate prose and offers a prompt for getting Claude to write more directly:

Fable 5.1 prompting rules

Readability is part of this too. Eric has his agents put the answer in the first sentence and keep it under a paragraph, then use a diagram if the subject needs more explanation. That’s a preference you can write down and test. I’d include it alongside correctness in the answer key, because having to ask for a shorter answer every time is work I’d rather the model saved me.

Something I’m personally working on is reducing switching costs. Models get rate-limited, repriced, and deprecated without much warning, so have a second favorite you’ve already tested before the day your primary disappears, and so that you can try “the latest and greatest” on your own use case.

Make it a habit, not a project

The test set gets more useful as I add to it. One of my nightly background tasks reviews that day’s agent usage and leaves the hardest examples for me to read in the morning. Those are the cases I want to try again when a new model comes out. There’s a ton of alpha in just inspecting your own usage, especially the exchanges where you had to correct the answer or ask the agent to try again.

I’ll be sharing more on this soon (how to analyze your own traces).

Try it this week

Here’s the whole thing in 5 steps you can finish before Friday:

1. Pick one task you did more than twice this week.
2. Ask your agent to pull your last 5 sessions doing it.
3. Write a one-line “good vs. bad” note on each.
4. Run your current model, your top challenger, and one you’re curious about through all 5.
5. Read the outputs yourself and pick the one you’d actually ship.

If another model ships while you’re reading this, you’ll have a way to answer the question that matters: is it better at my job?

If you try this, tell me which task you tested and what the models missed. I’d especially like to hear about a result that changed your mind.

Resources referenced

• Video recording of the lightning lesson
• Slides from the session

AI Product Playbook is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.