Agentics: agent evals considered harmful

My hot take is that 90% of the people writing evals need to stop. We’ve created a cottage industry of people creating evals and people selling eval tools to the people creating evals and virtually no one has any idea what they are doing.* In fact, all the evals people run are almost certainly making the agents dumber.

In the agent/AI tooling world, an eval is meant to be a measure of capability. It’s basically interchangeable with “test” or “benchmark.” The point of an eval is to get signal on fuzzy, hard to measure things. How do you know if an AI system can recognize handwriting? Well, you create a dataset of handwritten text where you know what the text says, and then you compare your answers against the AI’s. In theory, if the AI matches your answers, you could make a claim that the AI is probably good at recognizing handwriting. Congrats, you have an eval.

But note the “probably.” The only thing an eval measures is whether an AI is good at that eval. You can only really make the other, stronger claim through correlation. Well, correlation ain’t causation, and writing good evals is hard. Are you sure the agent knows how to recognize the letter “w”? Or is it cheesing your test in some way? Maybe all of your handwriting was done on grid paper, and it will totally fail on plain sheets. Maybe you only wrote freehand and now the agent won’t pick up script. Maybe it doesn’t speak Spanish.

As the AI tools have gotten better, writing good evals has become exactly as hard as designing tests for people, whether those are physics midterms or SWE interview pipelines. First, you have to make sure that doing well on the test actually measures something useful. What’s useful? Who knows, totally subjective, depends on the problem. Second, you have to make sure that a motivated adversary can’t beat your benchmark. Turns out the SAT was basically useless during COVID because it was trivial for kids to just have the answers when they were taking the test at home.

The only thing worse than no eval is a bad eval. If you have a test that you think measures competence but actually measures nothing, you will lock in practices that optimize for the wrong thing.

Here’s something that I see a lot that was the inspiration for this post: in the era of vibe-everything, folks are creating vibe-evals (of course they are creating vibe-evals). They will vibe-code a skill or harness or set of tool calls that they never look at themselves, and then they will vibe-code a set of tests that they never look at that supposedly locks in the behavior of that skill/harness/tool call. Most of these end up being poorly written slop files that simply restate the knowledge the agent already knows. When a new model generation comes out, your “tooling” will effectively kick that model into behaving like a previous generation model, because the skill only contains information that the previous gen model knows. And the vibe-evals hold that behavior in place, or at least mask any problem.

We are creatures of comfort. We hate risk. We can’t stand the idea of accidentally making things worse; most of us would rather do nothing at all. The eval acts as a security blanket. “Keeping this explanation of how AWS works in the skill file makes the eval pass” makes us feel like we are being rigorous and data driven. The problem is that the eval may literally say something like “the skill file must contain an explanation of how AWS works.” The agents already know how AWS works, anything you’re adding in your skill will be strictly net negative.

All of this to say, evals take time to create and maintain and use properly. Unless you can justify having a group of people whose full-time job is to track and maintain evals — i.e. you’re a large well funded team — I think it’s better to go off the actual feeling of using the tools. And if you insist on using evals to track the quality of extremely fuzzy unscientific changes to agent configuration, do yourself a favor and make sure you test the quality of the model without any skills or configs at all.

Editor’s Note: we are hosting Jesse Vincent, CEO of Prime Radiant and the creator of the popular superpowers set of skills, on Wednesday November 4th in NYC. We’ll be doing a fireside chat where we just shoot the shit about skills, agent identity, and effective use of AI tools. Sign up here to attend!

If you haven’t yet, check out our latest Agentics Post Production video! These are video versions of the Agentics blog, saved on the post production playlist.

If you want thoughts on how to write good evals, follow Jiro (our internal Nori agent) on Twitter or me on substack or join our Agentics community slack.

Agentics is the study of how to use and reason about agents. Learn more about how to use agents at our agent learning hub.

If you want AI employees, we can help! Nori is a platform to build and deploy AI employees. Check out noriagentic.com for more.

12 Grams of Carbon is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.