Why agents that pass tests fail in production

Zero-Shot Learning is a podcast about how AI gets built, secured, and deployed. Hosted by Nancy Wang, 1Password CTO, and Dev Tagare, Senior Director of Engineering at Google, it’s a builder’s view of the architecture and the complex decisions it takes to ship with AI. Adarsh Hiremath, Co-founder and CEO of Mercor, joined Zero-Shot Learning to talk about why we need to rethink how we measure agentic performance and the infrastructure we can build to get there. Mercor is an AI-powered hiring platform that organizes human expertise to train AI. Their talent assessment engine connects many of the leading AI labs and frontier models with specialized experts who evaluate and train the next generation of LLMs and autonomous agents. Out of that work came the APEX benchmarks, an evaluation suite that measures whether frontier AI models and agents can perform economically valuable work. In this conversation, Adarsh shares how the framework Mercor has built can help CTOs answer whether their agents actually do what they're supposed to. What do people get wrong about testing agents? "I think there are a lot of enterprise teams that move to production without fully thinking through whether the agent is calibrated to the specific use case," Adarsh said. He pointed out that many teams measure an agent's success by whether it delivers the right result, not by how it gets there. "You could have a model that just answers correctly the first time, but it's making all the wrong decisions along the way," Adarsh said. "Then when you adapt it to a slightly different context, all of a sudden you've got an agent that's totally broken in production." Testing is also complicated by AI’s tendency to “cheat” on tests rather than arriving at the correct answer on its own. In February 2026, OpenAI stopped reporting SWE-bench Verified scores after finding that all frontier models could reproduce their benchmarking test answers verbatim. The test was sourced from open-source training…