How to pick the best AI model for your agent

Every few weeks there is a new model announced and we have the AI champs of the world going gaga about how this one is a game changer. As a product manager managing multiple agents, each of these announcements creates an additional overhead and backlog item on pipeline evaluation with new models. And you know thats its not as easy as a simple model update, the instructions and other model facing configs around it also have to be evaluated and updated accordingly. And while you are excited about the capabilities a new model could unlock, for platforms with defined scope there is limited upside to model update. Or so did I think.

This piece is my attempt to give the model selection judgment a structure. Its the way how I now go about reading benchmark research and translating it into a decision that is specific to my agentic platform.

The Opaqueness Problem

Every major frontier lab, Anthropic, OpenAI, Google, Meta, publishes benchmark results when they release a model. The numbers are always impressive. The model always leads on something. And the benchmarks they choose to highlight are almost always the ones where they look best.
This is not dishonesty. It is marketing. But it creates a problem for builders. If every lab is selecting its own scorecard, you cannot compare across labs on their terms. You need a neutral one.

The closest thing I found based on my research is independent benchmark by Artificial Analysis. They run the same evaluations on every model, under identical conditions, and publish the results continuously. They do not have a model to sell. That independence is why most serious practitioners treat their leaderboard as the reference rather than any lab’s own release notes. Having said that, their category weights and which evaluations count are still authored choices. What they offer is independence from any single lab’s incentive to look good.

Hard Constraints First

Before we deep dive into the framework, there is a prior question that eliminates certail models entirely. Three constraints can make a model unsuitable regardless of where it sits on any benchmark.

The first is modality. Most agent design defaults to text in, text out. But if your agent needs to process a user uploading a screenshot, a scanned document, a voice command, or a video, your model must support that input format natively. Not all frontier models do. And among those that claim multimodal capability, performance varies significantly by modality. A model that handles images well may handle audio poorly. If your agent’s input surface includes anything beyond text, modality support is a hard filter that runs before any tradeoff analysis.

The second is data residency. If your product handles data that cannot leave your infrastructure, regulated industries, enterprise contracts, compliance requirements, you cannot call a third-party API regardless of how capable the model is. Open weights models, whose parameters are publicly released and can be self-hosted, exist specifically to solve this.

The third decision that often gets made too late is model size. Large language models (LLMs) are the frontier models most benchmark discussions center on. They have the highest capability ceilings, handle complex multi-step reasoning, and are what most agent design defaults to. Small language models (SLMs) are models typically under 15 billion parameters. They run faster, cost less, and can be self-hosted on modest infrastructure. What they trade away is reasoning depth and the ability to handle tasks that require holding a lot of context or decomposing genuinely complex goals.

Four Dimensions of Model Evaluation

Once we have worked through the hard constraints, model performance thereafter can be broken down into four dimensions that move independently. A model can lead on one and sit in the middle of the pack on another. The four dimensions are intelligence, cost, latency, and hallucination. Getting the right model for your agent means knowing which of these four your task is actually sensitive to and in what order.

Intelligence
Intelligence is the dimension that gets the most attention and is the most misunderstood. Artificial Analysis publishes an Intelligence Index, a composite score built from nine evaluations across four capability categories. The categories and their weights reflect a deliberate judgment about what actually matters in production:

The 34% weight on agents is intentional and important. It reflects the judgment that in 2026, the most economically valuable thing a model can do is complete multi-step work autonomously. If you are building agents, this category is the one most relevant to your use case.

This composite score is a starting point. A model that ranks third overall might rank first on the agentic subcategory specifically. For agent design, that subcategory ranking is the more useful number. Morever, I would also like to highlight that this score does not measure how well a model handles your specific tasks, your specific tools, or your specific user base. It also does not measure dialogue quality, calibrated uncertainty, or the ability to know when to ask a clarifying question rather than proceed. Those gaps only surface once you run your own evals

Cost
Cost in model selection is not the same as price per token. Price per token is what a provider charges for input and output. Cost per task is what it actually costs to complete one unit of real work, accounting for how many tokens that work requires, including any reasoning tokens the model generates internally before producing an answer.

Reasoning models, the ones that think before they respond, generate significant internal token overhead. A model with a lower list price can be meaningfully more expensive per task if it burns three times the tokens getting there.

Artificial Analysis calculates cost per task by running models through their full evaluation suite and measuring actual token consumption. While this may not be same as your true actual cost, it is a better proxy than price per token.

The other cost variable is caching. When your agent repeatedly processes similar context, system prompts, tool descriptions, user history, providers charge a fraction of the standard input price for cached tokens. A well-architected agent with high cache hit rates can run at a fraction of the apparent cost. Cache hit rate is a design decision, not a pricing decision.

Latency
Latency has two distinct components that builders frequently conflate:

Time to first token, how long from sending the request to receiving the first word back. This is the number that determines perceived responsiveness. For any agent interaction that is user-facing, this is the latency that shapes UX.

Time to first answer token, for reasoning models, the model thinks before it answers. The first token it emits is a reasoning token, not an answer. Time to first answer token measures the gap between sending the request and receiving actual output the user sees. On frontier reasoning models at maximum effort, this can be measured in minutes, not seconds.

Output speed, tokens per second once generation starts. This matters for long-form outputs, detailed plans, structured documents, long tool call sequences.

The relevant metric depends entirely on where in your architecture the agent sits. A user-facing planner agent needs low time to first token. A background feedback agent that runs post-execution has no meaningful latency constraint. Using the same model for both because it is the most capable is a design decision that will cost you money without improving the experience.

Hallucination
Hallucination is the dimension that gets treated as binary, either the model hallucinates or it does not, when it is actually a rate with meaningful variation across models.

Artificial Analysis measures this through AA-Omniscience, a 6,000-question knowledge benchmark with a specific scoring structure: correct answers are rewarded, confident wrong answers are penalized, and refusing to answer carries no penalty. This scoring structure means the benchmark is measuring calibrated confidence along with accuracy. A model that says “I don’t know” when it doesn’t know scores better than one that guesses confidently and guesses wrong.

The practical implication is that hallucination rate does not always track with general intelligence score. Some mid-tier models are significantly better calibrated than frontier ones. A model ranking fifth on the intelligence index might rank first on groundedness.

For agent design, hallucination risk is not uniform across dimensions. A model generating a structured JSON output from a validated plan is in a low hallucination risk situation, the inputs are constrained and the output format is defined. A model interpreting an ambiguous user request and deciding what to ask next is in a high hallucination risk situation, it can confidently invent context that was never there.

The Tradeoff

There is no model that leads on all four dimensions simultaneously. The question is not which model is best, it is which model is best for the specific sensitivity profile of the agent you are building.

Here is a practical way to work through that for any agent:

1. Identify your failure cost. What happens when this agent gets it wrong? If a wrong output breaks a downstream workflow silently, failure cost is high. If a wrong output is immediately visible to a user who can correct it, failure cost is lower. High failure cost raises the weight you put on intelligence and hallucination. Low failure cost lets you trade those for cost and speed.

2. Identify your latency surface. Is this agent in a user-facing interaction or running in the background? User-facing means time to first token is a real constraint. Background means it is not. This determines how much of your budget goes toward speed.

3. Identify your volume. How many times does this agent run per day? Low volume means cost per task is not the binding constraint. High volume means even a small cost difference per task compounds into a significant number at scale.

4. Rank your four dimensions. Given the above, order intelligence, cost, latency, and hallucination by how much they matter for this specific agent. That ranked order is your selection filter.

5. Shortlist, then eval. Use the ranked filter on the Artificial Analysis leaderboard to get to two or three candidate models. Then run those candidates on twenty to thirty real examples from your actual use case. The benchmark tells you which models are worth testing. Your eval tells you which one to use.

Where This Breaks Down

Two situations where this framework could potentially breakdown.

1. When your agent spans multiple dimensions with conflicting requirements. A single agent that is both user-facing and doing complex agentic reasoning will pull toward low latency and high intelligence simultaneously, which often point to different models. The answer is usually to split the interaction: a fast model handles the immediate response, a capable model handles the reasoning in the background, and the results are stitched together before the user sees them. This is an architecture decision that benchmark reading alone will not surface.

2. When the model changes under you. Providers silently update models. The same model endpoint can perform differently across providers. An agent that passed evals on day one can degrade without a deployment change. Contonuos monitoring and evals is the only solution to this

The Practical Cadence

You do not need to re-evaluate models every week. The field moves fast but not that fast. A reasonable cadence is

Weekly: Scan the Artificial Analysis change log. Two minutes. Look for whether any new model was added that is relevant to your use case. If the top three on the agentic index changed, flag it for evaluation. Their recommender reflects this structure directly. Step one sets your priority sliders across intelligence, speed, and cost. Step two asks for hard requirements, open weights, reasoning vs non-reasoning, modality, context length. The tool is well designed. But the sliders only work if you already know how to set them.

On major model releases: Run your golden eval set, your twenty to thirty real examples, against the new model. Compare to your current model on your ranked dimensions. Switch if it wins on what matters to you, not on the composite score.

On pricing changes: Recalculate cost per task for the models you are using. A pricing shift that changes the cost ranking is worth a re-evaluation.

Quarterly: Review each agent’s dimension priority ranking. As your product evolves, the agents themselves change. An agent that was low volume six months ago may be high volume now, which changes how much cost matters in the selection.

Conclusion
Benchmark research is a filter, not a decision. It narrows the field from five hundred models to three or four that are worth testing. Your own evaluation on your own tasks makes the actual call. The intelligence index tells you which models can, in principle, do the kind of work your agent needs to do. The cost per task tells you what that work will cost at your volume. The latency metrics tell you whether the model can operate within your UX constraints. The hallucination rate tells you how much of your own validation layer you will need to compensate for the model’s confidence. None of those four numbers alone is the answer. Together, weighted by what your specific agent is sensitive to, they get you close enough to run a real test which is your evals. And the real test is the answer.

References

Artificial Analysis Intelligence Index — https://artificialanalysis.ai/methodology/intelligence-benchmarking

Artificial Analysis Endpoint Accuracy Index — https://artificialanalysis.ai/articles/endpoint-accuracy-index

The Six Dimensions That Make an AI Agent Intelligent — link to your first article

Anthropic, Building Effective Agents — https://www.anthropic.com/engineering/building-effective-agents

How to pick the best AI model for your agent was originally published in Bootcamp on Medium, where people are continuing the conversation by highlighting and responding to this story.