Metrics of Agent Ability

Figure sources: scripts/tikz/2026-07-24-metrics-of-model-ability-*.tex Build with scripts/tikz/build-metrics-of-model-ability.sh.
• *Goals of this post > > Goal:** a simple way to compare a variety of capability metrics in an idealized setting where we observe only a function from $x$ to $s$. > > Non-goals: > > 1. Treatment of practical difficulties (an incomplete list of other considerations appears at the bottom). > 2. Recommendations about which metric to use. > 3. Representation of a consensus METR view. {:.metrics-callout}

This post reviews alternative metrics of agent capability.

This post contains a fairly comprehensive taxonomy of capability metrics for the situation in which you observe score curves for an agent and a human.
Every metric is based off the following two scor…