Leading-Edge AI Models Are Losing Their Allure With Developers

As a lifelong, overly enthusiastic San Francisco Giants fan, this was a rough summer. Essentially out of contention since week six, the Giants ended the season with a dubious distinction: one of the largest gaps between projected and actual wins of any MLB team in nearly 25 years.

Why is this relevant for our Pro subscribers? Well, a FanGraphs postmortem on the season’s collapse used a methodology that is easily forked to LLM tokens. Instead of measuring actual wins and against performance measures like runs scored or runs allowed, I used model quality ratings from Artificial Analysis and token use data from OpenRouter to examine how much incremental use more intelligent models earn.