Impact of modelling assumptions on time horizon results

As METR’s time horizon task suite saturates, the results are becoming more sensitive to analysis choices. One example of this was the recent update to fix a modelling mistake with regularization, which decreased recent models’ 50% time horizon results by up to 20%, but had a smaller impact on earlier LLMs’ 50% time horizons. 1 In this post I’ll: Give a refresher on the current model used to calculate time horizon results and more detail about the regularization mistake METR recently fixed Go over what I see