Why do OpenAI's GPT-2 weights beat mine? Part two: the bugfix

I'm digging into why my GPT-2 style models score worse on an instruction-following eval than OpenAI's original weights; I gave the details in this post.

While I was writing up the results of my first experiment into possible causes, I ran the post past ChatGPT -- I always use an "editorial board" of AIs to check my posts for flow, style, and any technical errors (though all writing is always mine). It took a look at the eval code that I was running, and highlighted a bug.

Luckily, it doesn't change the important results -- OpenAI's models continue to be better than mine at instruction-following. But it was enough to change the baseline numbers, re-ordering how well my own models did. So I fixed it and regenerated the baseline so that future experiments are based on solid ground.

The bug

The eval takes a model, and trains it over multiple epochs on a split of a subset of the Alpaca instruction-following dataset. At the end of each epoch, it evaluates the resulting model against a held-back validation split; if the eval loss starts rising, it bails out. Finally, it runs a test split of the dataset through the resulting model, and saves the result.

Once I've run it for a bunch of different models, I use an LLM-as-a-judge script to get GPT 5.5 to score results -- for each question-answering result, it sees all of the responses for all of the models in the same prompt, shuffled in order each time, to try to make it judge models against each other as consistently as possible.

Now, the idea was that the generation of the test split answers would use the model from the epoch prior to the rising-loss one. So I had code like this:

    for epoch in range(100):
        model.train()
        for input_batch, target_batch in tqdm(train_loader, desc=f"Epoch {epoch}"):
            optimizer.zero_grad()
            loss = calc_loss_batch(
                input_batch, target_batch, model, device
            )
            loss.backward()
            optimizer.step()…