Why do OpenAI's GPT-2 weights beat mine? Part two: IFT dropout

Giles Blog，Giles Thomas 的技术博客，30 年技术经验分享，关于 AI、创业与软件开发。关注

Wait, dropout!

https://arxiv.org/pdf/2101.03961 -- p11 --

Regularizing large sparse models. Our paper considers the common NLP approach of pre-training on a large corpus followed by fine-tuning on smaller downstream tasks such as summarization or question answering. One issue that naturally arises is overfitting since many fine-tuning tasks have very few examples. During fine-tuning of standard Transformers, Raffel et al. (2019) use dropout (Srivastava et al., 2014) at each layer to prevent overfitting. Our Switch Transformers have significantly more parameters than the FLOP matched dense baseline, which can lead to more severe overfitting on these smaller downstream tasks.

Could that be it?! Model conf has dropout in it, so...

Drop rate was zero in the GPT-2 weights JSON files I had! Nothing in the paper but Raschka and also https://huggingface.co/docs/transformers/v4.48.2/en/model_doc/gpt2#transformers.GPT2Config

When I did the extended/two-epoch runs, I wound up doing many more epochs for the IFT.

Matching pre-training

OK, firstly let's do a run where we match the models' pre-train dropout.

Test loss Base dropout IFT epochs IFT score IFT rank
OpenAI weights: medium 3.231442 Yes 2 42.43 1
JAX, overtrained one long epoch 3.324953 No 3 19.35 4
JAX, overtrained two normal epochs 3.326482 No 4 18.68 7
JAX, with MHA bias, no dropout 3.418784 No 4 18.96 6
JAX, no MHA bias, no dropout 3.420089 No 5 21.45 3
JAX, no MHA bias, with dropout 3.476802 Yes 7 17.17 10
OpenAI weights: small 3.499677 Yes 4 23.66 2
1xrtx3090-stacked-interventions 3.538161 No 4 14.06 13
8xa100m40-stacked-interventions-1 3.577761 No 4 10.51 17
Cloud FineWeb, 8x A100 40 GiB 3.673623 Yes 6 19.25 5
1xrtx3090-baseline 3.683835 Yes 6 13.37 14
8xa100m40-baseline 3.691526 Yes 4 14.25 12
Cloud FineWeb, 8x H100 80 GiB 3.724507 Yes 5 14.31 11
Cloud FineWeb, 8x A100 80 GiB 3.729900 Yes 4 11.48 16
Cloud FineWeb, 8x B200 160 GiB 3.771478 Yes 4 11.50 15
Local FineWeb train 3.943522 Yes 7 9.09 18
Local FineWeb-Edu extended train 4.134991 Yes 7 17.70 8
Local FineWeb-Edu train 4.166892 Yes 7 17.55 9

Epochs in keeping with previous ones except for OpenAI small which took 4 rather than 2, and oddly "Cloud FineWeb, 8x H100 80 GiB" which took an extra epoch.

The ordering has swapped around a bit, but nothing drastic. All models are scoring a bit higher than they were, apart from OpenAI medium and small. So maybe signal? Small is still way ahead though.

Dropout always on

Test loss Base dropout IFT epochs IFT score IFT rank
OpenAI weights: medium 3.231442 Yes 2 42.22 1
JAX, overtrained one long epoch 3.324953 No 19 7.26 17
JAX, overtrained two normal epochs 3.326482 No 16 12.71 13
JAX, with MHA bias, no dropout 3.418784 No 13 13.09 11
JAX, no MHA bias, no dropout 3.420089 No 20 5.20 18
JAX, no MHA bias, with dropout 3.476802 Yes 7 18.00 4
OpenAI weights: small 3.499677 Yes 4 23.94 2
1xrtx3090-stacked-interventions 3.538161 No 13 15.28 7
8xa100m40-stacked-interventions-1 3.577761 No 19 8.32 16
Cloud FineWeb, 8x A100 40 GiB 3.673623 Yes 6 20.77 3
1xrtx3090-baseline 3.683835 Yes 6 14.16 10
8xa100m40-baseline 3.691526 Yes 4 15.17 8
Cloud FineWeb, 8x H100 80 GiB 3.724507 Yes 5 14.86 9
Cloud FineWeb, 8x A100 80 GiB 3.729900 Yes 4 12.35 14
Cloud FineWeb, 8x B200 160 GiB 3.771478 Yes 4 12.99 12
Local FineWeb train 3.943522 Yes 7 10.31 15
Local FineWeb-Edu extended train 4.134991 Yes 7 17.33 6
Local FineWeb-Edu train 4.166892 Yes 7 17.94 5

As you'd expect, epochs the same for the models base trained with dropout. Much higher for non-dropout base models.

Models base trained with dropout seem to have been largely unaffected, as you'd expect -- points go up or down but pretty much within a range of +/-1. But all models that were base trained without performed catastrophically worse, with the exception of 1xrtx3090-stacked-interventions, which actually went up a bit (though within the noise).

Interesting.

Dropout always off

Test loss Base dropout IFT epochs IFT score IFT rank
OpenAI weights: medium 3.231442 Yes 2 42.81 1
JAX, overtrained one long epoch 3.324953 No 3 18.62 5
JAX, overtrained two normal epochs 3.326482 No 4 18.94 4
JAX, with MHA bias, no dropout 3.418784 No 4 17.60 6
JAX, no MHA bias, no dropout 3.420089 No 5 20.60 3
JAX, no MHA bias, with dropout 3.476802 Yes 5 12.55 15
OpenAI weights: small 3.499677 Yes 2 25.86 2
1xrtx3090-stacked-interventions 3.538161 No 4 13.15 14
8xa100m40-stacked-interventions-1 3.577761 No 4 10.11 18
Cloud FineWeb, 8x A100 40 GiB 3.673623 Yes 3 16.78 7
1xrtx3090-baseline 3.683835 Yes 4 14.94 8
8xa100m40-baseline 3.691526 Yes 3 13.71 12
Cloud FineWeb, 8x H100 80 GiB 3.724507 Yes 4 13.86 11
Cloud FineWeb, 8x A100 80 GiB 3.729900 Yes 3 10.75 17
Cloud FineWeb, 8x B200 160 GiB 3.771478 Yes 4 13.60 13
Local FineWeb train 3.943522 Yes 5 12.03 16
Local FineWeb-Edu extended train 4.134991 Yes 5 14.19 10
Local FineWeb-Edu train 4.166892 Yes 5 14.75 9

Training epochs stable vs first run in this post for models base trained with no dropout, as you'd expect. Models trained with dropout all get fewer epochs apart from Cloud FineWeb, 8x B200 160 GiB, which is stable

OpenAI weights unharmed.

Other with-dropout models mostly harmed apart from 1xrtx3090-baseline, Local FineWeb train.

Need a better comparitive table.

添加评论
点赞收藏
点踩分享查看原文
评论
?
参与讨论