Loss Exploded.

If you want to see what a very painful couple of months looks like for an ML research team, FAIR’s logbook of the OPT-175 pretraining from 2021 should top your list. The first few runs are basically: At each point the team tweaks some of the hyperparameters: learning rates, weight decay, clipping and so on, […]