Understanding how GIL Affects Checkpoint Performance in PyTorch Training

I have been spending time learning about model training infrastructure lately, and something that stood out to me was GIL contention when saving training checkpoints in PyTorch. Having spent years in the Ruby world dealing with the GVL (Global VM Lock, Ruby’s equivalent), I was naturally drawn to it. The symptoms are familiar, like - you spin up background threads expecting parallelism, and instead everything gets slower.
So, as one does, I decided to go down some rabbit holes as an opportunity to learn more about what the GIL actually is, why it makes saving checkpoints in the background counterproductive when done with threads, what the fix looks like, and how I measured all of it on an H100 (rented for a few hours) with a real Llama model to get a better mental model.