A Record of the Process of Increasing MFU from 21% to 50% While Pre-training 21B MoE Using a Single GH200 Chip (Includes Code and Logs)

I am an independent researcher. Due to budget constraints, I shifted my 21B MoE pre-training—which I had originally planned to run on 64 H100s—to a single NVIDIA GH200 ($2.29 per hour), and I have compiled the metrics I measured during that process into a paper. preview.redd.it/77womlml91vh1.png I hope this will be helpful to those starting out with LLMs on a limited budget. These are the metrics I recorded at each stage while combining existing techniques. Setup Model: MoE with 20.99B total parameters and 3.36B active parameters (30 layers, 4 experts selected out of 64) The 238.5 GiB of FP32 training data was entirely stored in CPU memory, and the GPU’s optimizer kernel updated it directly across NVLink-C2C GPU power was capped at 700 W by the cloud provider Results Benchmark MFU: 21.1% → 50.7% (13 measurement points, one of which involved a model change) Real-data training MFU: 36.5% → 48.9–49.1% (10.9B tokens; two GH200s connected via DiLoCo starting from 4B tokens) Including synchronization time, we estimate this to be 47.6–47.8% There were also few failures Transformer Engine FP8, row-wise FP8: GEMM became faster, but the quantization kernel became even slower Method of offloading all activation values to the CPU; cumulative partitioning between GPU and CPU CPU addition issue where speeds varied between 22% and 32% per run despite identical settings (resolved using a C++ OpenMP kernel) Limitations Only one model and one type of hardware were tested The per-stage metrics are not a clean ablation test, as they mix code versions and measurement methods We were unable to verify the impact of FP8 on quality We only covered the first approximately 11B tokens of Stage 1 Code, benchmark records, and training logs: github.com/DW-dev-UE/LLM-from-scratch/tree/main/apex3 Paper: zenodo.org/records/23316135 Please let me know if you find any errors in the measurement methods or numerical analysis. I’m currently observing additional emergent capabilities while pre-training on 21B. Additionally, if the opportunity arises, I’d like to submit this to Arvix as well 🫠