I trained a 21B MoE (3.36B active) from scratch on 70B tokens (emergent ability observations)

Due to budget constraints, training only went up to 70B tokens. With SuperBPE tokenization providing roughly +27% token efficiency, this is effectively equivalent to training on around 100B tokens. A stable checkpoint was also released so anyone can continue training from it. huggingface.co/YOON1v/Apex-3-59B-stable I trained Apex-3, a 20.99B-parameter MoE (3.36B active per token) from scratch on 70B tokens. No external pretrained weights. It uses 64 routed experts plus 1 shared expert (top-4), a SuperBPE tokenizer, and was trained with Muon and DiLoCo on GH200s. Model card: huggingface.co/YOON1v/Apex-3 Code and training notes: github.com/DW-dev-UE/LLM-from-scratch This post covers what showed up in the training logs, not the final SFT scores. Evidence comes from the A, B, and C runs in run_logs/apex3_20261026/ (icl, arith, val, and moe logs) plus the 70B base model benchmarks. Numbers below are from run A. B and C were nearly identical. Final arithmetic full accuracy was 0.665 for A, 0.655 for B, and 0.655 for C. In-context learning (induction head) — the only clear phase transition At 0.4B to 1.16B tokens the ICL score (later-token CE minus earlier-token CE) sat between -0.08 and -0.14. It then moved to -0.32 at 1.27B, -0.55 at 1.38B, and -0.69 at 1.49B. After 1.7B it stagnated between -0.69 and -0.76. It grew about 5x in roughly 0.33B tokens (about 150 steps). In the same window the training CE drop rate sped up from -0.0048 to -0.0066 per step and the loss curve bent. This is about 0.1% of the original 1.1T schedule. The later drift back toward -0.62 happened because earlier-token prediction improved and the gap narrowed, not because the capability regressed. Arithmetic Exact match looks like a sudden jump, but the underlying correct-token probability rises continuously. For add3 the correct probability went from 0.003 at 15B to 0.021 at 22B, then 0.080 at 30B, then 0.224 at 40B. When the probability crosses the decision threshold, exact match jumps. This is mostly a metric artifact. Decay from 60B to 70B produced a larger lift. From 49B to 59B, add3 probability stayed around 0.28 to 0.29 and add4 stayed around 0.11. After decay they moved to 0.60 and 0.38. Exact match overall went from 0.52 to 0.67, add3 from 0.44 to 0.78, and chain3 (a+b+c) from 0.13 to 0.38. Domain-specific language modeling (validation CE) Every domain decreased smoothly with no phase transition. Textbook went from 2.77 to 1.75, math from 2.76 to 2.07, qa from 2.54 to 1.92, and web from 4.22 to 3.35. After about 40B the curves were mostly flat. During decay the data mix changed: instruct improved from 1.09 to 0.86 while web worsened from 3.35 to 3.49 and books from 2.74 to 3.04. There was no clear grokking signal. Code validation CE stayed higher, which looks like a distribution mismatch. MoE load balancing Early on (run B) the maximum expert load in a layer reached up to 12.8 times the average. By the end of 70B it had settled to 1.07 to 1.38 times the average — essentially even. Final base-model benchmarks (70B Apex-3-Base, greedy pass@1) HumanEval+ was 19.5%, MBPP 38.9%, MBPP+ 32.3%, and GSM8K 8-shot 19.9% strict / 21.5% flexible. These were only measured at the end, so I do not know when the abilities actually appeared. Caveats Each arithmetic task used 64 problems, so exact match standard error can reach about plus or minus 6 percentage points. Single-point noise is meaningless, so numbers are 3-point averages. Arithmetic scores are teacher-forced (correct answer already in context), not free generation. Nothing was measured before 8.15B. Bottom line: the only clean phase transition I observed is the ICL / induction-head jump around 1.3B to 1.5B tokens. Arithmetic looks abrupt in exact match but is continuous in probability. Everything else was gradual. Happy to answer questions about the setup or logs.