I trained a 348M model trained from scratch on 22.7B tokens that does 14 digit arithmetic [P]

Hello this is my fifth small language model I've made and apart of my third series and it has been a lot of work but it payed off: 348M parameters, 22.7B tokens, then fine-tuned into a math model that solves arithmetic by showing the work — column addition with carries, borrow chains, partial-product multiplication — rather than guessing at an answer. Last time I posted a 326M model trained on 10B tokens. This has about 2.3× the data, and the math side is WAY better than my previous two math models. --- ## The benchmarks 99.4% average across the nine GPT-3 arithmetic sub-tasks, which does much better past even where I trained it. | Task | GPT-3 175B (few-shot, direct) | This model (348M) | |---|:--:|:--:| | 2-digit add | ~100% | 100% | | 3-digit add | 80.4% | 100% | | 4-digit add | 25.5% | 100% | | 5-digit add | 9.3% | 100% | | 2-digit sub | ~99% | 99.3% | | 3-digit sub | 94.2% | 98.3% | | 4-digit sub | 26.8% | 98.3% | | 5-digit sub | 9.9% | 99.0% | | 2-digit mult | 29.2% | 100% | n=300 per sub-task, greedy, exact match. GPT-3's numbers are direct-answer; mine uses trained-in worked steps. Neither uses a calculator. ## The cool part It adds cleanly up to 14 digits, and the reason it couldn't before was the vocabulary, not actually arithmetic. Training only ever named six place values (ones … hundred-thousands). The model learnt the pattern and invented two more on its own — millions and ten-millions appear in zero training examples — so it handled 7 and 8 digits fine. At 9 digits it ran out of names, and skipped the column, then returned an answer exactly one digit short: ` 483729164 + 519248637 ... ten-millions: 8 + 1 + 1 (carry) = 10, write 0 carry 1. Final carry: write 1. The answer is 102977801 ← eight columns for a nine-digit problem Every column it computed was perfect. One was never enumerated. Extending the place-name list from 6 entries to 19 moved the clean ceiling from 8 digits to 14: | Width | 6 | 7 | 8 | 9 | 10 | 12 | 14 | 16 | 18 | |---|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:| | before | 100 | 100 | 100 | 0 | 0 | 0 | 0 | 0 | 0 | | after | 100 | 100 | 100 | 100 | 95 | 100 | 90 | 65 | 25 | A six-item list became a nineteen-item list. That was the entire fix. ## Other things it does - 3×3 multiplication: 98% — it folds partial products pairwise through the column routine instead of asserting the sum - Negative results: 85% (100% at 1 digit, 58% at 5 — the magnitude comparison is the weak step, not the arithmetic) - Reasoning traces are load-bearing: 95.3% of the time the working is valid and the answer is right; only 0.7% are "valid working, wrong answer." If the columns look right, the answer almost certainly is. There were 15000 votes and 6842 were rejected. Here's how many counted: Start with 15000. Then subtract 6842. Subtract 15000 - 6842 column by column: ones: 10 - 2 = 8, borrow 1. tens: 9 (after borrow) - 4 = 5, borrow 1. hundreds: 9 (after borrow) - 8 = 1, borrow 1. thousands: 14 (after borrow) - 6 = 8, borrow 1. ten-thousands: 0 (after borrow) - 0 = 0. So 15000 - 6842 = 8158. The answer is 8158. ## What it's bad at, tbh - Word problems: GSM8K 4%. Best word-problem set is ASDiv at 16.5%. It converts one sentence into one operation reasonably often and basically cannot chain operations. - The failure mode is operation selection, not arithmetic. "drops in 836 more" gets read as subtraction. There's a visible tell: traces that say "multiply X Y" and show columns are reliable; traces that open "First, calculate…"` and assert a number in prose are not. - No division at all. 4×4 multiplication is a hard wall. - Greedy decoding required — sampling corrupts the column routine mid-chain. - One caveat I'll flag myself: the arithmetic harness orders subtraction operands, so no answer in that table is negative. Negatives are measured separately (the 85% above) rather than folded into the average. ## Base and instruct The math model sits on a base and instruct pair. lm-eval-harness, 0-shot, full test sets: | | ARC-E | ARC-C | HellaSwag | OpenBookQA | PIQA | WinoGrande | MMLU | Avg | |---|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:| | 350M V3 base | 56.6 | 33.3 | 35.9 | 34.6 | 67.0 | 51.5 | 23.8 | 43.2 | | 350M V3 instruct | 50.9 | 29.7 | 35.9 | 33.8 | 66.6 | 51.2 | 24.4 | 41.8 | Instruction tuning lowers MC benchmark scores for this family which is a pretty common cost of instruct tuning. ## Notes Trained on 2× Tesla V100 plus some rented time. LLaMA-architecture, so it runs anywhere — F16 GGUF and safetensors for all three. The math model took 10 full fine-tuning rounds and 3 LoRA adapters. The LoRAs cost about 1% of the post-training tokens and did most of the useful work; the ten full rounds spent most of their budget undoing each other's regressions (round 7 gained subtraction and lost 23 points of 2-digit addition, that sort of thing). All of it is documented on the model card, failures included. Also worth saying: someone independently tested it after I published and found two of my numbers were wrong — one understated* the model by 40 points because I'd measured it at n=24. Both are corrected on the card now. If you find something broken, I'd genuinely like to know. Math: huggingface.co/nkthebass/tinybrainbot-350mV3-math Instruct: huggingface.co/nkthebass/tinybrainbot-350mV3-instruct Base: huggingface.co/nkthebass/tinybrainbot-350mV3-base LMK what yall think.