I fine-tuned an LLM to clean up transcriptions for my voice-to-text app

I built a voice-to-text app, WhisperNow . But there are a thousand and one WisprFlow clones out there, so I went further and fine-tuned my own LLMs for the cleanup step. They remove fillers, false starts, self-corrections, and formats lists, all locally. To finetune an LLM for this task, I first had to get data. The dataset The dataset has 66,557 rows. I started by combining public datasets (aawaaz, handy and Disfl-QA) and adding synthetic dictations, such as emails, meeting notes, recipes and dictated commands. gpt-6.1-sol wrote the cleaned version of every transcript, using the same prompt the app sends to cloud models. While training the models, I found some weak spots: long dictations, words the speech recognizer misheard, cut-off words ("interf interface") and correction words used literally ("scratch behind the dog's ears"). I generated more examples of each and retrained. The models I did full fine-tunes of three base models, each on a single H100 or H200 rented on Modal: model memory in use 100-word dictation, RTX 5070 GPU 100-word dictation, Ryzen 9 8945HS CPU LFM2.5 1.2B (the app's default) 1.6 GB 0.18 s 2.3 s LFM2.5 2.6B 3.3 GB 0.34 s 5.4 s Qwen3.5 4B 5.3 GB 0.62 s 8.9 s Through their APIs, Claude Haiku 5.5 and gpt-6-luna take about 1.4 s on the same dictation, glm-5.3-flash 2.6 s, and glm-5.3 at max reasoning 5.6 s. To score the models, a judge model compares each output with the reference and counts factual errors: facts dropped, changed, kept from the abandoned side of a correction, or invented. The test rows are a held-out test split plus two benchmarks, one of long dictations and one of short dictations full of disfluencies (4,170 rows in all). The models do best on short dictations: Up to 25 words , all three make fewer factual errors than glm-5.3-flash at low reasoning. The 2.6B and 4B also make fewer than glm-5.3 at max reasoning, and the 1.2B ties it. The 4B makes fewer errors than glm-5.3-flash at any length, and keeps up with glm-5.3 at max reasoning up to 100 words. The 1.2B and 2.6B make 1.0–1.5 times glm-5.3-flash's errors from 26 to 200 words, and 1.5–1.9 times above 200 words. Chart: factual errors by transcript length, for the three fine-tuned models and the two cloud glm-5.3 models Cloud models I also scored seven more cloud models, including the off-the-shelf LFM2.5 2.6B, on a smaller set of 1,670 rows. The off-the-shelf 2.6B makes nearly four times the factual errors of the fine-tuned one. If you'd rather use a cloud model, Claude Haiku 5.5 and gpt-6-luna (both at low reasoning) have the best price-to-performance ratio. glm-5.3-flash is about as accurate as Haiku and glm-5.3 at max reasoning is more accurate, but they're about two and four times as slow. Chart: factual errors for every model I tested Links App (MIT, Windows, macOS and Linux): github.com/stephan271c/WhisperNow Models and dataset: huggingface.co/collections/category271/whispernow-6ae00accee969c7447cec326 How the dataset was made: github.com/stephan271c/WhisperNow/blob/master/training/making-the-dataset.md Happy to answer questions about the data or the training.