Lure-TTS and Lure-Turbo: 600M voice models to run 4x faster than real time, with inline (tags), direction, and voice cloning. Trained for under $200.

I trained Lure TTS by finetuning Qwen3-TTS using some of the latest developments in model training, such as RLVR, OPD, and multi-teacher OPD. My goal was to create a faster, smaller TTS model that I could listen to as a voice assistant and not get angry at it. Mostly, I just wanted a faster model that sounded good so I didn't have to whip out a big gpu when I needed it. The first thing I did was use labeled data by training classifiers to spot intensity, events (laughs, cries, sighs, gulps, etc.), emotion and direction, to clean out YODAS and the Open Yap sample. Then I actually did teh SFT stage to instill the new tagging into the Qwen 3 TTS 0.6B because it just read the tags out loud. My next plan was multi-teacher OPD, where each skill (voice cloning, understanding directions, text tags, etc.) was trained separately and a final on-policy distillation pulled all the teachers back into the model. I initially trained teachers for robustness and speaker similarity. But I ended up liking the speaker similarity one while trying to combine the teachers. After that I really wanted speed. I started building a 300M model with 12 layers (IT SOUNDED OKAYISH JUST TINN-N-Y). But then I realized I could get faster times because of the 15-pass code predictor by training that down too and turning it into 3 passes, so this model can run 4x faster than real time speech. I used OPD from my current checkpoint to train the Turbo, and that was much faster. ^ this btw runs RTF of 0.14 vs 0.195 for the 5B Fish S2 Pro, while keeping the same speaker similarity (ECAPA 0.699). All the training was able to happen under $200 and didn't use a GPU higher than an RTX PRO 6000. (H100s for RTF testing). I'm working on making more specific tags like (laugh: big) and (whispering) work better on longer lines, emotion switches, and trying to get freeform paraphrases. AND just building a better tagger as well so I can redo the entire cycle. Also, I released an engine (also called Lure-TTS) that is built on top of faster-qwen3-tts (CUDA), mlx-audio (Mac) and audio.cpp (CPU). Compared with running the qwen-tts reference code on the same 4090, it gets first audio out in 0.17 s instead of 10.6 s and runs about 5x faster overall (RTF 0.53 vs 2.54), because it streams 0.32 s chunks, uses CUDA graphs and batches up to 8 requests at once. It runs Lure, Lure Turbo and every Qwen3-TTS 12Hz checkpoint on a Mac, an NVIDIA GPU or any CPU behind one Python API, CLI and OpenAI-compatible server. (This was built because it enables voice direction for 0.6B model as well as best configurations for the Lure and Turbo model.) My training code: github.com/smirki/lure-train Lure TTS Engine: github.com/smirki/lure-tts TBH I'm still learning and practicing in this domain. If anyone wants to help judge outputs or play around with checkpoints reach out to me! (or if you have compute or research ideas!)