How I use fine-tuning, quantization and speculative decoding to speed up a voice bot

My voice bot is not allowed to pause. One second of silence on a call and the caller talks over it, or just hangs up. In a chat a slow model hides behind a “typing…” dot. In a voice call it cannot.

The client wanted the whole thing on their own modest hardware: no cloud, no data-center cards, no per-token bill leaving the building. So the smartest model available had to run on a single consumer gaming GPU and still answer in real time. That constraint shaped every choice below.

A custom 12B on a gaming GPU

I picked Gemma 4 12B, the accurate one, and fine-tuned it for the task with QLoRA, then quantized it to run in llama.cpp.

• QLoRA trains a small add-on layer instead of all 12 billion weights. The adapter came out to 131 MB, not a fresh 24 GB model, so the whole fine-tune fits on one 16 GB gaming card.
• Quantization stores each weight in 4 bits instead of 16, so the model gets much smaller and faster with barely any quality loss. The unquantized 12B is 24 GB and won’t even load on a 16 GB card; the 4-bit version fits with room to spare.

Even so, the 12B answers at about 34 tokens per second on that GPU. Smart, but too slow for a live call.

Making it fast enough for voice

A big model is smart but slow: it writes one word at a time. The fix is to pair it with a much smaller, faster model that runs ahead and drafts the next few words. The big one only reads that draft and approves or rejects it, far quicker than writing every word itself. The technical name is speculative decoding, and the small one is the “draft” model.

The first attempt barely moved the needle: an off-the-shelf small model as the drafter gave about 3% faster, because it kept proposing words the big model threw away. An assistant you correct on every line saves nothing.

The fix was the whole point: I fine-tuned the draft on the same data as the big model. Now it guesses the way the big one thinks, so the big one accepts most of its drafts. Speed jumped from a rounding error to 1.7x on average (34 to 59 tokens per second), and 2.2x on the best prompt. The answers stay identical to the 12B alone, so the quality cost is zero.

In product terms, that jump is the difference between a bot that stalls and one that answers like a person, all on the client’s own cheap hardware.

The lesson

A helper model only speeds you up if it thinks like the expert it is helping. Train it on the same data, and a tiny model beats a generic one of the same size, by a mile.