Self-Hosted Voice Dictation with Whisper and Parakeet
I run speech recognition on my home server (a GMKtec NucBox with Intel Iris Xe graphics) for voice dictation. My phone runs Mouthful, my Android dictation app, and my weaker devices run Vocalinux. Both send their audio to mouthful-server, a small server in Mouthful’s repo that puts a speech engine behind whisper-server’s API: OpenAI’s Whisper or NVIDIA’s Parakeet. It applies my custom dictionary of jargon, can have an LLM clean up each transcript, and logs stats about each dictation. It all runs in Docker Compose on my Arch Linux server.
Engines
The server’s config chooses the engine and whether an LLM cleans up the transcript, so that I can switch without changing my clients. To compare the choices, I read two of my blog posts into both my laptop and my phone. Then I split each recording at pauses into chunks of up to 12 seconds, like Vocalinux does, and scored each engine:
| Setup (laptop / phone) | Jargon spelled right (of 21) | Word error rate on plain prose | Time per chunk |
|---|---|---|---|
| Whisper | 12 / 16 | 5.6% / 2.8% | 1.2 to 1.3 s |
| Whisper with cleanup | 12 / 16 | 5.1% / 2.2% | 2.8 to 3.1 s |
| Parakeet | 13 / 13 | 2.2% / 1.7% | 0.3 s |
| Parakeet with cleanup | 13 / 14 | 2.2% / 1.1% | 2.0 to 2.3 s |
| Parakeet with hotwords | 15 / 15 | 2.2% / 1.7% | 0.3 s |
Jargon counts in any capitalization, e.g. “zsh” for Zsh, since I care about the words more than their case. Over Wi-Fi, my phone’s round trip adds about 0.2 seconds to these times, about 40% of which is the upload.
Whisper (small.en-q8_0) can take my dictionary as a prompt, but in my tests that only fixed capitalization (see below). On my NucBox, it transcribes a 10-second utterance in about 1 second. medium.en-q8_0 took 2.6 seconds and large-v3-turbo-q8_0 took 3.1 seconds. Published benchmarks show that the larger models are more accurate on realistic speech (e.g. about 30% fewer errors on Common Voice for large-v2 vs small.en). But on my recordings, large-v3-turbo made only slightly fewer errors than small.en on plain prose and no fewer on jargon. Shorter audio doesn’t help, since Whisper always processes a full 30-second window. (The --audio-ctx option shrinks that window, but it destroyed accuracy in my tests.) The q8_0 models were also faster than their f16 versions on this GPU.
Parakeet (TDT 0.6B v3) is about four times as fast and makes fewer errors on plain prose, especially from my laptop’s noisy mic. But it can’t take a prompt, so on its own it gets the case of most of my jargon wrong, e.g. “ZSH”, and misspells some of it. It runs on the GPU through whisper.cpp’s libparakeet, which only comes with a command-line tool, so I wrap it in a small server. Earlier, I ran Parakeet TDT 0.6B v2 on the CPU through onnx-asr, which was slower and less accurate.
Hotwords make Parakeet spell my dictionary’s terms right without an LLM. Parakeet decodes one token at a time and picks the token with the highest score. I patched libparakeet so that my wrapper can change those scores before each step. The wrapper splits each term into Parakeet’s own tokens, and then it raises the score of each term’s first token by 1. Once the transcript ends with a term’s first few tokens, it raises the score of the term’s next token by 6. Stronger boosts for first tokens raised the error rate on my laptop’s plain prose from 2.2% to 2.8%. With only my real dictionary, Parakeet with hotwords spelled more jargon right than Parakeet with cleanup, and the boosts cost no time that I could measure. So it’s the default now.
Cleanup has Qwen3.5 4B fix misheard words with my dictionary as a glossary, and fix punctuation, capitalization, and filler words. It takes about 1.1 seconds on transcripts as short as my real dictations, which are shorter than the chunks above. Parakeet with cleanup made the fewest errors on plain prose, since the LLM also fixes ordinary words that aren’t in my dictionary. But it’s off by default, since hotwords spell more jargon right in a seventh of the time. If the LLM changes too much of a transcript, e.g. by answering a dictated question, the server keeps the original.
Since cleanup took most of the time, I tuned llama.cpp for it before I added hotwords. On 49 short transcripts, these changes cut its mean time from 1.58 to 1.16 seconds with the same accuracy:
- Speculative decoding drafts tokens by copying them from the transcript, since cleanup mostly repeats it, and the LLM checks each draft in one pass. Drafts of up to 7 tokens beat the 24 that I started with. llama.cpp’s Vulkan backend runs batches of up to 8 tokens with fast matrix-vector kernels, but a batch of 9 to 34 tokens costs as much as 3 to 4 single tokens.
- If there’s nothing to copy, Qwen3.5’s own multi-token prediction (MTP) head drafts 2 tokens. This needs Unsloth’s MTP version of the model.
- Qwen3.5’s recurrent layers can’t roll back to the end of the system prompt, which every cleanup shares. llama.cpp works around this with checkpoints, but they split every prompt into two batches. Instead,
mouthful-serversaves the LLM’s state after the system prompt once and restores it before each cleanup. This cut the time to read each transcript from 450 to 300 ms.
Whisper uses Silero VAD to skip audio without speech. Without it, Whisper often transcribes silence as “Thank you.” But VAD’s default padding of 30 ms also clips short words at the edges of speech. When my phone kept losing the first word of a dictation, I first suspected the app’s mic, but it starts recording within 60 ms of the tap. In 24 short clips that started just before a phrase, VAD dropped a first or last word (e.g. “and” or “or”) in 4. Padding of 200 ms fixed all of them, while 400 ms let Whisper hallucinate a lone “and” again.
Dictionary
Speech recognition often misspells jargon, e.g. “Vocalinox” for Vocalinux and “ripcrep” for ripgrep. mouthful-server uses my dictionary to fix this in four ways:
- It sends the dictionary’s terms to Whisper as a prompt, which makes Whisper more likely to capitalize them right. On the passage about Unix shells above, my dictionary raised the jargon that Whisper wrote with the exact capitalization from 3 to 7 terms on my laptop and from 5 to 8 on my phone, with almost no added latency. Ignoring capitalization, it made no difference (12 and 16 of 21 either way), so the prompt only fixes capitalization, e.g. “zsh” to “Zsh”. But Whisper only reads the last 224 tokens (roughly 700 characters) of a prompt.
- It sends the terms to Parakeet, which boosts them as hotwords (see above).
- It gives the terms to the cleanup LLM as a glossary.
- It replaces known mishearings in the transcript, e.g. “ripcrep” with “ripgrep”. This fixes words that the prompt can’t fix, works for both engines, and has no size limit.
(Vocalinux has its own custom dictionary, but its Remote API engine doesn’t send the terms to the server.)
Create ~/.config/mouthful/dictionary.txt. Each line is a term, optionally followed by a colon and comma-separated mishearings. Mishearings match case-insensitively and only as whole words, and a space also matches a hyphen. To fix a term’s capitalization, list the term itself as a mishearing. Put the most important terms first, since only they fit in Whisper’s prompt. The server rereads the file on every request, so edits apply immediately.
# Only list terms that speech recognition gets wrong, and only list mishearings that I never mean literally
Vocalinux: vocalinox, vocal linux
KeePassXC: keypass xc, keepass xc, key pass xc
Syncthing: syncthing, sync thing, singthing
tmux: teamux, teamox, team mox
ripgrep: ripcrep, rip crep
yq: ik
Setup
mouthful-server lives in the server folder of Mouthful’s repo. It needs Linux with Docker and an Intel or AMD GPU, although I’ve only tested Intel. For NVIDIA GPUs, edit compose.yaml and the Dockerfile there.
git clone https://github.com/artnc/mouthful.git
cd mouthful
make server
This downloads the four models (about 3.5 GB) at pinned versions, checks their hashes, and starts up to three containers:
engineruns either Whisper or Parakeet. Whisper’swhisper-servercomes from Arch’s packages, which I install from the Arch Linux Archive’s snapshot of a fixed date. On my Iris Xe, they took 1.0 seconds per dictation vs 1.45 seconds for whisper.cpp’s own Vulkan image, which has an older Mesa and no AVX-512. Parakeet runs on the GPU through whisper.cpp’slibparakeet, which I build from the source of the same release with my hotword patch.llamaruns llama.cpp’s Vulkan image with Qwen3.5 4B for cleanup. It runs only if cleanup is on, since it takes about 4 GB of memory.serverrunsmouthful-serveritself, a Python script with no dependencies. It forwards each request to the engine, applies the dictionary, runs cleanup, and logs stats. It’s the only container that publishes a port, namelywhisper-server’s usual 8080.
Parakeet without cleanup is the default. To change it, I put e.g. ENGINE=whisper and CLEANUP=1 in server/.env and rerun make server, which takes a few seconds.
Stats
For each dictation, mouthful-server adds a line to ~/.local/share/mouthful/dictations.jsonl with the time, the audio length, the engine and its timing, the cleanup result, the word count before and after cleanup, the number of dictionary fixes, and the client’s user agent. It never logs audio or text. If a client loses a response and sends the same audio again, the server logs the dictation only once. A line looks like this:
{"asr_model":"parakeet-tdt-0.6b-v3-q8_0","asr_ms":342,"audio_ms":11000,"cleanup":"applied","cleanup_ms":1051,"fixes":0,"llm_model":"Qwen3.5-4B-Q4_0","raw_words":22,"time_ms":1793226433313,"total_ms":1404,"user_agent":"curl/8.23.0","words":22}
Run make stats in the repo for a summary per engine and cleanup setting. Words per minute include pauses. Here’s mine so far:
engine cleanup dictations words wpm median_s rejected
ggml-small.en-q8_0 false 121 876 117 0.788 0
Clients
To set up Vocalinux on a client:
- Run
yay -S python-onnxruntime vocalinuxand thenvocalinux - In Vocalinux’s Settings > Speech Model > Advanced > Engine, select Remote API
- Set the server URL to
http://<server hostname or IP>:8080, and keep the endpoint at/inference
Clients of OpenAI’s API can also POST to /v1/audio/transcriptions.
Why not VocaGateway?
The makers of Vocalinux also publish VocaGateway, a self-hosted transcription server for their apps. It’s far more polished than my setup, with a web UI, phone pairing, and dozens of models. So I ran its Docker image (commit f7180fb, CPU build, with its recommended models) on my NucBox. Then I transcribed the same two passages as above, read into both my laptop and my phone:
| Setup (laptop / phone) | Word error rate on the jargon passage | Word error rate on plain prose | Time per chunk of up to 12 s |
|---|---|---|---|
| VocaGateway with Parakeet TDT 0.6B v2 | 13.9% / 10.8% | 3.4% / 4.5% | 0.6 s |
| Same, with its transcript cleanup | 13.9% / 10.8% | 3.4% / 4.5% | 3.5 to 4.0 s |
VocaGateway with small.en | 18.6% / 9.3% | 3.9% / 3.4% | 2.5 s |
| My Whisper, with greedy decoding at the time | 18.6% / 9.3% | 6.7% / 5.1% | 1.0 s |
| My Parakeet with cleanup | 10.3% / 12.9% | 2.2% / 1.1% | 2.0 to 2.3 s |
| My Parakeet with hotwords | 8.8% / 11.3% | 2.2% / 1.7% | 0.3 s |
Both error rates ignore capitalization and punctuation.
- VocaGateway has no custom dictionary. On my laptop, it made more errors on the jargon passage than my Parakeet with hotwords, but on my phone, the two were about even.
- Its transcript cleanup uses a small LLM (Qwen3 1.7B) that fixes only grammar. It didn’t change Parakeet’s transcripts at all, but it added about 3 seconds on my CPU, and it made the
small.entranscripts slightly worse. - Its Linux image runs on the CPU. You can build a CUDA or Vulkan image yourself, but its Vulkan image uses Ubuntu’s older Mesa. On my Iris Xe, whisper.cpp’s own Ubuntu-based Vulkan image was 45% slower than Arch’s packages.
- Its Parakeet TDT 0.6B v3 model rejected one 4-second chunk outright (“The selected model returned nothing for en”), which would have lost that dictation.
- Its
small.enmade fewer errors on plain prose than mine, because it decodes with beam search. On my GPU, beam search costs only about 0.2 seconds, so I now use it too.
What didn’t help
- Whisper’s
small.en-q5_1model: it was about 0.1 seconds faster than q8_0, but less accurate. --no-fallback: when Whisper’s output looks unreliable, Whisper decodes again at higher temperatures. I suspected this of causing one 7.7-second outlier, but turning it off changed nothing. Decoding the same audio gives the same result every time, so the outlier was probably the GPU being busy with the LLM.- A lower VAD threshold (0.35): it still dropped short first words.
- whisper.cpp’s SYCL image for Intel GPUs (
main-intel): it got every word wrong and took 7 to 14 seconds per chunk. - Parakeet on the CPU: through ggml, Parakeet took 1.35 seconds per 11 seconds of audio, vs 0.3 seconds on the GPU.
- Smaller cleanup LLMs: a 2B model didn’t improve Whisper’s transcripts, and with Parakeet it got the exact capitalization of only 4 of 21 jargon terms right.
- MTP drafts alone: without drafts copied from the transcript, the MTP head made cleanup slower than my original setup.
- Other draft lengths: drafts of 10 or 12 tokens were slower than 7, and shorter drafts didn’t help.
- Other llama.cpp options: turning off flash attention, fewer CPU threads, no CPU polling, and Vulkan’s other matrix-vector kernels didn’t help. Turning off Vulkan’s integer dot products made cleanup twice as slow.
- Asking the LLM for a list of fixes instead of a rewrite: cleanup left most transcripts unchanged, yet it spent about a second writing each one again. So I asked Qwen3.5 4B to list only its fixes, e.g.
get hub => GitHub. That halved the median time, but the model missed real fixes and made up others, so it made more errors than the full rewrite (11.3% / 13.9% vs 10.3% / 12.9% on the jargon passage). - Hotwords through sherpa-onnx: it supports Parakeet hotwords with beam search, but only on the CPU on my NucBox. Its int8 model dropped words (7.3% / 3.4% on plain prose), and its fp32 model took 1.5 seconds per chunk.
Ideas that I haven’t tried
- Running Whisper’s encoder through OpenVINO, Intel’s own inference stack. The encoder always processes a 30-second window, which takes most of Whisper’s time.
- Transcribing chunks while I’m still speaking, so that only the last chunk is left when I stop. Parakeet with hotwords already takes only about 0.3 seconds per chunk, so this would save little.
- Quantizing the LLM’s output layer (q6_K) and
ssm_outtensors (q5_K) to q4_0. The output layer alone takes about 16 ms of each 90 ms decoding step, so this might save about 0.1 seconds, but it would cost some accuracy. - Building llama.cpp from Arch’s packages, since llama.cpp’s own Vulkan image probably has the same older Mesa as whisper.cpp’s.
- Beam search with hotwords on the GPU.
libparakeetonly decodes greedily, and with the same extra terms in my dictionary, sherpa-onnx’s beam search spelled 17 / 21 jargon terms right vs 16 / 17 for my greedy hotwords.