Self-Hosted Voice Dictation with Whisper and Parakeet

I run speech recognition on my home server (a GMKtec NucBox with Intel Iris Xe graphics) for voice dictation. My phone runs Mouthful, my Android dictation app, and my weaker devices run Vocalinux. Both send their audio to mouthful-server, a small server in Mouthful’s repo that puts a speech engine behind whisper-server’s API: OpenAI’s Whisper or NVIDIA’s Parakeet. It applies my custom dictionary of jargon, can have an LLM clean up each transcript, and logs stats about each dictation. It all runs in Docker Compose on my Arch Linux server.

Engines

The server’s config chooses the engine and whether an LLM cleans up the transcript, so that I can switch without changing my clients. To compare the choices, I read two of my blog posts into both my laptop and my phone. Then I split each recording at pauses into chunks of up to 12 seconds, like Vocalinux does, and scored each engine:

Setup (laptop / phone)Jargon spelled right (of 21)Word error rate on plain proseTime per chunk
Whisper12 / 165.6% / 2.8%1.2 to 1.3 s
Whisper with cleanup12 / 165.1% / 2.2%2.8 to 3.1 s
Parakeet13 / 132.2% / 1.7%0.3 s
Parakeet with cleanup13 / 142.2% / 1.1%2.0 to 2.3 s
Parakeet with hotwords15 / 152.2% / 1.7%0.3 s

Jargon counts in any capitalization, e.g. “zsh” for Zsh, since I care about the words more than their case. Over Wi-Fi, my phone’s round trip adds about 0.2 seconds to these times, about 40% of which is the upload.

Whisper (small.en-q8_0) can take my dictionary as a prompt, but in my tests that only fixed capitalization (see below). On my NucBox, it transcribes a 10-second utterance in about 1 second. medium.en-q8_0 took 2.6 seconds and large-v3-turbo-q8_0 took 3.1 seconds. Published benchmarks show that the larger models are more accurate on realistic speech (e.g. about 30% fewer errors on Common Voice for large-v2 vs small.en). But on my recordings, large-v3-turbo made only slightly fewer errors than small.en on plain prose and no fewer on jargon. Shorter audio doesn’t help, since Whisper always processes a full 30-second window. (The --audio-ctx option shrinks that window, but it destroyed accuracy in my tests.) The q8_0 models were also faster than their f16 versions on this GPU.

Parakeet (TDT 0.6B v3) is about four times as fast and makes fewer errors on plain prose, especially from my laptop’s noisy mic. But it can’t take a prompt, so on its own it gets the case of most of my jargon wrong, e.g. “ZSH”, and misspells some of it. It runs on the GPU through whisper.cpp’s libparakeet, which only comes with a command-line tool, so I wrap it in a small server. Earlier, I ran Parakeet TDT 0.6B v2 on the CPU through onnx-asr, which was slower and less accurate.

Hotwords make Parakeet spell my dictionary’s terms right without an LLM. Parakeet decodes one token at a time and picks the token with the highest score. I patched libparakeet so that my wrapper can change those scores before each step. The wrapper splits each term into Parakeet’s own tokens, and then it raises the score of each term’s first token by 1. Once the transcript ends with a term’s first few tokens, it raises the score of the term’s next token by 6. Stronger boosts for first tokens raised the error rate on my laptop’s plain prose from 2.2% to 2.8%. With only my real dictionary, Parakeet with hotwords spelled more jargon right than Parakeet with cleanup, and the boosts cost no time that I could measure. So it’s the default now.

Cleanup has Qwen3.5 4B fix misheard words with my dictionary as a glossary, and fix punctuation, capitalization, and filler words. It takes about 1.1 seconds on transcripts as short as my real dictations, which are shorter than the chunks above. Parakeet with cleanup made the fewest errors on plain prose, since the LLM also fixes ordinary words that aren’t in my dictionary. But it’s off by default, since hotwords spell more jargon right in a seventh of the time. If the LLM changes too much of a transcript, e.g. by answering a dictated question, the server keeps the original.

Since cleanup took most of the time, I tuned llama.cpp for it before I added hotwords. On 49 short transcripts, these changes cut its mean time from 1.58 to 1.16 seconds with the same accuracy:

  1. Speculative decoding drafts tokens by copying them from the transcript, since cleanup mostly repeats it, and the LLM checks each draft in one pass. Drafts of up to 7 tokens beat the 24 that I started with. llama.cpp’s Vulkan backend runs batches of up to 8 tokens with fast matrix-vector kernels, but a batch of 9 to 34 tokens costs as much as 3 to 4 single tokens.
  2. If there’s nothing to copy, Qwen3.5’s own multi-token prediction (MTP) head drafts 2 tokens. This needs Unsloth’s MTP version of the model.
  3. Qwen3.5’s recurrent layers can’t roll back to the end of the system prompt, which every cleanup shares. llama.cpp works around this with checkpoints, but they split every prompt into two batches. Instead, mouthful-server saves the LLM’s state after the system prompt once and restores it before each cleanup. This cut the time to read each transcript from 450 to 300 ms.

Whisper uses Silero VAD to skip audio without speech. Without it, Whisper often transcribes silence as “Thank you.” But VAD’s default padding of 30 ms also clips short words at the edges of speech. When my phone kept losing the first word of a dictation, I first suspected the app’s mic, but it starts recording within 60 ms of the tap. In 24 short clips that started just before a phrase, VAD dropped a first or last word (e.g. “and” or “or”) in 4. Padding of 200 ms fixed all of them, while 400 ms let Whisper hallucinate a lone “and” again.

Dictionary

Speech recognition often misspells jargon, e.g. “Vocalinox” for Vocalinux and “ripcrep” for ripgrep. mouthful-server uses my dictionary to fix this in four ways:

  1. It sends the dictionary’s terms to Whisper as a prompt, which makes Whisper more likely to capitalize them right. On the passage about Unix shells above, my dictionary raised the jargon that Whisper wrote with the exact capitalization from 3 to 7 terms on my laptop and from 5 to 8 on my phone, with almost no added latency. Ignoring capitalization, it made no difference (12 and 16 of 21 either way), so the prompt only fixes capitalization, e.g. “zsh” to “Zsh”. But Whisper only reads the last 224 tokens (roughly 700 characters) of a prompt.
  2. It sends the terms to Parakeet, which boosts them as hotwords (see above).
  3. It gives the terms to the cleanup LLM as a glossary.
  4. It replaces known mishearings in the transcript, e.g. “ripcrep” with “ripgrep”. This fixes words that the prompt can’t fix, works for both engines, and has no size limit.

(Vocalinux has its own custom dictionary, but its Remote API engine doesn’t send the terms to the server.)

Create ~/.config/mouthful/dictionary.txt. Each line is a term, optionally followed by a colon and comma-separated mishearings. Mishearings match case-insensitively and only as whole words, and a space also matches a hyphen. To fix a term’s capitalization, list the term itself as a mishearing. Put the most important terms first, since only they fit in Whisper’s prompt. The server rereads the file on every request, so edits apply immediately.

# Only list terms that speech recognition gets wrong, and only list mishearings that I never mean literally
Vocalinux: vocalinox, vocal linux
KeePassXC: keypass xc, keepass xc, key pass xc
Syncthing: syncthing, sync thing, singthing
tmux: teamux, teamox, team mox
ripgrep: ripcrep, rip crep
yq: ik

Setup

mouthful-server lives in the server folder of Mouthful’s repo. It needs Linux with Docker and an Intel or AMD GPU, although I’ve only tested Intel. For NVIDIA GPUs, edit compose.yaml and the Dockerfile there.

git clone https://github.com/artnc/mouthful.git
cd mouthful
make server

This downloads the four models (about 3.5 GB) at pinned versions, checks their hashes, and starts up to three containers:

Parakeet without cleanup is the default. To change it, I put e.g. ENGINE=whisper and CLEANUP=1 in server/.env and rerun make server, which takes a few seconds.

Stats

For each dictation, mouthful-server adds a line to ~/.local/share/mouthful/dictations.jsonl with the time, the audio length, the engine and its timing, the cleanup result, the word count before and after cleanup, the number of dictionary fixes, and the client’s user agent. It never logs audio or text. If a client loses a response and sends the same audio again, the server logs the dictation only once. A line looks like this:

{"asr_model":"parakeet-tdt-0.6b-v3-q8_0","asr_ms":342,"audio_ms":11000,"cleanup":"applied","cleanup_ms":1051,"fixes":0,"llm_model":"Qwen3.5-4B-Q4_0","raw_words":22,"time_ms":1793226433313,"total_ms":1404,"user_agent":"curl/8.23.0","words":22}

Run make stats in the repo for a summary per engine and cleanup setting. Words per minute include pauses. Here’s mine so far:

engine              cleanup  dictations  words  wpm  median_s  rejected
ggml-small.en-q8_0  false    121         876    117  0.788     0

Clients

To set up Vocalinux on a client:

  1. Run yay -S python-onnxruntime vocalinux and then vocalinux
  2. In Vocalinux’s Settings > Speech Model > Advanced > Engine, select Remote API
  3. Set the server URL to http://<server hostname or IP>:8080, and keep the endpoint at /inference

Clients of OpenAI’s API can also POST to /v1/audio/transcriptions.

Why not VocaGateway?

The makers of Vocalinux also publish VocaGateway, a self-hosted transcription server for their apps. It’s far more polished than my setup, with a web UI, phone pairing, and dozens of models. So I ran its Docker image (commit f7180fb, CPU build, with its recommended models) on my NucBox. Then I transcribed the same two passages as above, read into both my laptop and my phone:

Setup (laptop / phone)Word error rate on the jargon passageWord error rate on plain proseTime per chunk of up to 12 s
VocaGateway with Parakeet TDT 0.6B v213.9% / 10.8%3.4% / 4.5%0.6 s
Same, with its transcript cleanup13.9% / 10.8%3.4% / 4.5%3.5 to 4.0 s
VocaGateway with small.en18.6% / 9.3%3.9% / 3.4%2.5 s
My Whisper, with greedy decoding at the time18.6% / 9.3%6.7% / 5.1%1.0 s
My Parakeet with cleanup10.3% / 12.9%2.2% / 1.1%2.0 to 2.3 s
My Parakeet with hotwords8.8% / 11.3%2.2% / 1.7%0.3 s

Both error rates ignore capitalization and punctuation.

What didn’t help

Ideas that I haven’t tried