Self-Hosted Voice Dictation with Whisper and Parakeet

I run speech recognition on my home server (a GMKtec NucBox with Intel Iris Xe graphics) for voice dictation. My phone runs Mouthful, my Android dictation app, and my weaker devices run Vocalinux. Both send their audio to mouthful-server, a small server in Mouthful’s repo that puts a speech engine behind whisper-server’s API: OpenAI’s Whisper or NVIDIA’s Parakeet. It applies my custom dictionary of jargon, can have an LLM clean up each transcript, and logs stats about each dictation. It all runs in Docker Compose on my Arch Linux server.

Engines

The server’s config chooses the engine and whether an LLM cleans up the transcript, so that I can switch without changing my clients. To compare the choices, I read two of my blog posts into both my laptop and my phone. Then I split each recording at pauses into chunks of up to 12 seconds, like Vocalinux does, and scored each engine:

Jargon only counts if the capitalization is exact. Over Wi-Fi, my phone’s round trip adds about 0.2 seconds to these times, about 40% of which is the upload.

Whisper (small.en-q8_0) spells the most jargon right on its own, because it can take my dictionary as a prompt. On my NucBox, it transcribes a 10-second utterance in about 1 second. medium.en-q8_0 took 2.6 seconds and large-v3-turbo-q8_0 took 3.1 seconds. Published benchmarks show that the larger models are more accurate on realistic speech (e.g. about 30% fewer errors on Common Voice for large-v2 vs small.en). But on my recordings, large-v3-turbo made only slightly fewer errors than small.en on plain prose and no fewer on jargon. Shorter audio doesn’t help, since Whisper always processes a full 30-second window. (The --audio-ctx option shrinks that window, but it destroyed accuracy in my tests.) The q8_0 models were also faster than their f16 versions on this GPU.

Parakeet (TDT 0.6B v3) is about four times as fast and makes fewer errors on plain prose, especially from my laptop’s noisy mic. But it can’t take a prompt, so it misspells most of my jargon. It runs on the GPU through whisper.cpp’s libparakeet, which only comes with a command-line tool, so I wrap it in a small server. Earlier, I ran Parakeet TDT 0.6B v2 on the CPU through onnx-asr, which was slower and less accurate.

Cleanup has Qwen3.5 4B fix misheard words with my dictionary as a glossary, and fix punctuation, capitalization, and filler words. It takes about 1 to 1.5 seconds on my real dictations, which are shorter than the chunks above. Parakeet with cleanup made the fewest errors and spelled the most jargon right. If the LLM changes too much of a transcript, e.g. by answering a dictated question, the server keeps the original.

Whisper uses Silero VAD to skip audio without speech. Without it, Whisper often transcribes silence as “Thank you.” But VAD’s default padding of 30 ms also clips short words at the edges of speech. When my phone kept losing the first word of a dictation, I first suspected the app’s mic, but it starts recording within 60 ms of the tap. In 24 short clips that started just before a phrase, VAD dropped a first or last word (e.g. “and” or “or”) in 4. Padding of 200 ms fixed all of them, while 400 ms let Whisper hallucinate a lone “and” again.

Dictionary

Speech recognition often misspells jargon, e.g. “Vocalinox” for Vocalinux and “ripcrep” for ripgrep. mouthful-server uses my dictionary to fix this in three ways:

1. It sends the dictionary’s terms to Whisper as a prompt, which makes Whisper more likely to spell them right. On the passage about Unix shells above, my dictionary raised the jargon that Whisper spelled right from 3 to 9 terms on my laptop and from 5 to 8 on my phone, with almost no added latency. Ignoring capitalization, Whisper already got most of them (12 and 16 of 21), so the prompt mostly fixes capitalization, e.g. “zsh” to “Zsh”. But Whisper only reads the last 224 tokens (roughly 700 characters) of a prompt.
2. It gives the terms to the cleanup LLM as a glossary.
3. It replaces known mishearings in the transcript, e.g. “ripcrep” with “ripgrep”. This fixes words that the prompt can’t fix, works for both engines, and has no size limit.

(Vocalinux has its own custom dictionary, but its Remote API engine doesn’t send the terms to the server.)

Create ~/.config/mouthful/dictionary.txt. Each line is a term, optionally followed by a colon and comma-separated mishearings. Mishearings match case-insensitively and only as whole words, and a space also matches a hyphen. To fix a term’s capitalization, list the term itself as a mishearing. Put the most important terms first, since only they fit in Whisper’s prompt. The server rereads the file on every request, so edits apply immediately.

# Only list terms that speech recognition gets wrong, and only list mishearings that I never mean literally
Vocalinux: vocalinox, vocal linux
KeePassXC: keypass xc, keepass xc, key pass xc
Syncthing: syncthing, sync thing, singthing
tmux: teamux, teamox, team mox
ripgrep: ripcrep, rip crep
yq: ik

Setup

mouthful-server lives in the server folder of Mouthful’s repo. It needs Linux with Docker and an Intel or AMD GPU, although I’ve only tested Intel. For NVIDIA GPUs, edit compose.yaml and the Dockerfile there.

git clone https://github.com/artnc/mouthful.git
cd mouthful
make server

This downloads the four models (about 3.5 GB) at pinned versions, checks their hashes, and starts up to three containers:

• engine runs either Whisper or Parakeet. Whisper’s whisper-server comes from Arch’s packages, which I install from the Arch Linux Archive’s snapshot of a fixed date. On my Iris Xe, they took 1.0 seconds per dictation vs 1.45 seconds for whisper.cpp’s own Vulkan image, which has an older Mesa and no AVX-512. Parakeet runs through whisper.cpp’s libparakeet on the GPU.
• llama runs llama.cpp’s Vulkan image with Qwen3.5 4B for cleanup. It runs only if cleanup is on, since it takes about 4 GB of memory.
• server runs mouthful-server itself, a Python script with no dependencies. It forwards each request to the engine, applies the dictionary, runs cleanup, and logs stats. It’s the only container that publishes a port, namely whisper-server’s usual 8080.

Parakeet with cleanup is the default. To change it, I put e.g. ENGINE=whisper and CLEANUP=0 in server/.env and rerun make server, which takes a few seconds.

Stats

For each dictation, mouthful-server adds a line to ~/.local/share/mouthful/dictations.jsonl with the time, the audio length, the engine and its timing, the cleanup result, the word count before and after cleanup, the number of dictionary fixes, and the client’s user agent. It never logs audio or text. If a client loses a response and sends the same audio again, the server logs the dictation only once. A line looks like this:

{"asr_model":"parakeet-tdt-0.6b-v3-q8_0","asr_ms":342,"audio_ms":11000,"cleanup":"applied","cleanup_ms":1051,"fixes":0,"llm_model":"Qwen3.5-4B-Q4_0","raw_words":22,"time_ms":1793226433313,"total_ms":1404,"user_agent":"curl/8.23.0","words":22}

Run make stats in the repo for a summary per engine and cleanup setting. Words per minute include pauses. Here’s mine so far:

engine              cleanup  dictations  words  wpm  median_s  rejected
ggml-small.en-q8_0  false    121         876    117  0.788     0

Clients

To set up Vocalinux on a client:

1. Run yay -S python-onnxruntime vocalinux and then vocalinux
2. In Vocalinux’s Settings > Speech Model > Advanced > Engine, select Remote API
3. Set the server URL to http://<server hostname or IP>:8080, and keep the endpoint at /inference

Clients of OpenAI’s API can also POST to /v1/audio/transcriptions.

Why not VocaGateway?

The makers of Vocalinux also publish VocaGateway, a self-hosted transcription server for their apps. It’s far more polished than my setup, with a web UI, phone pairing, and dozens of models. So I ran its Docker image (commit f7180fb, CPU build, with its recommended models) on my NucBox. Then I transcribed the same two passages as above, read into both my laptop and my phone:

Jargon only counts if the capitalization is exact.

• VocaGateway has no custom dictionary, so it spelled much less of my jargon right.
• Its transcript cleanup uses a small LLM (Qwen3 1.7B) that fixes only grammar. It didn’t change Parakeet’s transcripts at all, but it added about 3 seconds on my CPU, and it made the small.en transcripts slightly worse.
• Its Linux image runs on the CPU. You can build a CUDA or Vulkan image yourself, but its Vulkan image uses Ubuntu’s older Mesa. On my Iris Xe, whisper.cpp’s own Ubuntu-based Vulkan image was 45% slower than Arch’s packages.
• Its Parakeet TDT 0.6B v3 model rejected one 4-second chunk outright (“The selected model returned nothing for en”), which would have lost that dictation.
• Its small.en made fewer errors on plain prose than mine, because it decodes with beam search. On my GPU, beam search costs only about 0.2 seconds, so I now use it too.

What didn’t help

• Whisper’s small.en-q5_1 model: it was about 0.1 seconds faster than q8_0, but less accurate.
• --no-fallback: when Whisper’s output looks unreliable, Whisper decodes again at higher temperatures. I suspected this of causing one 7.7-second outlier, but turning it off changed nothing. Decoding the same audio gives the same result every time, so the outlier was probably the GPU being busy with the LLM.
• A lower VAD threshold (0.35): it still dropped short first words.
• whisper.cpp’s SYCL image for Intel GPUs (main-intel): it got every word wrong and took 7 to 14 seconds per chunk.
• Parakeet on the CPU: through ggml, Parakeet took 1.35 seconds per 11 seconds of audio, vs 0.3 seconds on the GPU.
• Smaller cleanup LLMs: a 2B model didn’t improve Whisper’s transcripts, and with Parakeet it spelled only 4 of 21 jargon terms right.

Ideas that I haven’t tried

• Running Whisper’s encoder through OpenVINO, Intel’s own inference stack. The encoder always processes a 30-second window, which takes most of Whisper’s time.
• Transcribing chunks while I’m still speaking, so that only the last chunk is left when I stop.
• Building llama.cpp from Arch’s packages, since llama.cpp’s own Vulkan image probably has the same older Mesa as whisper.cpp’s.
• Parakeet with hotwords (e.g. through sherpa-onnx), which could spell my jargon right without the LLM.