How it works
Stages one to three run continuously while you speak; stages four to six fire when you pause or release the key.
Capture and the 16 kHz decision
Your microphone samples the world tens of thousands of times per second, but speech recognition models were almost all trained on 16 kHz mono audio — a convention inherited from telephony and early speech corpora. Every app therefore resamples your expensive 48 kHz studio signal down to 16 kHz before anything else happens. This is why exotic audio gear rarely improves dictation accuracy: the model literally cannot hear the difference you paid for.
Finding the speech: VAD and chunking
The voice-activity detector scans the stream frame by frame, labelling each as speech or not-speech. Its output drives two things: when to start and stop recognition, and how to cut the audio into chunks the model can digest. Transformer speech models have a fixed attention window — famously 30 seconds in Whisper's case — so long dictation is segmented, transcribed in overlapping windows, and stitched. Stitch seams are why a word spoken right at a chunk boundary occasionally comes out garbled or duplicated.
Aggressive chunking is also the source of clipped beginnings: if the VAD fires a hundred milliseconds late, the first syllable never reaches the model. This is the pipeline stage responsible for most 'it missed my first word' complaints.
The model itself: spectrograms in, tokens out
Recognition proper starts with a Fourier transform: the raw waveform becomes a spectrogram, a picture of energy across frequencies over time. The encoder half of the model reads that picture and compresses it into a representation of what sounds occurred; the decoder half generates text tokens one by one, each conditioned on the audio representation and the text so far. This is why speech models sometimes 'correct' your audio toward more probable sentences — the decoder has a language model's sense of what word should come next, and under noisy audio it follows it.
Streaming versus batch is the big design fork here. Batch apps (most Whisper-based local tools) wait for a pause, then transcribe the whole chunk: simpler, slightly more accurate, but text arrives in lumps. Streaming apps (the cloud leaders) transcribe incrementally with partial hypotheses that revise as you speak — the flickering, self-correcting text you see in Wispr Flow is the model literally changing its mind mid-sentence.
After recognition: cleanup and injection
The raw transcript then passes through the cleanup layer covered in our LLM-layer guide — filler removal, punctuation, formatting — and the result is handed to the injection mechanism covered in our text-injection guide. Each stage in this article links to its own deep dive; the point of seeing the whole chain is that every stage trades a little accuracy for a little usability, and the app's character is the sum of those trades.
It also explains the latency budget. Capture and VAD are essentially free; recognition dominates at a few hundred milliseconds per chunk locally (less with Parakeet-class models, more with Whisper large on a weak machine); the LLM cleanup adds its own call; injection is instant unless it falls back to synthetic typing. The best apps overlap these stages; the worst run them in series, which is the difference between dictation that feels telepathic and dictation that feels like fax.
References
- Radford et al. — Robust Speech Recognition via Large-Scale Weak Supervision (Whisper paper) — the encoder–decoder design most local apps still run
- NVIDIA — FastConformer / Parakeet architecture notes — the newer, faster local alternative
Frequently asked questions
Why does dictated text sometimes change after it appears?
That's streaming recognition revising its partial hypothesis as more audio arrives — the model's first guess at a word is updated once it hears the rest of the sentence. It's a feature of streaming pipelines; batch apps don't do it but deliver text later.
Why 30-second chunks?
Whisper's attention window is 30 seconds of audio, a training decision that became an industry convention. Longer dictation is split, transcribed in overlapping windows, and stitched — seams and all.