I built local dictation for macOS, then took the LLM back out

I start a maths degree this autumn, and I write in Typst. Dictation is obvious for that: your hands are on paper, the notation is long to type, and the good commercial option costs $15 a month and ships your voice to someone else's server. whisper.cpp runs on my laptop, so I built the small thing around it.

That part took an evening. Hold the right option key, an AVAudioEngine tap records 16 kHz mono into RAM, release posts the buffer to a whisper-server child process that keeps large-v3-turbo hot, and the result goes in through the pasteboard and ⌘V with the clipboard restored afterwards. Fourteen seconds of German comes back in 1.3 seconds, language auto-detected. Fine.

Then I added the step everyone adds — a small local model to strip the ähms and fix the punctuation — and that is where it got interesting.

The cleanup pass invented a theorem

I gave llama3.2:3b a transcript of me talking about convergence and a system prompt telling it, in so many words, to remove filler and change nothing else. Here is the input and what came back, unedited:

Read the output again. The filler is gone, which is what I asked for. It also finished the definition. sodass für alle x > N gilt: |x − L| < ε was never spoken; the model completed the sentence the way a textbook would. It did not tidy my dictation — it answered it.

That is the specific failure that makes an LLM the wrong tool here. A wrong word in a dictated sentence is visible; you reread it and fix it. A wrong theorem in your own handwriting-equivalent is not, because it looks exactly like something you would have written, and it is correct-sounding mathematics. It just is not what you said.

Hardening the prompt helped and did not fix it. With explicit delimiters and a list of prohibitions — do not complete, do not answer, every word of your output must appear in the input — the 3B model stopped inventing on that sentence and instead left every filler word in place on the next one. mistral:7b did better on content and then wrapped its answer in quotation marks the prompt had explicitly banned, replaced es gibt with existiert, and took 3.8 seconds.

Meanwhile the thing I was trying to improve on — raw whisper large-v3-turbo — already punctuates German reasonably and swallows most ähms by itself. So the default in Echo is now: if there is no API key, the cleanup step is off, and you get the raw transcript. A hallucinating cleanup is worse than no cleanup, and the bar it has to clear is higher than I assumed.

Two guards, for the people who want it anyway

Ollama is still there behind an explicit opt-in, with two checks that throw the model's answer away and keep the raw text:

• Length ratio. The result must be between 50% and 115% of the original. Catches the confident expansion.
• Novel words. At most 20% of the output words may be absent from the input. Cleanup may reorder and delete; it may not bring much new vocabulary.

I only added the second one because the first was not enough, and I know that because I tested the guards against the actual recorded failures rather than against my idea of them. The epsilon-N fabrication above is 128 characters against 119 — a ratio of 1.08, comfortably inside any length window I would have picked. It fails the word check at 33% novel words. The other hallucination that day failed on length and passed on words. One check would have shipped a hole.

Maths mode is a table, not a model

Hold shift while you press the key and Echo renders spoken notation. Delta Phi plus Yx ist gleich 2 Omega Quadrat becomes $Delta Phi + Y x = 2 Omega^2$ in Typst, or \Delta \Phi + Yx = 2 \Omega^2 in LaTeX, or ΔΦ + Yx = 2Ω² in Unicode.

Given the section above, this is deliberately a lookup table: Greek letters, relations, quantifiers, set operations, arbitrary exponents and subscripts. A table can be wrong or absent. It cannot invent a theorem. In maths mode the LLM cleanup is skipped entirely, because it would rewrite exactly the literal words the renderer keys on.

Two things fell out of building it that I did not expect. The first is German grammar: German capitalises every noun, so whisper writes Epsilon whether you mean ε or not. Capitalisation carries no information for letters without a distinct uppercase form, so those always render lowercase, and only letters that have a real uppercase glyph — Delta, Omega, Phi — follow the case whisper produced.

The second is that Yx breaks Typst. In Typst maths, a run of letters is an identifier, not multiplication, so Yx is an undefined name and the document fails to compile. Whisper, naturally, writes dictated variables together. I only found this because I piped every line the renderer produced through the real typst compile instead of reading them and nodding — which is the same mistake as trusting the cleanup model, one layer up. Runs containing a capital are now split back apart; leftover German words get quoted so they render as upright text instead of breaking the file.

The war story: whisper says “you” when it hears nothing

For an hour, every dictation inserted the word you. Nothing else, whatever I said.

That is whisper's signature hallucination on silence — the training data is full of YouTube subtitles, and an empty audio buffer decodes to you or Thank you. remarkably often. So the recording contained no sound. The microphone permission had to be the problem.

Except the app was not in the microphone list in System Settings at all — not denied, absent. An app only appears there once it has actually asked, and there is no button to add one by hand. I moved the app to /Applications, reset every TCC entry, requested access from a menu item instead of at launch, watched the permission dialog appear, clicked Allow, and it still did not show up.

The cause was in my build script. I sign with --options runtime, and the hardened runtime blocks microphone access at the process level unless the app carries com.apple.security.device.audio-input. I had never created an entitlements file. The permission dialog still appears, because TCC is a different layer and perfectly happy; the audio engine behind it just receives buffers of zeros. No error, no exception, no log line. Silence, and whisper politely says you.

Two lines of XML. The lesson I actually take from it is about the failure mode rather than the fix: three independent systems — hardened runtime, TCC, and a speech model — each behaved reasonably on its own, and together they produced a confident wrong answer with no error anywhere in the stack. I spent that hour chasing code signatures and file paths, and I ruled each of them out by reproducing them, which is the only reason I did not “fix” something that was never broken.

Echo now refuses to insert anything when the peak amplitude of a recording is below a threshold, and shows Kein Ton instead. The last recording is always written to ~/.config/echo/last.wav, so the next person can settle the question in one command instead of an hour.

What it is not

• You have to build it yourself. I have an Apple Development certificate, not a Developer ID, so I cannot ship a signed binary that another Mac will run without a fight. ./scripts/build.sh signs with whatever identity is in your keychain and installs to /Applications. This is the honest cost of not paying Apple $99 a year, and it is the main reason Echo is a repo rather than a download.
• The hosted cleanup path is untested. The Anthropic branch is implemented and I have no API key, so I have not verified it end to end. It says so in the README too. A cleanup model with actual capability may well clear the bar the 3B models did not — I just have not measured it, and I am not going to claim it.
• It polls, it does not watch. No resident watcher process, no event API. For a hold-to-talk key that is the right trade; it also means there is nothing clever to admire.
• Secure input fields block it, because macOS blocks synthetic keystrokes there. Echo detects that and says so rather than silently doing nothing.

The thing I keep turning over is that the whole project is an argument with itself. The transcription is a neural model and it is excellent — nothing hand-written comes close. The two steps after it both started as models and both ended up as deterministic code, because their failure mode was not noise but confident invention, in a domain where I cannot afford to check every line. Same stack, opposite conclusions, twenty lines apart.