Confucius4-R2T2 in TypeWhisper: real-time dictation on a Mac, powered by audio.cpp

[scriptease]

TypeWhisper, the open-source macOS dictation app, now has an official Confucius4-R2T2 add-on, and audio.cpp runs it: https://www.typewhisper.com/en/addons/confucius4-r2t2/

You pick a model in the plugin settings and dictate. Text appears while you speak and is final almost as soon as you stop. Everything runs on-device with Metal.

How it uses audio.cpp

• It streams 16 kHz PCM to the audiocpp_server live route (/v1/audio/transcriptions/live) and turns the append-only SSE deltas into live text. TypeWhisper dictionary terms go in as the prompt.
• By default, the plugin downloads the pinned v0.9.0 release (audio-v0.9.0-bin-macos-arm64-metal.tar.gz) and a GGUF of your choice: Q4_K_M, Q8_0 or F16. It checks the size and SHA-256 of each, then runs audiocpp_server on a random loopback port, restarting it if it crashes. People with their own build can point the plugin at their own server instead.
• Source: TypeWhisperPluginSDK/Plugins/R2T2Plugin, merged in Add Confucius4-R2T2 community transcription plugin TypeWhisper/typewhisper-mac#1432.

v0.9.0 is now in official use. The released TypeWhisper add-on (plugin-r2t2-v0.2.0) downloads and runs the stock audio.cpp v0.9.0 release, unmodified, for every user who picks the built-in server. The binary comes from your GitHub release, pinned by version, size and SHA-256.

What made it possible. Everything the plugin needs is in stock v0.9.0:

• feat(model): add Confucius4-R2T2 real-time streaming ASR (r2t2_asr) #604 by @davidxifeng: the R2T2 port itself
• Accept a prompt query parameter on the live transcription route #618: the prompt parameter on the live route
• confucius4_r2t2: make streaming chunks incremental instead of O(utterance) #696: incremental streaming, so finalize no longer re-transcribes the whole utterance (a 60 s dictation used to take 25 s to finalize and now takes about 0.4 s)
• 955c872 by @0xShug0 (K-quant weights) and confucius4_r2t2: add community Q4_K_M GGUF package (pinned, no re-hosting) #702 by @NairoDorian (the Q4_K_M package, half the size of Q8_0 and faster)
• confucius4_r2t2: fix null deref when the streaming final flush has no tokens #712 by @hdcola: the empty-transcript crash fix; that crash used to take the server down at the end of a silent stream

Thanks to @0xShug0 for reviewing and merging all of this so quickly, and for answering #700 about pinning release assets. That answer is what made download-and-run possible.

One small thing we hit along the way: the release tarball ships audiocpp_server without the executable bit, so the plugin runs chmod 755 after extracting it. Happy to open an issue if that isn't intended.

The longer story, written up as a blog series: The Whisper saga.

[0xShug0]

@scriptease haha I like "A Team of Strangers". That's a great title for this kind of open-source story. Your writing actually reminds me a little of those old-school stories about hacker culture and engineering teams. Great articles!

For executable bit, would you like to submit a PR to fix it?

[scriptease]

I like writing blog articles with AI. Basically, my AI knows everything about all my sessions, so I just let it source all the information from all the sessions and then write it down with my own voice. It feels kind of scary when you read it out loud and it sounds exactly like yourself.

Here you go #754 :D