Local Voice Assistant
A microcontroller captures voice and streams it to a host PC running the full speech-to-text → LLM → text-to-speech pipeline — entirely on hardware I own, with no cloud APIs and no internet dependency.
How it works
An ESP32-S3 samples the mic at 16kHz, runs voice-activity detection with an adaptive noise floor, and streams the clip over a persistent TCP connection. The server transcribes it locally with faster-whisper, routes the transcript through a small dispatcher (regex intent matching first, an Ollama-hosted LLM only for open-ended conversation or RAG), and streams the reply back through Piper TTS to a Bluetooth speaker — while a Flask + SocketIO web UI mirrors the conversation in real time for a projector display.
Key design decisions
Server-owned state machine
The microcontroller never decides what happens next — it just executes whatever command the server sends. New interaction modes (multi-turn sessions, grocery/recipe/cooking sub-states) shipped entirely in Python, with zero firmware changes.
Regex before LLM, everywhere it’s viable
Every tool-shaped interaction (inventory, recipes, settings, navigation) is handled by hand-written regex before it ever reaches the LLM — a regex match is ~0ms versus 2–3s for an LLM round trip, and deterministic commands shouldn’t pay that latency or risk that unpredictability.
Streaming, sentence-at-a-time TTS
LLM generation is the slowest stage by a wide margin. Each completed sentence is spoken as soon as it’s ready, while the model keeps generating the next one — the single biggest perceived-latency win in the system.
Adaptive VAD threshold
A fixed energy threshold works in a quiet room and fails the moment there’s a TV or street noise nearby. An exponential moving average of background energy sets the effective threshold dynamically (4× ambient), so it holds up across real environments.
Bluetooth output over an onboard speaker
An early hardware revision drove a speaker off the microcontroller’s DAC through a cheap amp and clipped at any usable volume. Routing final audio to a standard Bluetooth speaker over A2DP sidesteps analog amplifier design entirely.
Measured latency (CPU-only, no GPU)
The pipeline auto-detects and uses CUDA where available for both STT and TTS, with no code path changes required.