Offline Voice Control with ONNX/TFLite Models

One set of small, fixed-keyword models — standard ONNX for phones, browsers and desktops, INT8 TFLite for microcontrollers — that listens for a wake word and a handful of spoken commands, fully offline. This article shows when this kind of model is the right tool, how the same files run on every platform, and walks through a working ESP32-S3 demo you can reproduce with your own words.

When do you need a model like this?

A keyword-spotting (KWS) model is the right tool when the vocabulary is fixed and small — one wake word plus a few dozen commands at most — and the product needs to listen always-on without sending audio anywhere. If users can say anything, you need a full ASR model; but ASR demands an order of magnitude more compute and power, so on an ESP32-class microcontroller or a battery product it is usually out of reach — and always-on listening with it is simply impractical.

Concretely, these are the situations where teams reach for a fixed-keyword model:

• Smart home: “turn on the lights”, “turn on the AC” — one wake word plus a handful of device commands
• Shared devices & attendance: “clock in”, “start shift” — voice check-in where typing is impractical
• Robots & cameras: “take a photo”, “start”, “stop” — control while your hands are busy
• Presentations & kiosks: “next page”, “go back” — no remote, no touching the screen
• Toys & gadgets: a custom wake word that becomes the product’s name
• Push-to-talk radios: voice activation without reaching for the button

All of these have fixed vocabularies and pay the KWS trade-off gladly: you give up open-ended speech and get a model that is tiny, instant, private and free to run. Need a KWS model for a fixed set of words like these? You can train one on voicute.com — in English, German, French, Japanese or Chinese; the steps below walk through the whole path.

One model format, every platform

The open-source View the open-source inference engine and demo models → pipeline ships models in two formats from the same training run: ONNX for anything with an ONNX Runtime, and INT8 TFLite for TFLite Micro devices like the ESP32-S3. The pipeline is three files:

Audio (16 kHz) → melspectrogram.onnx → embedding_model.onnx → your_word.onnx → probability

• melspectrogram.onnx and embedding_model.onnx are shared across all wake words — included in the repo.
• your_word.onnx is the per-word classifier. Swapping the wake word = swapping one small file (and a model_info.json entry).
• Multi-class models put 2–10 keywords in one classifier file, outputting N+1 softmax probabilities.

Open-source runtimes in the same repository load these files on Android (android/), Web (web/, ONNX in the browser), Python on Windows/Linux/macOS (linux/), and ESP32-S3/P4 (esp32/, TFLite). A Wyoming sidecar connects the same models to Home Assistant. No AccessKey, no cloud inference, no per-device runtime fees — the model file you download is yours to embed.

Worked example: wake word + voice commands on ESP32-S3

To show the whole chain end to end, here is the same model family running on the smallest end of that platform list. The demo combines exactly the two model types this pipeline produces: a single-keyword wake model for “Hallo Lumo”, and a multi-keyword model that recognizes the four follow-up commands. The board is an ESP32-S3 HMI DevKit (dual-core 240 MHz, 8 MB PSRAM, ES7210 mic codec) running ESP-IDF 6.0.1. Say “Hallo Lumo” and the LED confirms the wake. Within a 6-second window, say “Licht rot / blau / grün / weiß” and the light changes to that color, then the device returns to wake mode.

Espressif’s built-in MultiNet supports command words in Chinese and English only. Both models here are trained by our own pipeline on mel features, so the same flow works for any language — German is just what this demo speaks.

• Demo models (German wake + commands): onnx-wakeword/models/de
• ESP32 firmware and examples: onnx-wakeword/esp32

Want keyword models of your own — a single wake word, or a multi-keyword command set in any of five languages? That is what voicute.com trains, and the rest of this article walks through it.

How it works: two small models, time-divided

One device runs two compact INT8 TFLite models on the same core, scheduled so they never compete:

1. Wake model, always on. ESP-SR AFE handles the audio front end (AEC, NS, VAD); the wake model consumes the latest mel window. When it fires, the device switches modes.
2. Command model, only after a wake. For the next 6 seconds a second multi-keyword classifier maps speech to one of the configured commands (here: four colors). After the window — or after a command fires — the device drops back to wake-only.

The gating is what keeps false commands low: the command model is never listening on its own. The wake word carries the false-trigger burden, which is exactly the part the pipeline can optimize (Step 4).

Step 1 · Run the demo model yourself (free)

Both demo models are on GitHub, no signup. The fastest sanity check needs no hardware at all — the Python runtime loads the same model files on a desktop:

git clone https://github.com/voicute/onnx-wakeword.git
cd onnx-wakeword
pip install onnxruntime numpy sounddevice
python linux/infer.py --model-dir ./models/de/

The full embedded path is the tested ESP-IDF example:

cd onnx-wakeword/esp32/examples/esp32s3_hmi_devkit
idf.py set-target esp32s3
python patch_led_strip.py
idf.py build
idf.py -p COM6 flash monitor

The Web runtime runs the same demo models entirely in the browser — useful for verifying a keyword’s sound before any hardware is involved.

Step 2 · Train your own wake word (free)

On voicute.com, type your phrase and pick a language — English, German, French, Japanese or Chinese. Training produces a first-round model (R0): a standard ONNX classifier for desktop/mobile/browser plus the matching INT8 TFLite for ESP32-S3. Nothing is charged at this step.

Step 3 · Test it with your real voice (free)

In the console, play the Basic model against your own microphone, distance and background noise. This is the step that decides everything else, so do it before thinking about payment: say the phrase the way you actually say it, from the distance you will actually use. If recall feels weak, add a few real-voice recordings of the phrase and retrain — recordings are an optional training input, still free.

Step 4 · Decide what you need

After testing, you are in one of four situations:

1. It misses the phrase sometimes → add 5 real-voice recordings and retrain (free). Most recall problems are coverage problems, solved at this step.
2. It wakes reliably but triggers on TV or chatter → run the false-trigger optimization (paid per round) — how it works is the next section.
3. You want voice control, not just a wake word → train a multi-keyword model with 2–10 fixed commands in one file, or skip the wake word entirely and use a command word as the trigger. The demo above is exactly this architecture.
4. It just works → download the model (paid, one-time) and deploy it on any of the runtimes above.

False-trigger control: R0 vs R1, and hard-negative mining

Every model starts as R0 — the first round trained on prepared speech data plus a fixed set of interferences (noise, reverb, volume changes). The real world has far more: TV, music, background chatter, ads. And it is keyword-specific: the same ad clip might set off “Hallo Lumo” yet leave another word’s model completely unmoved. So a fresh R0 with a high false-trigger rate is common to every keyword model — it is not a sign you trained it badly.

R1 is what fixes it, in two steps that mirror the console buttons:

1. Negative scan. Your R0 model listens to ~100 hours of per-language negative audio and flags the clips it reacts to — those clips are, by definition, this model’s hard negatives (up to 10,000 kept). A few minutes.
2. Full retrain. Those clips are added as negative samples to the original training set, and the model is retrained from scratch. Typically 40–80 minutes.

Because the negatives are picked against R0’s actual weaknesses, one round cuts false accepts from hundreds per hour to single digits on average (about 95%) — on our demo’s wake word, measured 319/h → 2.6/h over 24 hours of held-out audio. Both single-keyword and multi-keyword models support this: the four-color command model in the demo can run through the same scan-and-retrain loop. An R2 pipeline exists for brand-new interference sources, but for most keywords one R1 round is enough for daily use. The optimization is paid per round; current prices are on voicute.com.

Step 5 · Validate on your hardware

Console testing proves the model; the device proves the product. Three checks before shipping:

• Real-time budget. Measure Mel + Invoke() and keep it under your audio hop (160 ms in this demo). If it doesn’t fit, shrink the model or process only the latest window.
• False accepts per hour. Record an hour of the real environment — TV, conversation, music — that never goes into training, run it through the firmware, and count triggers. That one number is worth more than any short demo clip.
• Missed wakes at the same threshold. False-accept and recall numbers only mean something together; record both.

FAQ

When is a fixed-keyword model the wrong choice?

When the vocabulary isn’t fixed. Dictation, arbitrary questions, anything open-ended needs full ASR — a KWS model only recognizes what it was trained on, by design.

Does this work fully offline?

Yes. Wake word and commands both run locally. No cloud inference, no runtime account key.

Which platforms and languages work?

Android, Web, Python (Windows/Linux/macOS), ESP32-S3/P4, plus Wyoming for Home Assistant; models exist for English, German, French, Japanese and Chinese.

What is free and what is paid?

Training and console testing are free. Downloading model files and the false-trigger optimization are paid — current prices on voicute.com.

Can unrelated speech trigger a command?

Only within the 6-second post-wake window, and the wake gate blocks most of it. Speech inside the window can still be misclassified today; keyword hard-negative mining is supported in the pipeline for this.