How to Build Offline Voice Control with Custom Wake Words and Multi-Keyword ONNX Models

A small voice-control system does not always need cloud ASR or a large speech model. A custom wake word can start the interaction, and a compact multi-keyword model can recognize a fixed set of commands locally.

The two-stage voice-control architecture

The system in the demo uses two different keyword-spotting stages:

1. Wake word detection: a small model listens continuously for a custom phrase.
2. Command detection: after the wake word is detected, the application opens a short listening window and runs a multi-keyword model for fixed commands.

For example, a German device could listen for a custom wake phrase and then recognize commands such as “Licht an”, “Licht aus”, “lauter” or “leiser”. The application maps each output class to an action. There is no need to send the audio to a cloud speech-recognition service.

Why use a multi-keyword model?

Training one model for every command is simple at first, but it becomes difficult to manage as the command set grows. A multi-keyword model shares one acoustic backbone and produces several keyword outputs in one inference pass.

This approach is useful for fixed voice control because the application does not need to understand arbitrary sentences. It only needs to distinguish a known set of commands and ignore everything else. The model can therefore stay small enough for embedded and offline use.

R0 and R1: reducing false triggers

The first model is the R0 baseline. It is trained to recognize the target wake word or command phrase, but it may still react to similar speech, music or environmental noise.

R1 is a further training round rather than a separate runtime model. The current model is used to scan difficult negative audio. Clips that incorrectly trigger the model are collected as hard negatives and added to the next training round. The resulting model learns not only what the target phrase sounds like, but also which confusing sounds should be rejected.

This is especially important for a wake word that runs for long periods. A model that works in a clean microphone test can still produce too many false activations in a real room, vehicle or workshop. Online testing is useful for the first check; hard-negative scanning and real-device testing are needed before a product deployment.

Why run this locally?

• Low latency: detection does not wait for a network request.
• Privacy: the microphone stream can remain on the device.
• Small models: keyword spotting needs much less computation than continuous ASR.
• Offline reliability: the device can keep working without an internet connection.
• Simple deployment: a fixed command can map directly to a GPIO action, application callback or local service.

The two-stage design also avoids running the command model as an always-on conversational recognizer. The wake word is the entry point; the command model is active only during the command window.

ONNX and TFLite deployment

The same training idea can be delivered in different model formats for different runtimes:

• ONNX: suitable for ONNX Runtime on Android, Windows, macOS, Linux, Web browsers and Raspberry Pi-class computers.
• INT8 TFLite: suitable for microcontroller runtimes such as TFLite Micro on resource-constrained devices.

The file format is not the entire integration. The audio sample rate, feature extraction, tensor shape, frame size, threshold and post-processing logic must match the model. A model trained for one input pipeline should be validated with the same preprocessing on the target device.

The demo uses the open-source onnx-wakeword inference engine. The runtime is designed to make the same keyword-spotting workflow easier to reuse across desktop, mobile, web and embedded projects.

How to reproduce the workflow

1. Choose a wake phrase that is long and distinctive enough to separate from normal speech.
2. Generate a baseline wake word model and test its pronunciation in the browser.
3. Create a multi-keyword model for the fixed commands that follow the wake word.
4. Connect the wake-word output to a short command-listening window.
5. Export the format required by the target runtime: ONNX for general CPU/browser platforms or INT8 TFLite for suitable embedded targets.
6. Run the model on real hardware and collect false-trigger examples.
7. Use the hard negatives for an R1 retraining round, then test again at the target threshold.

When this architecture is a good fit

This design works well for smart-home controls, small robots, kiosks, desktop utilities, accessibility buttons, toys and other devices with a known command vocabulary. It is not intended to replace open-ended speech recognition. If users need to say arbitrary sentences, an ASR or speech-to-intent system is a better fit.

For a small set of local actions, however, a wake word plus multi-keyword KWS model can be easier to deploy, cheaper to run and more private than a full voice assistant stack.