Same Data, Same Recipe, 5× Different: Training Variance in Wake Word Models

Two retrains of the same wake word model — identical dataset, identical recipe, only the random seed different — landed at 2.74 and 14.20 false triggers per hour. That is a 5× difference from nothing but initialization and batch order. This post shows our production numbers and the workaround we shipped: every retrain now produces 5 candidate models, and you pick one.

The numbers first

We run an automated false-trigger reduction loop: a freshly trained model scans public speech corpora for clips it wrongly scores high, those clips become hard negatives, and the model is fully retrained (this loop deserves its own write-up; a Chinese version lives in our article index). Retrains used to produce a single model. Now each retrain trains 5 candidates with different seeds, and every candidate is evaluated on the same held-out corpus — false triggers per hour and recall against the previous version.

Here are seven production chains across five languages, each row one keyword, draws sorted ascending. Anonymized because they are real users' keywords:

Every single retrain beat its R0 baseline by 90%+. But the spread between candidates is consistently 2–5×, in every language. If you train one model, you accept a lottery ticket.

Where the variance comes from

Wake word models for edge deployment are tiny — ours are around 30K parameters. Small models, small parameter count, and a loss surface with many distinct minima mean the seed decides which minimum you land in. Three things amplify it in our setting:

• Hard negatives are a small, high-impact subset. A retrain mixes original positives, generic negatives, and a few thousand mined clips. Those mined clips carry most of the gradient signal that fixes false triggers, and small changes in batch order change how strongly each clip shapes the decision boundary.
• The metric is heavy-tailed. False triggers per hour is dominated by a handful of sounds — one TV catchphrase can be worth 20 triggers an hour. Whether a candidate happens to sit on the right side of that one boundary moves the number by multiples, not percentages.
• Nothing averages out at inference. We evaluated all candidates on the same corpus with the same pipeline, so the spread is real model difference, not measurement noise.

The fixes that don't work

Picking a lucky seed. Find a seed that gives 2.7 on keyword A, reuse it on keyword B — no guarantee. The mapping from seed to outcome is not stable across datasets and keywords, so "the good seed" doesn't transfer. We checked.

Ensembling. Average the scores of 5 models and the variance mostly vanishes. But these models deploy on ESP32-S3 and Android, where the budget is one 100 KB model and a few milliseconds of inference. Ensembling multiplies memory and compute by 5 for a device class that cannot afford it.

What we shipped instead

One retrain now produces 5 candidates. All 5 are evaluated on the same held-out corpus, and the console shows every candidate with both metrics — false triggers per hour and recall change — then the user picks which one to download.

The honest answer to "why not let the server pick the best one" is the interesting part. The server does rank them — on public corpora. But our evaluation set is generic speech; the user's living room is not. Which candidate is best depends on what sounds exist in the deployment environment, and on how the user weighs false triggers against recall. One real example from the table above: the French keyword had a candidate at 7.25 per hour that looked great until you noticed its recall dropped 7 points against R0 — for a user who answers the device rarely, that tradeoff might be fine; for a wake word used all day, it isn't. Those are user decisions, not server decisions.

Practical advice if you train your own

1. Never trust a single training run's false-trigger number. Re-train with a different seed before comparing two recipes — the seed may explain more of the difference than your change did.
2. Evaluate candidates on a fixed held-out set, and always report recall next to false triggers. A candidate that only wins on false triggers may be quietly going deaf.
3. If your deployment target can't run ensembles, train N candidates offline and select one — that is the poor man's ensemble, and it costs training time instead of runtime.
4. Watch the tail, not just the mean: the worst candidate in the German row (14.20) was 5× the best one. Selection is worth more than recipe tweaking once the recipe is sane.

Limitations

The 5 candidates share one dataset — this reduces seed variance, not data variance. Evaluation corpora are still generic public speech, and 20–30 hours of held-out audio gives coarse resolution at single-digit trigger rates. And 5× training compute per retrain is real cost; we pay it because a 100 KB model is cheap to train but the user's patience for a misfiring device is not.