Never put zero-tolerance data behind speech recognition

Voice UXPost-mortemASR
Updated 2026-09-07

If a step in your product cannot tolerate a single wrong character, do not make speech the only way through it. That is the whole lesson, and it cost a customer relationship to learn.

What we were building

A customer wanted a voice device for a school. The core flow was simple. A student speaks their student ID at the device, the system pulls up that student's learning records and the teacher's comments, and reads them back. No screen, no app, no login. Walk up and talk.

The logic was clean and the experience sounded good in a meeting room. It broke on step one.

Recognising intent and recognising digits are different problems

Speech recognition is very good at working out what someone meant. It is meaningfully worse at capturing an exact sequence of digits, and a student ID here was eight digits long. One wrong digit and there is no matching record, so the flow stops dead. There is no partial credit.

Three failure modes showed up in testing, and they stacked.

Accent and near-homophones

In Mandarin, 三 (three) and 四 (four) confuse easily, as do 一 (one) and 七 (seven). Regional accents widen the gap. English has the same class of problem with fifteen and fifty, or M and N over a poor channel. Any language has pairs like this, and an ID string is exactly where they hurt most.

Weak network

On a slow connection the recognition round trip stretches out. The user waits, and then the answer that finally arrives is still wrong. Latency does not just annoy people, it removes the sense that retrying will help.

How children actually speak

Children do not enunciate on cue, and the distance between mouth and microphone changes constantly because they move. Accuracy that looked fine in a quiet adult test drops noticeably in the real room.

No screen turned a bad step into a dead end

On a phone or a tablet, a misrecognition is recoverable. The user sees 38472619 on screen, notices the 7 should be a 1, and fixes it. On a screenless device the user never learns what the machine heard. All they get is "no record found". So they say it again, it fails again, and there is no path from the failure back to success.

This is worth stating plainly because it is easy to miss during specification. Removing the screen does not just remove display. It removes the correction loop.

What we changed

The student ID is now bound to an account during registration on a phone. On the device, speech handles identity confirmation and ordinary conversation, and never has to carry an exact digit string. Voice does what voice is good at, which is fuzzy, error-tolerant input where being approximately right is still useful.

We rebuilt it and went back to the customer. The delay was ours and their frustration was reasonable, and there is not much to say in defence of it beyond fixing the design.

The question we ask now

At this step, if the model gets it wrong, will the user notice, and can they correct it?

If yes, voice is fine. If no, do not put that step behind voice at all. Bind it earlier, move it to another channel, or add a confirmation the user can actually act on.

"An AI can do this" and "an AI can do this reliably in this specific situation" are separate claims. Keeping them separate is the most basic discipline in voice hardware, and it is usually cheaper to apply it at the specification stage than after tooling.

Related

• AI Recorder Card — open Bluetooth protocol, your own transcription stack
• 4G AI Learning Companion — school knowledge base and parent dashboard