Auditioning a Voice

Speech-to-text has a standard report card: word error rate. The fraction of words a model gets wrong. It's a clean number, it's easy to compare, and for our product it turned out to be almost useless. A transcription can score beautifully on word error rate and still fail at the one thing we actually need it to do.

So when we sat down to choose the engine that would hear every session our customers record, we stopped treating it like a benchmark and started treating it like a hiring decision. You don't hire someone on their GPA. You hire them on whether they can do the real job — and, just as important, on how they behave when they're wrong.

Why accuracy lies

Imagine a fifteen-minute appointment. A model transcribes 98% of the words perfectly. Excellent score. But the 2% it dropped happened to be "I'm allergic to anything with lanolin" and "my interview's on Thursday." Those two sentences were the entire reason the recording was worth making. Everything else was weather and small talk.

Word error rate treats every word as equally valuable. Our product does the opposite: it cares enormously about a handful of load-bearing details and barely at all about the filler around them. A metric that averages over all words is, structurally, blind to the thing we're optimizing for. So we threw out the leaderboard and wrote a different set of questions.

The four questions that actually mattered

1. Does it catch the detail that matters?

For each test recording, we identified the single fact a great pro would want waiting for them at the next visit — a color formula, an adhesive sensitivity, a "growing it out for the wedding." Then we asked, bluntly: did the model preserve that? A transcript that nails the small talk and drops the allergy fails, no matter how high it scores. This one question reordered our rankings more than any accuracy figure.

2. Does it know who is talking?

This is where we saw the most dangerous failures. We ran a recording of a young client who came in with his dad — two voices, one chair. Several models collapsed the two speakers into one. And when that happens, something genuinely harmful occurs downstream: the dad's life — his job, his travel, his opinions — gets attributed to the son and written into the son's permanent profile. Next visit, the pro is greeted with facts about a person who was never their client. A misspelled word is a typo. A misattributed speaker is a corrupted memory about the wrong human being. We weighted this failure mode the heaviest of all.

3. Does it make things up?

We watched one tool confidently invent a person's name that was never said. Another fabricated a detail out of nothing. This is the cardinal sin for us — a confident lie is far worse than an honest blank. A model that returns "I didn't catch that" is a model we can build around. A model that smoothly invents plausible content is a model that will, sooner or later, write fiction into someone's history. We'd rather work with the one that admits uncertainty.

4. Does it survive the real world?

Real audio isn't a podcast. We deliberately tested accented and non-native English, and names no dictionary contains — a Dutch street name, a specific product, a surname from anywhere. One model garbled a place name into nonsense; another reproduced it exactly. In a shop full of regulars whose names and lives are the product, getting the proper nouns right isn't a nicety. It's the job.

The shape of the test

The method that fell out of this was less a benchmark and more an audition. A small, deliberately adversarial set of recordings, each built to provoke a specific failure: the multi-speaker trap, the buried critical detail, the accented long-form, the recording where almost nothing is said. Then we judged each candidate not on an average, but on a question: did it pass the test that would actually break it?

You learn far more about a model from the one recording designed to break it than from a thousand recordings it was always going to get right.

It also kept us honest about cost and speed, which a pure-accuracy lens ignores entirely. The "best" transcript in the world is the wrong choice if it's too slow to feel responsive or too expensive to run on every session. Hiring is always a trade-off between the ideal candidate and the one you can actually afford to keep on staff.

The most honest part of any evaluation

Here's the section we're proudest of, and it's the one that admits failure. Our test recordings were good, but they were clean — sourced from decent audio with real conversation. We wrote down, explicitly, the thing our eval did not cover: a real shop floor, with clipper noise, music, and long stretches where nobody is saying anything useful. We literally noted: re-test on real audio before declaring victory.

Months later, that exact gap came back to bite us — a transcription model hallucinating an entire session out of microphone noise, a failure our clean test set was structurally incapable of catching. (We wrote up that whole saga in The Sound of Nothing.) The eval didn't prevent the bug. But because we'd written down what it didn't test, we knew exactly where to look when the bug appeared.

The most valuable part of an evaluation isn't the scores. It's the list of questions you couldn't answer yet — written down honestly, so future-you knows precisely which corners are still dark.

The model is half the answer

One last lesson: even a great hire needs good process around them. The best model we tested could still leak the wrong speaker's monologue into a profile if we let the raw transcript flow straight through. The fix wasn't a better model — it was a discipline applied after transcription: a dedicated pass that scrubs and attributes before anything is written down. The model gives you words. Turning words into a memory you can trust is a system, not a single API call.

So we didn't pick the engine with the best number. We picked the one that failed the least dangerously, told the truth when it was unsure, and got the details that were the whole point — and then we built the guardrails around it that no single model can give you. That's the difference between transcribing a conversation and remembering one.