How to measure word error rate on your own audio

Every speech model publishes an accuracy figure, and none of them is about your audio. The benchmarks are read speech from audiobooks, recorded cleanly, by speakers who are not your colleagues and do not use your vocabulary. If the decision in front of you is which engine to use for your recordings, you have to measure it yourself.

The WER calculation itself is easy. Everything around it is where the trouble is, and a transcription accuracy test that skips those parts produces a number you cannot act on.

What word error rate actually counts

Line the transcript up against the reference, count the edits needed to turn one into the other, divide by the length of the reference:

WER = (substitutions + insertions + deletions) / words in reference

That is edit distance, at the word level rather than the character level. Two consequences people find surprising. It can exceed 100%, if the system inserts more words than the reference contains. And a “10% error rate” is ten errors per hundred reference words, not a claim that 90% of the transcript is right.

The three error types are worth separating, because they say different things. Here is the breakdown from three recordings I measured:

Recording	Words	Substitutions	Insertions	Deletions	WER
Deploy	73	12	5	0	23.3%
Models	70	4	1	0	7.1%
Support	57	2	1	0	5.3%

Zero deletions across all three. The model never dropped a word — it substituted something wrong or added something extra. That pattern is worth knowing: modern recognisers do not go quiet when they are unsure, they guess. A transcript with no obvious gaps is not a transcript with no errors.

The hard part is the reference

Edit distance needs something to compare against, and that something has to be exactly right. Three ways to get it, each with a cost.

Transcribe by hand. The most honest and the most expensive. Ten minutes of audio takes about an hour if you are careful. Your own typos become the model’s errors.

Use a published dataset. LibriSpeech and its cousins come with references already. They also come with the same read-audiobook audio that made the published numbers useless for your case, so this tells you whether your setup is wired up correctly, not how it will do on your meetings.

Synthesise the speech from a text you already have. This is what I use for comparisons:

say -o reference.aiff -f reference.txt

The reference transcript is reference.txt, exactly, by construction. There is nothing to label and nothing to get wrong. The price is that synthetic speech has no room acoustics, no microphone colouring, no accent and no overlapping speakers, so absolute error rates come out optimistic. It is the right tool for comparing conditions — this codec against that one, vocabulary on against off — and the wrong tool for predicting what your real recordings will score.

The same transcript scored 23% and 29%

Here is the result that should make you suspicious of every published WER, including mine. One transcript, one reference, five ways of normalising before counting:

Normalisation	Errors	WER
Raw split() on whitespace	21	28.8%
Lowercased	20	27.4%
Punctuation stripped	19	26.0%
Numbers written out (10% → ten percent)	17	23.3%
Hyphens split (one-line → one line)	17	23.3%

Nothing about the transcription changed between those rows. Five and a half percentage points came out of the counting rules alone. “Ten percent” against “10%” is not a recognition error, it is a formatting choice, and whether you count it decides a fifth of your headline number.

So: a WER without its normalisation rules is not a measurement. When you publish one, publish the rules. When you read one, look for them, and be sceptical when they are missing.

The script

Thirty lines, no dependencies. It does the alignment and reports the breakdown:

import re, sys

def norm(text):
    text = text.lower().replace('%', ' percent')
    return re.sub(r"[^a-z0-9' ]", ' ', text).split()

def score(ref, hyp):
    r, h = norm(ref), norm(hyp)
    n, m = len(r), len(h)
    d = [[0] * (m + 1) for _ in range(n + 1)]
    for i in range(n + 1): d[i][0] = i
    for j in range(m + 1): d[0][j] = j
    for i in range(1, n + 1):
        for j in range(1, m + 1):
            d[i][j] = min(d[i - 1][j] + 1,
                          d[i][j - 1] + 1,
                          d[i - 1][j - 1] + (r[i - 1] != h[j - 1]))
    return d[n][m], n

errors, words = score(open(sys.argv[1]).read(), open(sys.argv[2]).read())
print(f"{errors} errors in {words} reference words — WER {errors / words:.1%}")

Run it as python3 wer.py reference.txt transcript.txt. If you would rather not maintain your own, jiwer does the same job with configurable normalisation built in.

Now the point of the previous section, made against this very script: on the file from the table above it prints 24.7%, not 23.3%. Its norm lowercases, strips punctuation and turns % into percent, but leaves 10 as digits, so 10 percent against ten percent costs it one error. Getting to 23.3% means also spelling numbers out — which is a decision about what counts as a mistake, not a bug in the code.

Adjust norm to match the decisions you want to make, then write down what you chose. That written-down list is as much a part of the result as the percentage.

Four traps, all of which caught me

Your sample is too small. This is the first thing to fix if you want to measure transcription accuracy in a way that survives scrutiny: seventy words means each error moves the figure by 1.4 points. Two unlucky words look like a real difference between engines. For anything you will act on, measure tens of minutes across several speakers.

Post-processing runs after recognition. If the app has a custom vocabulary or text clean-up, you are scoring the pipeline, not the model. I have a vocabulary feature that rewrites words after the fact; a measurement that forgets it is measuring something else.

Results get cached. Transcribing the same file twice may return the stored result rather than a fresh run — sensible behaviour, and a trap when you have just changed a setting and expect a different answer. Copy the file to a new name between runs.

Truncated output looks like empty output. Testing whether a flag suppressed some behaviour, I piped the output into head, and the broken pipe cut the process off before it printed. Two flags appeared to fix a problem they did not touch, and I nearly published it. Write to a file, and repeat every run.

What WER will not tell you

It counts words, and words are not equal. From a measurement of compression: an Opus file scored 3.8% against an MP3’s 5.4%, which reads as a clear win — until you read the transcripts. Opus turned the name Sarah into Third, so the meeting record lost the person who raised the concern. MP3’s extra errors were “currently” for “currency” and “a roll-out” for “the rollout”.

The better number produced the worse transcript. Whenever a comparison is close, read the actual errors before believing the ranking; the metric has no idea which words carried the meaning.

---

Two measurements made with this method, if you want the numbers rather than the recipe: what compression costs a transcript and a custom vocabulary taking a folder from 12.5% to 4.0%. And the published per-engine figures, with the caveat above attached, are in Whisper vs Parakeet vs SenseVoice.

Questions people ask

What is word error rate?

The count of substitutions, insertions and deletions needed to turn the transcript into the reference, divided by the number of words in the reference. It is edit distance at the word level. A 10% word error rate means one error for every ten words of the original, and it can exceed 100% if the system inserts more than it gets right.

How do I measure word error rate on my own recordings?

You need a reference transcript that is exactly right, then edit distance against it. Getting the reference is the hard part: either transcribe by hand, or synthesise speech from a text you already have so the reference is known by construction. The arithmetic afterwards is thirty lines of Python.

Why do published WER numbers disagree with each other?

Partly different audio, but also different normalisation. Scoring the same transcript against the same reference, I got 28.8% with a naive split, and 23.3% after lowercasing, stripping punctuation and writing numbers as words. Nothing about the transcription changed — only the rules for counting.

Is a lower word error rate always better?

No, and this is the trap the metric hides. In one measurement, an Opus file scored 3.8% against MP3's 5.4% — better on paper — but its single extra error deleted a person's name from a meeting transcript, while MP3's errors were function words. WER counts words, not what they cost you.

How much audio do I need for a meaningful measurement?

More than I used. A 70-word passage puts each error at 1.4 percentage points, so a couple of unlucky words move the figure more than a real difference between engines would. For comparing two systems on a decision that matters, aim for tens of minutes across several speakers and conditions.

Can I use synthesised speech to test transcription accuracy?

For comparing variants, yes — it is the only way to get a reference that is exactly right without hand-labelling. But synthetic speech has no room, no microphone and no accent, so the absolute error rates come out optimistic. Treat those numbers as a comparison between conditions, not as what your real recordings will score.