Fourteen Speaker Encoders Heard the Same Voice. Their Error Rates Differed Five-Fold.

A speaker encoder turns a few seconds of speech into a vector, and the distance between two vectors is treated as an answer to "are these the same person." That number gets used for authentication, for voice-cloning pipelines, and increasingly as the yardstick in papers asking whether a synthetic voice preserved someone's identity.

The number is not neutral. It is a reading from an instrument, and I wanted to know how much the instrument was contributing.

So I ran fourteen of them over an identical, frozen list of 11,935 trials.

The corpus is why the question is answerable

Most emotional-speech corpora use different sentences for different emotions. A model can then learn angry vocabulary instead of angry delivery, and you cannot separate them afterwards.

This one can't make that mistake. Eight speakers each recorded the same 1,360 sentences in six phonation states — neutral, happy, angry, scared, shouting, whisper. Lexical content is fixed while phonation varies.

Enrolment is neutral speech. The test is whether the encoder still recognises the person while they shout or whisper. The trial list was frozen in writing before a single model loaded, and every encoder scored the identical list — so every comparison is paired.

The headline

Equal error rate spans 0.047 to 0.233 across the panel — same trials, same audio, nothing varying but the encoder. 29 of 91 pairwise comparisons survive Holm–Bonferroni correction.

If you are choosing an encoder for expressive audio, that five-fold spread is your decision. It is larger than most architectural differences people argue about.

CAMPPlus is last, without exception

Highest error of the fourteen, all 13 of its comparisons surviving family-wise correction — for every one of the eight speakers and every one of the six conditions. Margins run from +0.087 [+0.064, +0.127] against ECAPA-TDNN to +0.185 [+0.150, +0.215] against the lowest-error encoder.

The mechanism is the useful part. Its genuine scores are unremarkable; its impostor scores are the problem. It places different people at cosine 0.28–0.34 where ECAPA-TDNN and ReDimNet place them near 0.034. That is a compressed embedding space — different speakers land close together.

It is visible at neutral speech too (0.031 EER where several ReDimNet checkpoints reach 0.000). Expressive speech doesn't cause it. Expressive speech makes it visible.

A widely used open-source TTS system conditions on CAMPPlus. To be precise: that is a verified fact about its source code, not a measurement of its audio. I measured encoders in isolation on human recordings. Whether that choice degrades output is a separate experiment I have not run.

Parameter count does not order the panel

Spearman ρ(parameters, expressive EER) = +0.135 over fourteen encoders. The 20.8M model ranks 13th of 14. A 4.81M model ranks 1st. Published VoxCeleb1-O position doesn't transfer either — that leaderboard is computed on calm read speech, which is not the condition failing you.

I retracted my claim about the winner, incidentally. It survives as an argmin (selected in 89.4% of speaker resamples, leading all eight leave-one-speaker-out refits) but after correction it is separated from only 7 of 13. The defensible wording is lowest observed error.

A clean pre-registered negative

Spectral denoising does not materially change the measurement: |ΔEER| ≤ 0.039 over 10,765 twinned trials, and whispered speech — where removing aspiration noise was the stated worry — is among the least affected.

Ten retracted claims, published on purpose

• An 11.7 dB spectral effect, asserted with an instrument whose F0 artifact budget is 12.09 dB. Sweeping F0 with the spectral envelope held fixed moved the descriptor further than the claimed effect. It was never a measurement.
• "Shouting does not transfer." Judged on RMS after peak-normalising every clip, so the figure was crest factor, not loudness.
• A tie-break defect in the EER estimator. argmin returns the first minimiser rather than the balanced operating point — invisible at unit weights, common under the integer speaker multiplicities a bootstrap runs at. Fixing it moved one comparison across Holm, 28 → 29 of 91. The correction favoured the paper, which is why it is published rather than quietly kept.

The rule I have now paid for three times: validate the instrument before measuring with it.

Statistics

The trial list is a directed graph on 8 speakers — genuine trials are self-loops, impostor trials the 56 ordered edges. The independent unit is the speaker, on both sides. So the resample is a vertex bootstrap: draw 8 vertices with replacement, take the induced sub-multigraph, weight the impostor edge by the product of multiplicities.

An earlier scheme resampled enrolment speakers only, dropping a speaker as enrollee while leaving them in as impostor. Fixing it widened marginal intervals (width ratio median 1.11, max 1.58) and cost two already-written claims. Paired differences barely moved, because both encoders score the same impostor clips and the dependence cancels in the difference.

B = 20,000, seed fixed, families declared before testing, Holm–Bonferroni at α = 0.05.

The companion result, which is the cheap fix

Enrolling on expressive material instead of neutral-only recovers the loss on all 14 encoders, every one surviving Holm.

The mechanism is the opposite of the intuitive one. Enrolling on a single expressive state helps that state and slightly harms the others — transfer-matrix diagonal median −0.082, off-diagonal median +0.010. Guessing which state a user will be in, and guessing wrong, is worse than enrolling on calm speech.

A pre-committed control ruled out the competing explanation: acoustic diversity alone delivers +0.002 where composition delivers −0.065. It is coverage, not expressiveness. Spread enrolment thinly across states.

What it does not claim

Nothing about any TTS system's output quality. Eight speakers, one studio, one recording chain. Enrolment and test share a session, so absolute error rates are optimistic and not comparable to VoxCeleb — only relative comparisons on identical trials are licensed. Architecture is fully confounded with training corpus; separating them requires training.

The generalisable part

An established metric is not automatically a neutral one. An encoder, an evaluator, a classifier, an LLM judge — any of them can become part of the experiment rather than a window onto it.

Before asking whether a system preserves identity under some transformation, ask whether your instrument has been validated under that transformation. Otherwise you attribute the encoder's behaviour to the speaker, and the number looks perfectly reasonable while you do it.

Paper: Speaker Encoders Disagree About Who You Are When You Shout — Zenodo, CC-BY-4.0.
DOI: 10.5281/zenodo.22158030
Code, data and the full decision record: github.com/Mormolykos/speaker-encoders-disagree
The eval library the analysis checks itself against: pip install spkproof