Technical benchmarks · Multi-Speaker 2.0
Every number, and the configuration behind it
One recording of several people goes in; a clean, labeled track per person comes out. This page is the evaluation — what each metric measures, the scoring configuration applied identically to every system, the results, and where the model stops working.
Benchmarks run August 2026. Comparator versions are named in each section.
Results at a glance
Five measurements. Jump to the one you care about.
01
Diarization — who spoke when
30% lower error than the open-source comparator, ahead in all four domains. Parity with the paid service on telephone and web video.
02
Speaker counting — how many people
About 24% more accurate than the paid service overall, and the margin widens as the cast grows.
03
Separation — how clean the tracks are
32% less unwanted signal than 1.0 on broadcast-style speech; 75% more noise and interference removed.
04
Downstream transcription — do the words survive
4× fewer word errors than the open-source separator, and 30% fewer than 1.0 on overlapped speech.
05
Failure modes — where it stops working
Sustained overlap above roughly 20%, large casts, and the metric caveats that cut against us.
01 — Diarization
Who spoke when
Against pyannote community-1, the strongest freely available diarizer, Multi-Speaker 2.0 makes 30% fewer errors on average and is ahead in all four domains and on all 15 datasets. Against the paid service it replaces, it leads on meetings and broadcast and sits at parity on telephone and web video.
4 / 4
domains ahead of both comparators
15 / 15
datasets ahead of open source
19–38%
per-domain margin vs open source
Results by domain
| Domain | MS 2.0 | community-1 | precision-2 |
|---|---|---|---|
| Meetings, far-field + headset (4) | 12.00 | 17.45 | 13.85 |
| Telephone conversations (5) | 13.81 | 22.34 | 14.23 |
| Broadcast / radio (1) | 12.63 | 20.07 | 16.88 |
| In-the-wild web video (5) | 23.54 | 29.15 | 23.76 |
Diarization error rate %, lower is better. DIHARD3 is deliberately excluded — pyannote trains on it and we do not, so including it would not be like-for-like.
Primary-speaker attribution, and non-English telephone
Recompute error keeping only the speaker who took the floor first at each instant, so all overlapped speech is excluded. The ranking holds: ahead of open source in all four domains, ahead of the paid service in three, with an exact tie on web video.
| CallHome split | Previous | MS 2.0 | Fewer errors |
|---|---|---|---|
| Japanese | 24.29 | 14.44 | 41% |
| Spanish | 21.36 | 15.14 | 29% |
| German | 16.15 | 12.10 | 25% |
| Mandarin | 14.94 | 12.50 | 16% |
The previous diarizer had no conversational telephone speech in its training mix, so this is new domain coverage rather than better generalisation. Our CallHome results use the free TalkBank release, not the LDC edition most published figures use.
How this was scored
No forgiveness collar. Overlapping speech included. Held-out test splits only, at 16 kHz. Diarizer evaluated in isolation — no denoise frontend, no separator in the path. Comparators: pyannote community-1 (open source) and pyannote.ai precision-2, run as pyannoteai-local 2.2.0.
A 250 ms collar and the exclusion of overlapping speech are both still common in published figures. Both remove the hardest frames from the denominator, so numbers scored that way are not comparable to these.
02 — Speaker counting
How many people are talking
Across 1,172 recordings we count speakers about 24% more accurately than the paid service. The useful part is the shape: below three speakers we are level with open source, and from three speakers up we lead both by roughly 18%.
24%
more accurate than the paid service, all recordings
18%
better than both on 3+ speakers
1,172
recordings scored
Results by speaker count
| Speakers present | Recordings | MS 2.0 | community-1 | precision-2 |
|---|---|---|---|---|
| 1–2 | 387 | 6.07 | 5.68 | 12.53 |
| 3–4 | 335 | 21.47 | 27.04 | 30.90 |
| 5–6 | 205 | 33.38 | 40.47 | 37.63 |
| 7 or more | 245 | 24.77 | 29.90 | 26.59 |
| All recordings | 1,172 | 19.16 | 22.93 | 25.11 |
Speaker-count error %, weighted per recording. A four-speaker file where a system finds three scores 25%. The 1–2 row is published rather than hidden: open source edges us there, which is why the headline comparator for this metric is the paid service.
03 — Separation
How clean the tracks are
The separator leaves 32% less unwanted signal than 1.0 on broadcast-style speech and 17% less on clean read speech — the gain is roughly twice as large on the harder material. With noise in the mix, the full 2.0 path removes 75% more noise and interference than 1.0's.
32%
less unwanted signal, broadcast-style speech
75%
more noise and interference removed
3.9
predicted listening quality, up from 3.0
Results by benchmark
| Benchmark | Character of the audio | Less unwanted signal vs 1.0 |
|---|---|---|
| LRS2-2Mix | Broadcast-style speech, real conditions | 32% |
| Libri2Mix | Clean read speech | 17% |
Two-speaker benchmarks, 3,000 utterances each, separator in isolation with oracle permutation. Reported as a reduction in unwanted signal rather than an absolute score, because the underlying metric is logarithmic.
A note on perceptual quality
Predicted listening quality on the separated tracks reaches 3.9 on a 5-point scale, close to a full point above the previous model, and the background sub-score moves from 3.2 to 4.1 — roughly the difference between background noise you can clearly hear behind the isolated voice and a track that is essentially clean.
This is predicted quality, a model of human ratings, not a human listening panel. It is also referenceless: a track that confidently contains the wrong speaker's words still scores well. We report it alongside the signal-level results, which catch exactly that error, and never on its own.
04 — Downstream transcription
Whether the words survive
The most direct test: transcribe every separated track and count the errors. Multi-Speaker 2.0 produces 4× fewer word errors than the open-source separator, and 30% fewer than 1.0 on overlapped speech. Up to 20% overlap, accuracy stays within about a point of non-overlapped speech.
4×
fewer errors than the open-source separator
30%
fewer errors than 1.0 on overlapped speech
20%
overlap before accuracy starts to fall
System comparison
| System | cpWER % |
|---|---|
| AudioShake Multi-Speaker 2.0 | 9.17 |
| Multi-Speaker 1.0 separator | 12.89 |
| Open-source speech separator | 37.75 |
Lower is better. LibriCSS, Whisper large-v3, cpWER via the open MeetEval toolkit, identical pipeline for every system. The open-source figure uses a public checkpoint outside its training domain — an accurate picture of what you get off the shelf, not a matched-training claim.
How accuracy holds as overlap increases
“20% overlap” means a fifth of total speech time has two or more people talking at once. It is a property of the recording, not of any single interruption.
| LibriCSS condition | cpWER % |
|---|---|
| 0S — no overlap, short silences | 5.81 |
| 0L — no overlap, long silences | 10.89 |
| 10% overlap | 7.04 |
| 20% overlap | 7.18 |
| 30% overlap | 13.66 |
| 40% overlap | 10.03 |
Conditions differ in more than overlap ratio — the two 0% conditions differ in silence length — so this is not a monotonic curve. See failure modes for the 30% and 40% conditions. No denoise frontend was enabled in these runs, so they measure the separator and diarizer rather than the complete shipping path.
05 — Structured output
What a job returns
One audio track per speaker at the input sample rate, an overlap-aware timeline of who spoke when, and two confidence scores — returned per 20 ms frame and as a whole-file value weighted by speech activity, so silence cannot inflate them.
assignment_confidence
Did this speech go to the right speaker?
The label-quality signal. Low values mean the audio may be clean but the identity behind it is unreliable — fine for cleanup, risky for anything routed by speaker.
separation_confidence
Did the overlapping voices come apart?
Most informative where people actually talk over each other. Low values mean a track probably carries bleed from another voice — fine for talk-time analytics, not for voice cloning or per-speaker ASR.
The two scores catch different failures and one number cannot tell them apart — audio can be cleanly separated but attributed to the wrong person, or correctly attributed and still carry bleed. If you need a single value, take the lower of the two.
Reading the two scores together
| Separation | Assignment | What it means |
|---|---|---|
| low | low | Both collapsed, usually on overlapping speech. Neither the audio nor the labels are trustworthy — drop, or route to review. |
| high | high | The strongest “trust this” signal. Also what you see where there is no speech at all. |
| high | low | Little to separate, but speaker identity is unreliable. Fine for cleanup; risky for anything routed by speaker. |
| low | high | Labels are trustworthy but the audio likely carries bleed. Fine for talk-time analytics; not for voice cloning or per-speaker ASR. |
We deliberately do not publish recommended thresholds. Score distributions vary across content types and the right cutoff depends on the application, so treat them as a relative ranking signal rather than a gate. For guidance on using them in code, see the API documentation.
06 — FAILURE MODES
Where it stops working
The point of publishing these is that you can predict which of your material will be hard before you run it.
01
Sustained overlap above roughly 20%
Beyond about a fifth of speech time spent in crosstalk, separation quality falls off and transcription error rises with it — the 30% and 40% LibriCSS conditions are where this shows. Most real conversation sits below this threshold; sustained multi-way talking does not.
02
Large casts
There is no fixed ceiling on speaker count, but counting accuracy degrades as the number of people grows — error roughly triples between two-speaker and five-to-six-speaker recordings. Passing a known speaker count, where you have one, removes this source of error.
03
Correct labels, contaminated audio — and the reverse
Two independent failure modes. Audio can be cleanly separated but attributed to the wrong person, or correctly attributed and still carry bleed from another voice. The two confidence scores are designed to tell these apart; one number cannot.
04
Metric caveats that cut against us
Predicted listening quality is referenceless and will score a confidently wrong speaker highly. The open-source separator comparison uses a public checkpoint outside its training domain. Several diarization corpora are in-domain for our training mix, as they are for the systems we compare against.
07 — DEFINITIONS
What each number measures
Every metric on this page, defined in the form we score it.
DER
Diarization error rate
The share of speech time attributed to the wrong speaker, missed entirely, or invented. Lower is better. We score with no forgiveness collar and overlapping speech included.
cpWER
Concatenated minimum-permutation word error rate
Transcribe every output track, then score under the speaker assignment that minimises total error. It penalises both transcription mistakes and speaker-attribution mistakes, which is why it is the closest proxy for what a real pipeline experiences.
Speaker-count error
How often the number of people is wrong
The absolute difference between speakers found and speakers present, divided by speakers present, averaged per recording. A four-speaker file where a system finds three scores 25%.
Overlap ratio
What “20% overlap” means
The share of total speech time in which two or more people are talking simultaneously. At 20%, a fifth of the speech in the recording is crosstalk. It is a property of the recording, not of any single interruption.
Get in touch.