Technical benchmarks · Multi-Speaker 2.0
Every number, and the configuration behind it
One recording of several people goes in; a clean, labeled track per person comes out. This page is the evaluation — what each metric measures, the scoring configuration applied identically to every system, the results, and where the model stops working.
Benchmarks run August 2026. Comparator versions are named in each section.
Draft — not for publication
We are broadening the comparison set so this page reads as standard open-source benchmarking rather than a single-vendor callout. Columns and rows marked pending are awaiting benchmark runs.
Diarization and speaker counting: NeMo Sortformer, DiariZen. Separation and transcription: MossFormer2, SepFormer. The cpWER runs should hold one ASR constant across every front-end, so the table measures separation quality rather than ASR choice.
Every result gets published as run, including any where we do not lead.
Results at a glance
Five measurements. Jump to the one you care about.
01
Diarization — who spoke when
30% lower error than pyannote community-1, ahead in all four domains and all 15 datasets.
02
Speaker counting — how many people
16% more accurate than the open-source comparator overall, and the margin widens as the cast grows.
03
Separation — how clean the tracks are
32% less unwanted signal than 1.0 on broadcast-style speech; 75% more noise and interference removed.
04
Downstream transcription — do the words survive
4× fewer word errors than TF-Locoformer, and 30% fewer than 1.0 on overlapped speech.
05
Failure modes — where it stops working
Sustained overlap above roughly 20%, large casts, and the metric caveats that cut against us.
01 — Diarization
Who spoke when
Against pyannote community-1, Multi-Speaker 2.0 makes 30% fewer errors on average and is ahead in all four domains and on all 15 of 15 datasets.
4 / 4
domains ahead of open source
15 / 15
datasets ahead of open source
19–38%
per-domain margin vs open source
Results by domain
| Domain | MS 2.0 | community-1 | Sortformer (pending) | DiariZen (pending) |
|---|---|---|---|---|
| Meetings, far-field + headset (4) | 12.00 | 17.45 | — | — |
| Telephone conversations (5) | 13.81 | 22.34 | — | — |
| Broadcast / radio (1) | 12.63 | 20.07 | — | — |
| In-the-wild web video (5) | 23.54 | 29.15 | — | — |
Diarization error rate %, lower is better. Sortformer and DiariZen columns are awaiting runs. DIHARD3 is deliberately excluded — pyannote trains on it and we do not, so including it would not be like-for-like.
Primary-speaker attribution, and non-English telephone
Recompute error keeping only the speaker who took the floor first at each instant, so all overlapped speech is excluded. The ranking holds: we stay ahead of open source in all four domains.
| CallHome split | Previous | MS 2.0 | Fewer errors |
|---|---|---|---|
| Japanese | 24.29 | 14.44 | 41% |
| Spanish | 21.36 | 15.14 | 29% |
| German | 16.15 | 12.10 | 25% |
| Mandarin | 14.94 | 12.50 | 16% |
The previous diarizer had no conversational telephone speech in its training mix, so this is new domain coverage rather than better generalisation. Our CallHome results use the free TalkBank release, not the LDC edition most published figures use.
How this was scored
No forgiveness collar. Overlapping speech included. Held-out test splits only, at 16 kHz. Diarizer evaluated in isolation — no denoise frontend, no separator in the path. Comparators: pyannote community-1, with NeMo Sortformer and DiariZen runs still to come.
A 250 ms collar and the exclusion of overlapping speech are both still common in published figures. Both remove the hardest frames from the denominator, so numbers scored that way are not comparable to these.
02 — Speaker counting
How many people are talking
Across 1,172 recordings we count speakers about 16% more accurately than pyannote community-1. The useful part is the shape: below three speakers the two systems are level, and from three speakers up our margin widens to about 18%.
16%
more accurate than open source, all recordings
18%
better than open source on 3+ speakers
1,172
recordings scored
Results by speaker count
| Speakers present | Recordings | MS 2.0 | community-1 | Sortformer (pending) | DiariZen (pending) |
|---|---|---|---|---|---|
| 1–2 | 387 | 6.07 | 5.68 | — | — |
| 3–4 | 335 | 21.47 | 27.04 | — | — |
| 5–6 | 205 | 33.38 | 40.47 | — | — |
| 7 or more | 245 | 24.77 | 29.90 | — | — |
| All recordings | 1,172 | 19.16 | 22.93 | — | — |
Speaker-count error %, weighted per recording. A four-speaker file where a system finds three scores 25%. Sortformer and DiariZen columns are awaiting runs. The 1–2 row is published rather than hidden: community-1 edges us there by four tenths of a point, and the margin turns in our favour from three speakers up.
03 — Separation
How clean the tracks are
The separator leaves 32% less unwanted signal than 1.0 on broadcast-style speech and 17% less on clean read speech — the gain is roughly twice as large on the harder material. With noise in the mix, the full 2.0 path removes 75% more noise and interference than 1.0's.
32%
less unwanted signal, broadcast-style speech
75%
more noise and interference removed
3.9
predicted listening quality, up from 3.0
Results by benchmark
| Benchmark | Character of the audio | Less unwanted signal vs 1.0 | Vs open source (pending) |
|---|---|---|---|
| LRS2-2Mix | Broadcast-style speech, real conditions | 32% | — |
| Libri2Mix | Clean read speech | 17% | — |
Two-speaker benchmarks, 3,000 utterances each, separator in isolation with oracle permutation. Reported as a reduction in unwanted signal rather than an absolute score, because the underlying metric is logarithmic. The open-source comparison column is awaiting MossFormer2 and SepFormer runs.
A note on perceptual quality
Predicted listening quality on the separated tracks reaches 3.9 on a 5-point scale, close to a full point above the previous model, and the background sub-score moves from 3.2 to 4.1 — roughly the difference between background noise you can clearly hear behind the isolated voice and a track that is essentially clean.
This is predicted quality, a model of human ratings, not a human listening panel. It is also referenceless: a track that confidently contains the wrong speaker's words still scores well. We report it alongside the signal-level results, which catch exactly that error, and never on its own.
04 — Downstream transcription
Whether the words survive
The most direct test: transcribe every separated track and count the errors. Multi-Speaker 2.0 produces 4× fewer word errors than TF-Locoformer, and 30% fewer than 1.0 on overlapped speech. Up to 20% overlap, accuracy stays within about a point of non-overlapped speech.
4×
fewer word errors than TF-Locoformer
30%
fewer errors than 1.0 on overlapped speech
20%
overlap before accuracy starts to fall
System comparison
| Separation front-end | cpWER % |
|---|---|
| AudioShake Multi-Speaker 2.0 | 9.17 |
| Multi-Speaker 1.0 separator | 12.89 |
| TF-Locoformer (open source) | 37.75 |
| MossFormer2 (open source, pending) | — |
| SepFormer (open source, pending) | — |
Lower is better. LibriCSS, Whisper large-v3, cpWER via the open MeetEval toolkit, identical pipeline for every system. The open-source figure uses a public checkpoint outside its training domain — an accurate picture of what you get off the shelf, not a matched-training claim.
How accuracy holds as overlap increases
“20% overlap” means a fifth of total speech time has two or more people talking at once. It is a property of the recording, not of any single interruption.
| LibriCSS condition | cpWER % |
|---|---|
| 0S — no overlap, short silences | 5.81 |
| 0L — no overlap, long silences | 10.89 |
| 10% overlap | 7.04 |
| 20% overlap | 7.18 |
| 30% overlap | 13.66 |
| 40% overlap | 10.03 |
Conditions differ in more than overlap ratio — the two 0% conditions differ in silence length — so this is not a monotonic curve. See failure modes for the 30% and 40% conditions. No denoise frontend was enabled in these runs, so they measure the separator and diarizer rather than the complete shipping path.
Compared across our models
The overlap conditions above, run across all three systems rather than ours alone. Scored with ORC-WER, which counts transcription errors without requiring that each word be assigned to the right speaker — a different measurement from the cpWER figures elsewhere on this page, and not directly comparable to them.
| ORC-WER % | 10% overlap | 20% | 30% | 40% |
|---|---|---|---|---|
| No separation | 10.1 | 14.0 | 19.6 | 27.8 |
| Multi-Speaker 1.0 | 10.2 | 15.1 | 17.7 | 18.8 |
| AudioShake Multi-Speaker 2.0 | 9.4 | 9.4 | 12.8 | 14.1 |
Lower is better. Every arm uses the same recordings and the same transcription model — the only thing that changes is what runs in front of it.
- Measured on LibriCSS, a standard benchmark of recorded multi-speaker sessions with controlled amounts of overlapping speech.
- Speaker attribution is measured separately — see the diarization section.
- LibriCSS has no condition above 40% overlap, so the table stops there. Nothing here should be read as a projection beyond that point.
- Zero-overlap conditions are excluded. This table is about overlapped speech, and separation is not the right tool for recordings that have none.
- Research evaluation pipeline, not the production stack. Figures are subject to change.
05 — Structured output
What a job returns
One audio track per speaker at the input sample rate, an overlap-aware timeline of who spoke when, and two confidence scores — returned per 20 ms frame and as a whole-file value weighted by speech activity, so silence cannot inflate them.
assignment_confidence
Did this speech go to the right speaker?
The label-quality signal. Low values mean the audio may be clean but the identity behind it is unreliable — fine for cleanup, risky for anything routed by speaker.
separation_confidence
Did the overlapping voices come apart?
Most informative where people actually talk over each other. Low values mean a track probably carries bleed from another voice — fine for talk-time analytics, not for voice cloning or per-speaker ASR.
The two scores catch different failures and one number cannot tell them apart — audio can be cleanly separated but attributed to the wrong person, or correctly attributed and still carry bleed. If you need a single value, take the lower of the two.
Reading the two scores together
| Separation | Assignment | What it means |
|---|---|---|
| low | low | Both collapsed, usually on overlapping speech. Neither the audio nor the labels are trustworthy — drop, or route to review. |
| high | high | The strongest “trust this” signal. Also what you see where there is no speech at all. |
| high | low | Little to separate, but speaker identity is unreliable. Fine for cleanup; risky for anything routed by speaker. |
| low | high | Labels are trustworthy but the audio likely carries bleed. Fine for talk-time analytics; not for voice cloning or per-speaker ASR. |
We deliberately do not publish recommended thresholds. Score distributions vary across content types and the right cutoff depends on the application, so treat them as a relative ranking signal rather than a gate. For guidance on using them in code, see the API documentation.
06 — FAILURE MODES
Where it stops working
The point of publishing these is that you can predict which of your material will be hard before you run it.
01
Sustained overlap above roughly 20%
Beyond about a fifth of speech time spent in crosstalk, separation quality falls off and transcription error rises with it — the 30% and 40% LibriCSS conditions are where this shows. Most real conversation sits below this threshold; sustained multi-way talking does not.
02
Large casts
There is no fixed ceiling on speaker count, but counting accuracy degrades as the number of people grows — error roughly triples between two-speaker and five-to-six-speaker recordings. Passing a known speaker count, where you have one, removes this source of error.
03
Correct labels, contaminated audio — and the reverse
Two independent failure modes. Audio can be cleanly separated but attributed to the wrong person, or correctly attributed and still carry bleed from another voice. The two confidence scores are designed to tell these apart; one number cannot.
04
Metric caveats that cut against us
Predicted listening quality is referenceless and will score a confidently wrong speaker highly. Open-source separator comparisons use public checkpoints outside their training domain. Several diarization corpora are in-domain for our training mix, as they are for the systems we compare against.
07 — DEFINITIONS
What each number measures
Every metric on this page, defined in the form we score it.
DER
Diarization error rate
The share of speech time attributed to the wrong speaker, missed entirely, or invented. Lower is better. We score with no forgiveness collar and overlapping speech included.
cpWER
Concatenated minimum-permutation word error rate
Transcribe every output track, then score under the speaker assignment that minimises total error. It penalises both transcription mistakes and speaker-attribution mistakes, which is why it is the closest proxy for what a real pipeline experiences.
Speaker-count error
How often the number of people is wrong
The absolute difference between speakers found and speakers present, divided by speakers present, averaged per recording. A four-speaker file where a system finds three scores 25%.
Overlap ratio
What “20% overlap” means
The share of total speech time in which two or more people are talking simultaneously. At 20%, a fifth of the speech in the recording is crosstalk. It is a property of the recording, not of any single interruption.
Get in touch.