Multi-Speaker 2.0: From Who-Spoke-When to Separated, Structured Conversation

AudioShake Multi-Speaker 2.0 builds on our 2025 1.0 release. It takes single-track conversational audio — podcasts, interviews, UGC, TV and film clips — and separates the file into individual speaker stems, or streams, tracks, or channels, depending on which industry you work in.

This post covers how we evaluated the model, the scoring configuration behind every number, and the places the model stops working. A general overview of the model and its capabilities is here, and audio samples are on Hugging Face.

The problem with who-spoke-when

Systems that handle multi-speaker audio have historically answered one question: who spoke when. That is diarization, and it is foundational to voice and ASR workflows. But a speaker label is not the same thing as a speaker track. When two people talk at once, diarization can mark the overlap. It cannot recover the two voices as separate audio.

Natural conversation is full of overlap — backchannels, interjections, simultaneous starts. In a diarization-only pipeline those regions stay mixed even when the labels are correct. The same is true of everything that is not a speaker label: room tone, environmental cues and the rest of the scene stay embedded in the recording rather than available as separate audio.

Multi-Speaker 2.0 recovers each overlapping voice individually as separate audio, and labels it. Overlap is represented directly in the timeline: when two people speak at once, both speakers are active, not one. The result is not just who-spoke-when metadata, but per-speaker audio that downstream systems can process.

How we evaluate: audio quality and downstream utility

A per-speaker track has to satisfy two different tests. First, is the recovered audio clean? Second, does that audio remain useful to the systems that consume it? Signal-level metrics answer the first question and task-level metrics answer the second. Neither is sufficient alone, because each has a known failure mode: SI-SDR rewards waveform fidelity a downstream recognizer may not care about, and ASR-based metrics tolerate artifacts a human would not accept.

So we report both. Task-level: cpWER — concatenated minimum-permutation word error rate, via the open MeetEval toolkit — for what the separated tracks enable in transcription, plus diarization error rate and speaker-count error for whether the labels are right. Signal-level: SI-SDR against clean references, and DNSMOS and personalized DNSMOS for predicted perceptual quality.

Diarization: are labels attributed correctly?

Separation makes the speaker tracks usable individually. Diarization determines whether those tracks are attributed correctly over time. This part of the evaluation runs on spontaneous audio: AMI, CallHome, VoxConverse and the rest are real meetings, real phone calls and real web video.

We measure who-spoke-when on 14 public datasets across four domains, scored in the strictest configuration in common use — no forgiveness collar, overlapping speech included, held-out test splits, 16 kHz, applied identically to every system. Those choices matter more than most published DER numbers admit. A 250 ms collar and overlap exclusion, both still common, remove the hardest frames from the denominator.

We compare against three open-source diarizers and one proprietary service. The strongest open-source system is not the same one in every domain, so the first table reports the best result available in each category rather than a single baseline.

The per-system results matter as much as the summary, because the open-source field is uneven. community-1 is the most widely deployed open-source diarizer and the one most teams will have in their pipeline today, but diarizen is stronger in three of four domains. sortformer-4spk-v2 is competitive on telephone audio and falls apart on broadcast.

Two checks help isolate what is driving the result.

The advantage is not only about detecting simultaneous speech. Recompute DER on primary-speaker annotations — at each instant keep only the speaker who took the floor first, applied identically to the reference and to every system's output, so all overlapped speech is excluded — and the ranking does not change. We stay ahead of community-1 in all four domains, and ahead of precision-2 in three, with a tie on web video.

Non-English telephone conversation is newly covered. The previous diarizer had no conversational telephone speech in its training mix, so this is new domain coverage rather than evidence of better generalization.

Task-level: does the separated audio improve transcription?

The most direct downstream test is to transcribe each separated track and count the errors. We are less interested in whether the waveform looks cleaner than in whether the recovered speaker tracks preserve the words. We separate LibriCSS, transcribe every output track with Whisper large-v3, and score cpWER with MeetEval — the identical pipeline for every system.

How accuracy holds as overlap increases. Up to 20% overlap, transcription of separated speakers is as accurate as transcribing clean single-speaker stems. There is no accuracy penalty for people talking over each other until you pass that point, and 20% sustained overlap is already well beyond what most real conversation produces.

Signal-level: what changed in the audio itself?

The transcription result tells us what survives downstream. SI-SDR lets us inspect the separator itself against clean references, with no diarizer or denoise frontend in the path.

Noise and interference. Overlap is only one source of contamination. On a two-speaker benchmark with WHAM noise added, the full 2.0 audio path removes 75% more of the combined noise and interference than 1.0's. In perceptual terms, predicted listening quality on the separated tracks reaches 3.9 on the 5-point personalized DNSMOS scale, close to a full point above the previous model, and the DNSMOS background sub-score moves from 3.2 to 4.1 — roughly the difference between background noise you can clearly hear behind the isolated voice and a track that is essentially clean. These are model-predicted ratings, not a human panel, and the metric is referenceless: a track that confidently contains the wrong speaker's words still scores well. We report it alongside the signal-level results, which catch exactly that error, and never on its own.

From separated audio to structured output

A Multi-Speaker 2.0 job returns more than audio: one track per speaker at the input sample rate, an overlap-aware diarization timeline, a speaker count, and two confidence scores. Both scores are in [0, 1] and are returned per 20 ms frame and as a whole-file value weighted by speech activity, so silence cannot inflate them.

assignment_confidence is how confident the model is that it assigned speech to the right speakers — the diarization-quality signal. separation_confidence is how confident the model is that it pulled apart overlapping speech correctly. It is most informative where speakers actually overlap; on single-speaker frames it sits near 1.0 unless there is bleed between streams.

The two scores correspond to two different failure modes. Audio can be separated cleanly but attributed to the wrong speaker, or attributed correctly and still carry bleed from another voice. One number cannot distinguish those cases. If you need one, take the minimum.

Where it stops working

• Accuracy degrades past roughly 20% sustained speaker overlap. The 30% and 40% LibriCSS conditions are where this shows.
• There is no fixed ceiling on speaker count, but speaker-counting accuracy falls as the number of speakers grows. Passing a known speaker count, where you have one, removes this source of error.
• DNSMOS and personalized DNSMOS results are model-predicted, not human-panel, and referenceless.
• The TF-Locoformer comparison uses a public checkpoint outside its training domain.
• Several diarization corpora are in-domain for our training mix, as they are for the systems we compare against.

How to try it

Multi-Speaker 2.0 is available through our self-serve Developer site. We also offer Enterprise contracts with custom terms, SLAs and additional metadata. Full scoring configuration and per-dataset results are on the benchmarks page.