Speaker diarization is the technology that analyzes an audio recording and determines who spoke and when. It structures conversations, meetings, and interviews by creating a timeline of speaker activity. This process makes multi-speaker recordings easier to transcribe, search, summarize, and analyze.
Key takeaways
- Speaker diarization answers “who spoke when” by assigning anonymous labels (e.g., Speaker 1, Speaker 2) to time segments, not by identifying speakers by name.
- A common pipeline involves three main steps: segmenting the audio into speech chunks, creating a numerical "speaker embedding" for each chunk, and clustering these embeddings to group segments from the same speaker.
- The primary metric for accuracy is the Diarization Error Rate (DER), which measures the percentage of time that is incorrectly labeled due to missed speech, false alarms, or speaker confusion.
- Modern End-to-End Neural Diarization (EEND) models can handle overlapping speech more effectively than traditional clustering-based systems but can face challenges with very long recordings or a high number of speakers (Source).
What is Speaker Diarization?
Speaker diarization is the process of dividing an audio stream into segments based on speaker identity (Source). Its goal is to answer the question, "who spoke when?" The system outputs a timeline that marks the start and end times for each speaker's turns. Crucially, it assigns anonymous labels like "Speaker 1" and "Speaker 2" rather than identifying speakers by their actual names.
This anonymous labeling is a key distinction. The separate task of matching a voice to a known person is called speaker identification. Diarization simply partitions the conversation. For example, in a meeting recording, diarization would tell you that Speaker 1 spoke for the first two minutes, followed by Speaker 2 for thirty seconds, and then Speaker 1 again.
The primary use of speaker diarization is to add structure to multi-speaker audio. This structure makes downstream tasks like transcription and analysis much easier. For applications like customer-service recordings and interviews, a diarized transcript allows users to easily follow the conversation and attribute quotes correctly (Source).
How Does Speaker Diarization Work?
Most speaker diarization systems use a modular pipeline that processes audio in several distinct steps to determine who spoke when. This involves finding speech, converting it into numerical representations, and grouping those representations by speaker. This clustering-based approach is widely used, though newer end-to-end models take a different path (Source).
Here is a breakdown of the traditional pipeline:
1. Voice Activity Detection (VAD)
The first step is to find the parts of the audio that contain human speech. A Voice Activity Detection (VAD) model scans the audio and separates speech from non-speech segments like silence, background noise, or music. This reduces the amount of data the system needs to process and prevents non-speech sounds from being mistaken for a speaker.
2. Speaker Segmentation
Once speech regions are identified, they are divided into short, uniform chunks. The goal is to create segments that are short enough to likely contain only one speaker. For very long recordings, this process is often done in overlapping windows to ensure consistency across the entire file (Source).
3. Speaker Embedding Extraction
This is the core of speaker differentiation. Each short speech segment is fed into a neural network that produces a speaker embedding. A speaker embedding is a fixed-size vector—a list of numbers—that captures the unique characteristics of a person's voice, such as pitch, tone, and timbre. The model is trained so that segments from the same speaker produce mathematically similar vectors. Similar ideas are used to create text embeddings for words.
4. Clustering
With a collection of speaker embeddings, the next step is to group similar ones. A clustering algorithm analyzes the mathematical distances between all the embedding vectors. Vectors that are close together are grouped into the same cluster. Each cluster represents a single speaker.
A common method used is Agglomerative Hierarchical Clustering (AHC). This algorithm starts by treating each embedding as its own cluster and then iteratively merges the closest pairs of clusters until a stopping condition is met, such as reaching a predefined number of speakers or a distance threshold.
5. Label Assignment
Finally, the system assigns a unique, anonymous label (Speaker 1, Speaker 2, etc.) to each cluster. It then consolidates the timestamps from all segments within a cluster to generate the final speaker timeline. The output is a list of events, such as: [00:01.23 - 00:05.45]: Speaker 1.
End-to-End Neural Diarization (EEND)
A more recent approach, End-to-End Neural Diarization (EEND), combines these steps into a single neural network. EEND models treat diarization as a multi-label classification problem, directly predicting which speaker (or speakers) are active at each moment in time (Source). A major advantage of EEND is its ability to naturally handle overlapping speech, a significant challenge for traditional clustering-based systems (Source). However, these models can struggle with very long recordings or a large number of speakers (Source).
A Worked Example: From Audio to Speaker Timeline
To make the process concrete, let's walk through an illustrative example of a 15-second audio clip with two speakers, Alice (Speaker 1) and Bob (Speaker 2).
Hypothetical Audio Timeline:
0.0s - 1.0s: Silence1.0s - 5.0s: Alice speaks.5.0s - 6.5s: Silence6.5s - 10.0s: Bob speaks.10.0s - 12.0s: Alice and Bob speak at the same time (overlap).12.0s - 15.0s: Silence
Here’s how a clustering-based diarization system would process it:
Voice Activity Detection: The system first identifies all periods containing speech.
- Output: Speech detected from
1.0sto5.0sand from6.5sto12.0s.
- Output: Speech detected from
Segmentation: The detected speech regions are split into uniform 1.5-second segments.
- Segment 1:
1.0s - 2.5s(Alice) - Segment 2:
2.5s - 4.0s(Alice) - Segment 3:
4.0s - 5.0s(Alice, shorter segment) - Segment 4:
6.5s - 8.0s(Bob) - Segment 5:
8.0s - 9.5s(Bob) - Segment 6:
9.5s - 11.0s(Bob + Alice) - Segment 7:
11.0s - 12.0s(Alice + Bob, shorter segment)
- Segment 1:
Embedding Extraction: A numerical vector (embedding) is generated for each segment.
- The embeddings for Segments 1, 2, and 3 will be very similar because they all contain Alice's voice.
- The embeddings for Segments 4 and 5 will be similar to each other but different from Alice's.
- The embeddings for Segments 6 and 7 will be different from the others because they contain a mix of voices.
Clustering: The algorithm groups the embeddings based on similarity.
- Cluster A: It groups the embeddings from Segments 1, 2, and 3.
- Cluster B: It groups the embeddings from Segments 4 and 5.
- Outliers: Segments 6 and 7 might be assigned to the closest cluster (likely Bob or Alice, causing an error) or left unassigned, depending on the system's design.
Final Output: The system assigns labels and merges timestamps.
[01.0s - 05.0s]: Speaker 1(from Cluster A)[06.5s - 09.5s]: Speaker 2(from Cluster B)- (Note: The overlapped speech from 9.5s-12.0s would likely be misattributed to either Speaker 1 or Speaker 2 by this simple system.)
How is Diarization Accuracy Measured?
The standard metric for evaluating speaker diarization accuracy is the Diarization Error Rate (DER). DER measures the percentage of time in a recording that is incorrectly labeled by the system. A lower DER indicates higher accuracy. The score is calculated by comparing the system's output (hypothesis) against a manually created, ground-truth transcript (reference).
DER is the sum of three distinct types of errors:
- Missed Speech (False Negative): The percentage of time a speaker is talking in the reference transcript, but the system output shows no speech.
- False Alarm (False Positive): The percentage of time the system output shows speech, but the reference transcript shows silence or non-speech noise.
- Speaker Confusion Error: The percentage of time speech is present, but the system assigns it to the wrong speaker.
The formula is:
DER = (Total Missed Speech Time + Total False Alarm Time + Total Speaker Confusion Time) / Total Reference Speech Time
It is important to know that a single DER number is not meaningful without context (Source). Results depend heavily on the evaluation setup. For example, evaluators often use a "forgiveness collar" of about 250 milliseconds around segment boundaries, meaning errors within this small window are ignored. How overlapping speech is scored—or if it is excluded from scoring altogether—also dramatically changes the final DER.
What's the Difference Between Diarization, Identification, and ASR?
Beginners often confuse speaker diarization with related speech technologies like speaker identification and Automatic Speech Recognition (ASR). While they can be used together, they perform distinct tasks. Diarization answers "who spoke when?", identification answers "who is this specific person?", and ASR answers "what was said?"
Here is a comparison:
| Technology | Primary Goal | Output | How They Work Together |
|---|---|---|---|
| Speaker Diarization | Who spoke when? | Anonymous speaker labels (e.g., Speaker 1) with timestamps. | Provides the speaker timeline needed to attribute transcribed text in a multi-speaker conversation. |
| Speaker Identification | Who is this specific person? | A known identity or name from a database (e.g., "Jane Doe"). | Can be used after diarization to map anonymous labels like "Speaker 1" to a real name. |
| Automatic Speech Recognition (ASR) | What was said? | A text transcript of the spoken words. | Takes audio segments identified by diarization and converts them into speaker-labeled text. For more, see our guide on Automatic Speech Recognition Explained: How It Works. |
What are the common mistakes and limits of diarization?
Speaker diarization systems work well in clean audio with distinct speakers but can struggle in more complex, real-world scenarios. It's important to understand these limitations to set realistic expectations for system performance.
- Overlapping Speech (Crosstalk): When multiple people speak at once, traditional clustering-based systems fail because they assume only one speaker per segment. This is a primary motivation for developing end-to-end models that can handle multiple speakers simultaneously (Source).
- Short Speaker Turns: Utterances shorter than one or two seconds may not provide enough audio data to generate a reliable speaker embedding. This can lead to these short turns being misidentified or creating many small, fragmented speaker segments.
- Many Speakers: As the number of speakers increases, the vocal characteristics become more crowded in the embedding space. This makes the clustering task significantly harder and increases the chance of speaker confusion.
- Similar Voices: People with very similar pitch, tone, and speaking style can be difficult for a system to distinguish. The algorithm may mistakenly merge their speech into a single speaker cluster.
- Poor Audio Quality: Background noise, echoes (reverberation), and distance from the microphone all degrade the quality of the audio signal. This distortion can alter a speaker's vocal features, making their embeddings less consistent and harder to cluster accurately.
How does diarization work on phone calls?
Phone calls are a common and important use case for speaker diarization, particularly in call centers for documentation and analysis (Source). How the system works depends entirely on whether the call is recorded on one or two channels.
Dual-Channel (Stereo) Recordings
In many professional call center systems, the agent and the customer are recorded on separate audio channels (a stereo recording). This makes diarization trivial. The system doesn't need to analyze voice characteristics. It can simply assign "Speaker 1" to all speech on the left channel and "Speaker 2" to all speech on the right channel. This method is extremely fast and accurate.
Single-Channel (Mono) Recordings
When both participants are mixed into a single audio channel, as is common with mobile phone calls or conference lines, diarization becomes much more difficult. The system must use the standard acoustic pipeline: VAD, segmentation, embeddings, and clustering. This process is susceptible to all the common challenges, including crosstalk and audio quality issues caused by network compression and background noise.
What to learn next
- How Does Voice Cloning Work? A Technical Guide - Learn how voice cloning works technically. This guide explains the step-by-step process, from speaker embeddings to vocoders, and covers the
- Automatic Speech Recognition Explained: How It Works: Learn how systems convert the audio segments identified by diarization into written text.
- Text Embeddings Explained: A Technical Guide: Dig deeper into the concept of vector embeddings, a core component of modern AI.
- Streaming vs Batch ASR: Latency vs. Accuracy: Explore the trade-offs between processing audio in real-time versus all at once, which applies to diarization as well.
