The Whisper model architecture uses an encoder-decoder Transformer to perform a range of speech-processing tasks. Unlike older systems, it combines acoustic and language modeling into a single, end-to-end network. This design allows it to perform highly accurate transcription and translation but also introduces unique failure modes, such as generating text from silence.
Key takeaways
- Unified Architecture: Whisper is an encoder-decoder Transformer, not a complex pipeline of separate parts. This single model handles transcription, translation, and language identification.
- Windowed Processing: The core neural network processes audio in fixed 30-second windows. A separate script handles longer files by segmenting the audio and stitching the results.
- Autoregressive Generation: The decoder generates text one token at a time. Each new token prediction depends on both the original audio and the tokens it has already produced.
- Cause of Hallucinations: When the audio input contains silence or noise, the encoder provides no linguistic signal. The decoder can then default to its powerful internal language model, generating plausible but entirely fabricated text.
What Is the Whisper Model Architecture?
The Whisper model architecture is an end-to-end, sequence-to-sequence Transformer. This design integrates the entire speech recognition process into a single network that learns to map audio sequences to text sequences. It differs from traditional Automatic Speech Recognition Explained: How It Works systems that use separate, individually trained components for acoustic modeling, pronunciation, and language modeling (Source name).
The model is also multitask and multilingual. It was trained to handle speech recognition across many languages, translation to English, and language identification (Source name). To control these tasks, the decoder is prompted with special tokens that specify the desired output, such as SOT (start-of-transcript), <|translate|>, or <|transcribe|>.
At its input, Whisper does not work with raw audio waveforms. Instead, the audio is converted into a log-Mel spectrogram. A log-Mel spectrogram is a 2D visual representation of sound that shows how the intensity of different frequency bands changes over time, mimicking how the human ear perceives sound.
How Whisper Processes Audio: A Step-by-Step Guide
The full process from a long audio file to a final transcript involves two layers: a pipeline for handling audio length and the core neural network model that performs the actual transcription within a 30-second window.
Step 1: Audio Preprocessing and Segmentation
First, the raw audio file is prepared for the model. Because the core model expects a 30-second input, a surrounding pipeline script slices any longer audio into 30-second chunks. Each chunk is then converted into a log-Mel spectrogram, which is the actual input to the neural network (Source name).
Step 2: The Encoder Creates Audio Representations
The spectrogram enters the encoder half of the Whisper model architecture. The encoder's job is to build a rich, contextual understanding of the audio. The encoder processes the spectrogram through its initial layers to extract features before the data flows into a series of standard Transformer blocks. Inside these blocks, a mechanism called self-attention allows every moment in the audio to be compared with every other moment, building a representation where each audio feature is aware of its full context.
Step 3: The Decoder Generates Text Autoregressively
The decoder generates the transcript one token (a word or part of a word) at a time. This is an autoregressive process, meaning each prediction depends on the previous ones. For each new token, the decoder uses two attention mechanisms:
- Masked Self-Attention: It looks at the text tokens it has already generated in the current sequence. This is "masked" to prevent it from seeing future tokens, ensuring it only uses past context.
- Cross-Attention: It looks at the full output of the encoder. This allows the decoder to focus on the most relevant parts of the audio to predict the next word.
The decoder combines these two sources of information—what it has already written and what is in the audio—to predict the most likely next token (Source name).
Step 4: Special Tokens Control the Task
The decoder's behavior is guided by special tokens. Before generating the main text, it is given a prompt sequence that might include tokens specifying the language, the task (transcribe or translate), and whether to predict timestamps. For example, to start transcription, the decoder is prompted with a <|startoftranscript|> token. It can then predict language tokens (e.g., <|en|>), task tokens (<|transcribe|>), and timestamp tokens (<|0.00|>) as part of its output.
Step 5: The Pipeline Manages Long-Form Transcription
For audio longer than 30 seconds, the pipeline script coordinates multiple runs of the model. After transcribing the first 30-second chunk, it uses the text from the end of that chunk to prompt the model for the next chunk. This helps maintain consistency. The model's ability to predict timestamps also helps align and merge the segments into a final, coherent transcript (Source name). However, errors or hallucinations in one window can sometimes affect subsequent windows.
A Conceptual Walkthrough of the Whisper Architecture
To make the architecture concrete, let's trace the data flow through the model for a single 30-second audio window.
- Input Audio: A standard 30-second audio clip is prepared for the model.
- Log-Mel Spectrogram: This audio is converted into a log-Mel spectrogram. This creates a two-dimensional representation of audio where one axis represents time and the other represents frequency, serving as the input for the encoder.
- Encoder Input: The spectrogram is processed by the encoder's initial layers, which perform feature extraction. Positional encodings are added to give the model information about the sequence order, and the resulting data is passed to the main Transformer encoder blocks.
- Encoder Output: The encoder processes this sequence through its self-attention layers. The output is a sequence of contextualized audio features. This tensor represents the model's complete understanding of the 30-second audio clip.
- Decoder Output: The decoder receives the encoder's output and begins generating a sequence of text tokens. For each step, it produces a probability distribution over its vocabulary of tokens. The chosen token is added to the output sequence, and the process repeats until an end-of-text token is generated.
Why Does Whisper Hallucinate on Silence?
Whisper generates fluent but incorrect text on silent audio because its decoder architecture balances two inputs: the audio signal from the encoder and its own internal language model. When the audio signal is empty or contains non-speech noise, the encoder produces a representation with no useful linguistic information (Source name).
With no guidance from the audio, the decoder's prediction is dominated by its other input: its own previously generated tokens and the language patterns it learned during training (Source name). The decoder essentially becomes a language model generating plausible-sounding text based on its training data, completely disconnected from the silent audio. This can result in repetitive phrases or sentences that are common in its training set but absent from the input. Studies have found that about 1% of tested transcriptions contained such hallucinated phrases (Source name).
This behavior can be worsened by certain decoding settings. For example, some research indicates that using a larger beam search size, a technique that explores multiple possible transcriptions, can actually increase the rate of hallucinations (Source name). Pre-processing audio with a voice activity detector (VAD) to remove silent segments is a common and effective strategy to reduce these errors.
What Are the Key Differences Between the Encoder and Decoder?
The encoder and decoder are both built from Transformer blocks, but they have distinct roles, inputs, and attention mechanisms. Understanding these differences is key to understanding the full Transformer Architecture Explained: From Tokens to Logits.
| Feature | Encoder | Decoder |
|---|---|---|
| Primary Purpose | To build a rich, contextual representation of the entire input audio segment. | To generate the output text sequence one token at a time. |
| Input | A log-Mel spectrogram representing 30 seconds of audio. | 1. The full output from the encoder. 2. The sequence of text tokens generated so far. |
| Attention Mechanism | Self-Attention: Each audio frame can attend to every other frame (past and future) within the 30-second window. This is bidirectional. | Masked Self-Attention: Each text token can only attend to previous tokens in the sequence. Cross-Attention: Each text token attends to all audio frames from the encoder's output. |
| Output | A sequence of contextual feature vectors for the decoder. | A sequence of token IDs, which are mapped to words, punctuation, and special tokens to form the final transcript. |
Common Mistakes and Limitations of the Whisper Architecture
While powerful, Whisper's architecture has specific limitations. Awareness of these can help you avoid common pitfalls and interpret its output correctly.
- Assuming fluent output is always accurate. Whisper's ability to generate fluent language means it can produce coherent but completely fabricated text, especially on silent or noisy audio (Source name). Always verify critical transcripts against the source audio.
- Using a large beam search to "improve" accuracy. Some studies have found that larger beam sizes can actually increase hallucination rates. A beam size of 1 (greedy decoding) may be safer for audio with non-speech segments (Source name).
- Expecting perfect word-level timestamps. Timestamps are also generated by the model and can be inaccurate or misaligned. For applications requiring precise timing, dedicated forced-alignment tools often perform better.
- Ignoring the need for voice activity detection (VAD). For recordings with significant pauses, music, or background noise, failing to use a VAD is a missed opportunity. Pre-processing audio to remove non-speech segments is one of the most effective ways to reduce hallucinations (Source name).
- Treating it as a real-time streaming model. The base Whisper architecture is not designed for low-latency streaming. Its reliance on 30-second windows means it must wait for a full chunk of audio before it can begin processing, making it unsuitable for live conversation transcription without significant modification.
What to learn next
- CTC Loss Explained: A Complete How-It-Works Guide - A technical explanation of Connectionist Temporal Classification (CTC) loss, including how it solves sequence alignment issues in ASR and a
- Voice Activity Detection: A Practical Guide - Learn how voice activity detection (VAD) works. This guide explains the full pipeline from audio frames to speech timestamps with examples a
- Latent Diffusion Models Explained: How They Work - An explanation of latent diffusion model architecture, including the roles of the VAE and U-Net, and the computational efficiencies of the l
- Transformer Architecture Explained: From Tokens to Logits: Dive deeper into the self-attention and cross-attention mechanisms that power models like Whisper.
- Automatic Speech Recognition Explained: How It Works: Explore the broader field of ASR and compare Whisper's end-to-end approach with traditional, modular pipelines.
- How Does Text-to-Speech Work? The Full Pipeline: Learn about the reverse process of converting text back into spoken audio.
Sources and further reading
- Whisper official repository and model documentation
- Robust Speech Recognition via Large-Scale Weak Supervision
- Careless Whisper: Speech-to-Text Hallucination Harms
- Investigation of Whisper ASR Hallucinations Induced by Non-Speech Audio
- Whisper Hallucination Detection and Mitigation via Hidden Representations
