Choosing between a speech to speech vs cascaded architecture is a fundamental decision when building a voice AI application. A cascaded system processes audio through a sequence of separate components, like an assembly line. A speech-to-speech system uses a more integrated model that handles audio from input to output.

The decision impacts latency, conversational realism, cost, and your ability to control and audit the system. This guide moves beyond simple latency comparisons to provide a framework for choosing the right path based on your product's specific needs.

Key takeaways

  • Cascaded systems (ASR → LLM → TTS) offer superior control, auditability, and modularity because they expose an intermediate text layer. This makes them ideal for regulated industries or applications that need explicit tool calls and policy checks.
  • Speech-to-speech (S2S) systems process audio directly to audio. This can create more natural, expressive conversations by preserving vocal prosody and timing.
  • The choice is not just about latency. Real-world performance depends on streaming, turn-taking logic, tool calls, and network conditions. A poorly optimized S2S system can feel slower than a well-tuned cascade.
  • Hybrid models exist. Production systems often blend approaches. They might use S2S for conversational flow while generating transcripts for analytics or using text-based logic for specific integrations (Source).

How Do Cascaded and Speech-to-Speech Architectures Work?

A cascaded system processes voice in a multi-stage pipeline, offering clear checkpoints. A speech-to-speech (S2S) system integrates these steps, processing audio more directly. While S2S consolidates the core models, both architectures require significant surrounding infrastructure to manage a real-time conversation.

The Cascaded Pipeline: An Assembly Line for Voice

A cascaded architecture breaks the task into distinct, inspectable stages. Think of it as an assembly line where each station performs a specific job before passing the work to the next.

  1. Voice Activity Detection (VAD) & Endpointing: The system first listens for human speech and determines when the user has finished talking. This "endpointing" decision is critical for natural turn-taking.
  2. Automatic Speech Recognition (ASR): The user's speech audio is sent to an ASR model, often while they are still talking. This component converts the audio into a text transcript. This is a critical step in any voice AI agent architecture.
  3. Language Model (LLM) & Business Logic: The final text transcript from the ASR is sent to a large language model or a deterministic policy engine. This component interprets the user's intent, manages the conversation state, and decides what to do next—which may involve calling an external tool like a booking API.
  4. Text-to-Speech (TTS): The text response generated by the LLM is sent to a TTS model. This component synthesizes the text into audible speech. Using streaming TTS can significantly lower perceived latency (Source).
  5. Audio Output: The synthesized audio is played back to the user. The system must be able to interrupt or cancel this playback if the user starts speaking again (an action called "barge-in").

The key feature of this pipeline is the intermediate text layer. This text provides a clean, auditable record that can be logged, checked against compliance rules, and used to trigger precise, text-based tools.

The Speech-to-Speech Pipeline: An Integrated Expert

A speech-to-speech (S2S) system is designed to accept audio and produce audio without requiring an application-visible text stage in the middle (Source). It's more like an expert craftsman who performs multiple steps of a task holistically.

  1. Voice Activity Detection (VAD) & Endpointing: Like a cascade, an S2S system must still determine when the user is speaking and when they have yielded the turn.
  2. S2S Model Processing: The user's speech audio is fed directly into a single, integrated model. This model is responsible for understanding the audio, reasoning about the response, and generating the reply audio all at once. It has access to the user's tone, pauses, and intonation, details which can be lost when converting to text.
  3. Audio Output: The model generates response audio, which is streamed back to the user.

While this appears simpler, it's not a "magic box." Production S2S systems still need orchestration for state management, tool calls, and safety checks (Source). For example, the model might internally generate a text representation to call a tool or pass through a safety filter, even if that text is never exposed to the application developer.

Speech-to-Speech vs. Cascaded: A Comparison Matrix

The best architecture depends on your application's priorities. A system for a bank has different needs than a conversational companion. This table compares the two approaches across key factors for production systems.

Factor Cascaded System Speech-to-Speech (S2S) System
Latency (TTFA) Higher. The time-to-first-audio is the sum of delays from each sequential stage (ASR, LLM, TTS). Potentially Lower. An integrated model can begin generating a response before it has fully "transcribed" the input, reducing handoff delays.
Conversational Expressiveness Lower. ASR reduces speech to text, losing nuances like tone and timing. The TTS must generate expressiveness from scratch (Source). Higher. The model has direct access to the input audio's prosody and can reflect it in the output audio for more natural turn-taking and intonation.
Audit & Compliance High. The intermediate text provides an explicit, human-readable record for logging, review, and policy enforcement before speech is ever generated (Source). Lower. Requires an auxiliary transcription model or relies on internal model logs, which may be less transparent or guaranteed.
Debugging & Observability Easier. Errors can be isolated to a specific component (e.g., a bad ASR transcript or a flawed LLM response). Each stage can be evaluated independently (Source). Harder. With an end-to-end model, it's more difficult to pinpoint whether a failure was due to misrecognition, faulty reasoning, or poor audio generation (Source).
Component Modularity High. You can mix and match the best ASR, LLM, and TTS providers and upgrade them independently as technology improves (Source). Low. The core components are tightly coupled within a single model from one provider.
Tool Use & Structured Data More Mature. Leverages well-established text-based tool-calling interfaces from major LLMs. Structured data validation is straightforward (Source). Developing. Can call tools, but the mechanism may be more proprietary. Ensuring schema adherence and handling confirmation can be complex.

A Worked Example: Tracing a User Request

To make these differences concrete, let's trace a user's request through both systems. The numbers are illustrative but based on typical production estimates.

Scenario: A user says, "Hi, I need to change my flight to tomorrow."

Cascaded Path

  1. User speaks: The system detects speech.
  2. ASR Processing (150 ms): As the user talks, a streaming ASR generates a partial transcript.
  3. User Finishes: The system's endpointing logic determines the user has stopped. The final transcript is sent to the LLM.
  4. LLM Processing (300 ms to first text token): The LLM analyzes the text "change my flight to tomorrow" and determines it needs to call the flight_booking tool.
  5. Tool Call (500 ms): The system calls an external API to check flight availability. This happens outside the core voice pipeline.
  6. LLM Generates Response: The LLM receives the tool's result and formulates the final text, e.g., "Okay, I see a flight available at 10 AM tomorrow. Is that okay?"
  7. TTS Processing (100 ms to first audio): The TTS model receives the text and begins generating speech.

Total Time-to-First-Audio (TTFA): After the user finishes speaking, the TTFA is the sum of the LLM and TTS delays: 300 ms + 100 ms = 400 ms. This ignores the tool call, which adds to the overall turn time. The key output is a clear, logged transcript and a structured tool call.

Speech-to-Speech Path

  1. User speaks: The system detects speech.
  2. S2S Model Processing (350 ms to first audio): The model processes the audio stream directly. It internally recognizes the intent, formulates a plan (including a tool call), and begins generating an audio response simultaneously. It might start speaking with a filler phrase like "Let me check on that for you..." while the tool call runs in the background.
  3. Tool Call: The tool call is executed, similar to the cascade.
  4. Model Continues Response: The model seamlessly incorporates the tool's result into its spoken response.

Total Time-to-First-Audio (TTFA): A typical S2S system might start speaking in 300–500 ms (Source). It feels faster because it can respond before the entire chain of logic is complete. However, the mechanism for the tool call is less transparent from the outside.

When Is a Cascaded Pipeline Still Better?

A cascaded system is the superior choice when control, auditability, and modularity are more important than achieving the absolute lowest possible latency or the most fluid vocal expression. This architecture is not legacy; it is a deliberate engineering choice for specific requirements.

Choose a cascaded pipeline when:

  • Strict audit trails are non-negotiable. In finance, healthcare, or legal contexts, having an exact text transcript of every interaction is often a compliance requirement. The intermediate text layer provides this record by default (Source).
  • You need to integrate complex, deterministic business logic. If your application relies on precise, text-based rules—like validating an account number with a checksum or parsing specific formats—a cascade provides a natural checkpoint to apply this logic. This is a common pattern in comparing agentic workflows vs. RPA.
  • You want maximum vendor flexibility. A cascaded design allows you to select the best-in-class ASR, LLM, and TTS from different providers. You can swap them out independently as technology improves or costs change, avoiding vendor lock-in (Source).
  • Your domain requires highly specialized models. If you need a custom ASR model tuned for specific medical jargon or a unique TTS voice for your brand, a modular cascade makes it easier to integrate these specialized components without changing the entire pipeline.

What Are the Common Mistakes and Limits to Know About?

Evaluating voice systems is complex. Teams often fall into pitfalls by oversimplifying metrics or misunderstanding the architectures.

  • Mistake: Treating latency as a single number. Reality: Perceived speed is not just one number. You must differentiate between time-to-first-audio (TTFA), time to complete a response, and turn-taking delay. A system with low TTFA can still feel slow if its endpointing logic waits too long to respond. A full voice agent latency budget is essential.
  • Mistake: Assuming S2S is operationally simpler. Reality: While S2S might collapse three models into one, you still need to build and manage the surrounding infrastructure for orchestration, state management, tool calls, safety monitoring, and error recovery (Source).
  • Mistake: Trusting transcripts as perfect ground truth. Reality: ASR systems make mistakes, especially with names, accents, or background noise. Any business logic that relies on a transcript must be designed to handle potential recognition errors (Source).
  • Mistake: Ignoring the user's ability to interrupt. Reality: Natural conversation involves frequent interruptions. A good voice system, whether cascaded or S2S, must be able to stop its own speech instantly when the user starts talking. This "barge-in" capability is crucial for user experience.

What to learn next

Sources and further reading