A real-time voice AI agent feels like magic, but its architecture is grounded in careful engineering. While many diagrams show a simple three-step process, a production system is a complex pipeline designed to handle the challenges of real-time conversation. Understanding this full architecture is key to building agents that are fast, reliable, and natural to talk to.

Key takeaways

  • A production voice AI agent architecture is more than a simple Speech-to-Text (STT) → LLM → Text-to-Speech (TTS) chain. It is a complex real-time system involving audio transport, turn detection, interruption handling, and tool execution.
  • The cascaded streaming pipeline (STT-LLM-TTS) is a dominant, practical architecture. It allows for independent component selection, tuning, and clear observability of each stage.
  • Perceived latency is an end-to-end metric. Optimizing the LLM alone is not enough; every step from audio capture to playback contributes to the total delay you hear.
  • Handling user interruptions correctly is a critical design challenge. This requires explicit state management to cancel in-flight generation and tool calls when a user speaks over the agent.
  • Architectural choices determine not just latency, but also cost, debuggability, model replaceability, and the agent's ability to perform reliable actions.

What Components Make Up a Voice AI Agent?

The core of a voice AI agent architecture is a three-stage pipeline that processes sound into meaning and back into sound. This cascaded model connects specialized components for each task, forming the foundation of most modern systems. While simple, it provides a clear data flow: audio in, processing, audio out.

The first component is Speech-to-Text (STT), also known as Automatic Speech Recognition (ASR). This model’s job is to convert the raw audio of a person's voice into a written text transcript.

Next, a Large Language Model (LLM) receives the text transcript. The LLM acts as the agent's brain. It analyzes the text to understand the user's intent, manages the dialogue, decides on a response, and generates that response as new text. This is where the core reasoning and conversational logic happen, powered by a model like a transformer.

Finally, Text-to-Speech (TTS) synthesizes the LLM's text response back into audible speech (Pipeline types). A high-quality TTS system generates audio that sounds natural and expressive, completing the conversational loop. This modular design allows developers to choose, tune, or replace each component independently (Pipeline types).

How a Production Voice Agent Works: The 10-Step Pipeline

A production voice AI agent architecture expands on the simple STT-LLM-TTS model to handle the complexities of real-time interaction. It adds components for audio transport, state management, and safety, turning a basic concept into a robust system that can manage a fluid conversation.

  1. Audio Transport. Before any AI processing, audio must be captured and delivered reliably. A real-time channel like WebRTC streams audio from the user's microphone to the agent, handling network issues like packet loss and jitter (Beyond the Demo: Building Voice Agents That Meet Enterprise Requirements).
  2. Turn Detection. The system must know when the user is speaking, pausing, or finished. Voice Activity Detection (VAD) or more advanced models analyze the audio stream to identify speech boundaries and detect when a user interrupts the agent (Voice communication channel documentation).
  3. Streaming STT/ASR. Instead of waiting for the user to finish talking, streaming STT converts speech to text incrementally. It provides partial transcripts that can be used to prepare the system, reducing perceived latency.
  4. Conversation State Management. The agent needs a memory. A state manager tracks the conversation history, user data, and results from any tools used. This context is essential for the LLM to have a coherent, multi-turn conversation.
  5. LLM Reasoning & Tool Selection. The LLM receives the final transcript and the conversation state. It interprets the user's intent and decides whether to respond directly or use an external tool, like calling an API to check the weather or book a reservation (Beyond the Demo: Building Voice Agents That Meet Enterprise Requirements).
  6. Response Generation. The LLM doesn't generate its entire response at once. It streams its answer as text fragments (tokens, words, or sentences). This allows the next stage to begin working immediately.
  7. Streaming TTS. As soon as the first text fragment arrives from the LLM, the TTS system converts it into an audio chunk (How to Build a Real-Time Voice Agent). This process continues in parallel with the LLM's generation, a key technique for minimizing response delay.
  8. Playback & Barge-In Control. The system plays the synthesized audio chunks to the user. Critically, it continues listening while speaking. This "full-duplex" capability allows it to detect if the user starts talking again, an event known as barge-in or interruption.
  9. Safety & Policy Enforcement. Before taking any action, a validation layer checks the LLM's instructions. This layer can redact sensitive data, confirm that a tool call is safe, and handle errors gracefully, ensuring the agent acts reliably.
  10. Telemetry & Observability. To monitor and improve the agent, data is logged at every stage. This includes latency measurements, transcription accuracy, interruption events, and tool success rates. This data is vital for diagnosing problems and optimizing performance.

Tracing a Request: A Worked Example of a Latency Budget

To make this concrete, let's trace a user's request and see how delays add up. The key metric is Time to First Audio (TTFA): the time from when the user stops speaking to when the agent begins its response. A low TTFA makes the agent feel responsive. Let's use the query: "What's the weather in San Francisco?"

Here is an illustrative latency budget for a well-optimized system.

  • User Finishes Speaking (Time = 0ms) The final sound wave for "Francisco" enters the microphone.

  • Audio Capture & Transport (40ms) The last audio packet is buffered on the user's device and sent over the network to the agent's servers.

  • Turn Detection (50ms) The system analyzes the incoming audio stream and a brief moment of silence, confidently deciding the user has finished their turn.

  • Streaming STT & Final Transcript (200ms) The STT model processes the final audio packets and finalizes the transcript: "What's the weather in San Francisco?".

  • LLM Time to First Token (150ms) The complete prompt is sent to the LLM. The LLM processes it and generates the first part of its response, like the text "The weather in". This is often the biggest source of delay.

  • Streaming TTS Time to First Audio (120ms) The first text fragment is immediately sent to the TTS model, which synthesizes it into a playable audio chunk.

  • Playback Buffer (20ms) The first audio chunk arrives at the user's device and is briefly buffered to ensure smooth playback without stuttering.

  • Agent Starts Speaking (Total TTFA = 580ms) The user hears the beginning of the response. The total delay is the sum of all preceding steps: 40 + 50 + 200 + 150 + 120 + 20 = 580ms.

This sub-second response time is competitive. One published self-hosted implementation reported a best-case TTFA of 729 ms under their test conditions (Building Enterprise Realtime Voice Agents from Scratch). In a streaming pipeline, these steps overlap significantly; the TTS model starts working long before the LLM has finished its complete thought (Building Enterprise Realtime Voice Agents from Scratch).

How Do Voice Agents Handle Interruptions and Barge-In?

A voice agent handles interruptions by listening and speaking at the same time, a capability known as full-duplex communication. This allows a user to "barge in" and speak over the agent, which is crucial for natural conversation. This process is managed by a state machine that controls the agent's behavior.

We can model the agent's state with four distinct modes:

  1. LISTENING: The agent is silent and actively processing the user's speech.
  2. THINKING: The user has finished speaking, and the agent is processing the request with the LLM or external tools.
  3. SPEAKING: The agent is playing back a synthesized audio response while also listening for the user to speak.
  4. INTERRUPTED: The agent has detected user speech while it was in the SPEAKING state.

When a user interrupts, the agent's state transitions from SPEAKING to INTERRUPTED. This transition triggers a cascade of cancellation signals. First, the audio playback is immediately stopped. Second, any TTS generation that is still in progress is aborted. Third, a signal is sent to stop the LLM from generating more text and to cancel any tool calls that have not yet completed.

Once all outgoing activity is halted, the system transitions back to the LISTENING state to process the user's new utterance. A common bug in simpler systems is failing to cancel this stale work, which causes the agent to speak its old, irrelevant response after the user has already moved on.

Architectural Trade-offs: Cascaded vs. Speech-to-Speech

Engineers building voice agents face a primary architectural choice between a cascaded pipeline and an end-to-end speech-to-speech model. The cascaded STT → LLM → TTS architecture is the current industry standard, prized for its modularity. Emerging speech-to-speech (S2S) models, which directly transform input speech to output speech, offer a different set of trade-offs.

Feature Cascaded (STT → LLM → TTS) Speech-to-Speech (S2S)
Modularity High. Each component (STT, LLM, TTS) can be swapped or tuned independently. Low. Often a single, monolithic model. Harder to replace just one part.
Debuggability High. Text transcripts between stages provide clear points for inspection and logging. Low. It is a "black box" from speech-in to speech-out, making it difficult to diagnose errors.
Tool Integration Straightforward. The LLM's text output can be parsed to trigger API calls or functions. Challenging. The model must be explicitly trained to emit special tokens or signals for tool use.
Latency Potential Moderate. Each conversion step adds latency, though streaming helps reduce it. Potentially Lower. Fewer conversion steps can lead to a faster response.
Expressiveness Limited by TTS. Emotional tone and prosody are generated by the TTS model from plain text. Potentially Higher. Can learn to replicate prosody, laughter, and other non-verbal cues directly.

The choice involves a trade-off between the control and observability of a cascaded system versus the potential for lower latency and greater expressiveness in an S2S model (Pipeline types). Cascaded systems remain a practical choice for most production applications requiring reliability and tool integration (Recent Engineering Accounts of Voice Agents).

When Is a Simple Voice Pipeline Enough?

A full production voice AI agent architecture is not always necessary. A simpler, non-streaming STT-LLM-TTS pipeline is sufficient when real-time, fluid conversation is not the primary goal. The simpler model works well for tasks that are asynchronous or have a clear, transactional structure.

Consider using a simple pipeline for:

  • Voice-based Data Entry: Transcribing and summarizing dictated notes, where the user speaks a chunk of text and waits for it to be processed.
  • Simple Command and Control: "Turn on the lights" or "Play the next song." These are one-shot commands that don't require conversational follow-up.
  • Prototyping: When building an initial proof-of-concept, a simple batch pipeline is easier to set up and allows you to focus on the core logic before optimizing for latency.
  • Accessibility Features: Reading screen text aloud on demand does not require the complexity of turn detection or interruption handling.

The complex, streaming architecture becomes essential when you need the agent to participate in a natural, back-and-forth conversation where users expect to be able to speak at any time and receive an immediate response.

Common Mistakes to Avoid

Building a responsive voice agent is challenging, and several common pitfalls can degrade the user experience. Avoiding these mistakes is critical for moving from a clunky demo to a smooth, production-ready system.

  • Focusing Only on LLM Latency: Measuring only the LLM's response time ignores delays from audio transport, STT, TTS, and network buffering. You must measure the full, end-to-end latency.
  • Not Using Streaming: Waiting for a complete transcript from STT or a complete answer from the LLM before starting the next step introduces significant, unnecessary delays.
  • Failing to Cancel Work on Interruption: If a user barges in, the agent must stop speaking and cancel any in-progress LLM or tool requests. A common failure is for the agent to finish its old thought after the user has already started a new one.
  • Poor Turn Detection: Relying on a fixed silence timer to decide when a user is done speaking is brittle. Real conversations include pauses and hesitations, and a robust system must handle them without cutting the user off.
  • Ignoring Tool and API Latency: If your agent needs to call an external API, the time it takes for that API to respond is added directly to the agent's "thinking" time. Slow tools make for a slow agent.
  • Neglecting Real-World Audio Conditions: Testing only with a high-quality microphone in a quiet room is not enough. You must test with phone-quality audio, background noise, various accents, and network packet loss.

What to learn next

Sources and further reading