Conversations with AI can feel slow. The unnatural pauses between you finishing a sentence and the agent starting its reply can break the flow and frustrate users. This delay, known as voice agent latency, is one of the biggest challenges in building believable conversational AI.
To solve it, engineers use a latency budget: a plan that allocates a specific amount of time to each step in the agent's response pipeline. This article explains how to create and use one.
Key takeaways
- Voice agent latency is the total time from when a user stops talking to when the agent's first audible response begins.
- A good budget plans for tail latency (the slowest 5-10% of responses), not just the average. Users remember the long pauses, not the typical ones.
- Significant sources of delay include fixed costs like endpointing, network round-trips, and database queries, which are often underestimated compared to AI model inference.
- Effective optimization relies on streaming and parallelism. You must start synthesizing and playing audio before the full text response is ready.
- You cannot fix what you cannot measure. Instrumenting every stage of the pipeline with precise timestamps is the essential first step to diagnosis.
What is Voice Agent Latency? (And Why Every Millisecond Matters)
Voice agent latency is the time between a user finishing their turn and the AI agent starting its own. This end-to-end delay is critical because it determines if a conversation feels natural or stilted. Delays above 4 seconds degrade the user experience, making the agent seem unresponsive and unintelligent (Source).
The gap between human and AI conversation speed is significant. One study found a median agent response time of 4.1 seconds, compared to just 1.2 seconds for humans in similar interactions (Source). While many production systems target a 1-2 second latency, achieving it requires careful engineering (Source).
This user-perceived latency is different from component-level metrics. Metrics like a model's "time to first token" (TTFT) or simple network ping time are parts of the whole, but they don't capture the full picture. The only measurement that truly matters is the one the user experiences: the silence after they speak.
Where Does the Time Go? A Step-by-Step Latency Breakdown
Achieving a sub-second response requires understanding every component that adds delay. A typical voice agent turn consists of several operational stages, each consuming a piece of your latency budget. In one reported system, the average total latency was 0.94 seconds per turn (Source). Here’s where that time goes.
Endpointing: Before anything else, the system must decide the user has finished speaking. This is often done by detecting a period of silence. A conservative endpointer that waits for excessive silence can consume a large part of the budget, adding a fixed latency cost at the start of the process.
Speech-to-Text (ASR): The user's final audio is transcribed into text by an Automatic Speech Recognition (ASR) model. Modern systems use streaming ASR, which provides partial transcripts as the user speaks. Acting on a stable partial transcript can save time, but waiting for the final, most accurate version adds delay. In one study, the mean ASR latency was about 50 milliseconds (Source).
Network Transport (Uplink): The final transcript text must be sent from the ASR service to the core agent logic. This network round trip contributes uplink, serialization, and queueing delay. For short user utterances, this transport time can dominate the request (Source).
Agent Logic & Tool Calls: The agent processes the text to understand intent. This stage may involve calling external tools, like checking a database or an API. For agents using retrieval-augmented generation (RAG), a vector database query alone can add 50 to 300 milliseconds before the main AI model even sees the request (Source).
LLM Generation: A large language model (LLM) generates a text response. For voice agents, the most important metric here is time to first token (TTFT)—how quickly the model produces the first piece of its response. The total time to generate the full sentence is less critical if you can start the next step immediately. Still, this is often a large variable component, with one paper reporting a mean of 670 ms (Source).
Text-to-Speech (TTS): The first few words from the LLM are sent to a Text-to-Speech (TTS) engine to be converted into audio. Like ASR and LLM generation, this process must be streamed. The key metric is time to first audio byte (TTFAB). A fast TTS system can begin producing audio in under 50 milliseconds from receiving text (Source).
Network, Buffering & Playback: The first chunk of audio data travels over the network to the user's device. The device's audio client then loads it into a small buffer, called a jitter buffer, to ensure smooth playback without glitches. This downlink transport and client-side buffering directly increase perceived response delay.
Barge-in Detection: During the agent's speech, the system must constantly listen for the user to interrupt (or "barge in"). If detected, it must immediately stop generation, synthesis, and playback. The time it takes to stop is a separate but equally critical metric called stop latency.
A Practical Voice Agent Latency Budget (Worked Example)
A latency budget forces you to plan for delays and identify which components need optimization. This budget should cover start latency—the time to begin a response. Crucially, you must budget for both the typical case (median or p50) and the worst-case scenario (tail latency, like p95). A system that feels fast on average but has frequent, jarringly slow responses offers a poor user experience.
Here is an illustrative budget for a voice agent targeting a median start latency of around 800 milliseconds.
| Pipeline Stage | Median (p50) Budget (ms) | Tail (p95) Budget (ms) | Notes |
|---|---|---|---|
| 1. Endpointing | 300 | 350 | Fixed cost based on silence detection threshold. Relatively stable. |
| 2. ASR (Finalization) | 50 | 150 | Time to get a final, stable transcript after endpointing. |
| 3. Network to Agent | 40 | 150 | Round-trip time for the transcript to reach the agent logic. |
| 4. Tool Call (RAG Query) | 100 | 300 | A database query can have high variability under load (Source). |
| 5. LLM (Time to 1st Token) | 200 | 600 | The largest source of variability. Depends on model load and context. |
| 6. TTS (Time to 1st Audio) | 50 | 150 | Includes TTS model inference and audio encoding (Source). |
| 7. Network & Playback | 60 | 250 | Transport for the first audio chunk plus client-side jitter buffering. |
| Total (Sequential Sum) | 800 ms | 1950 ms | This simple sum is misleading. |
Notice two things. First, the sum of the p95 budgets is over double the median. If you only design for the average, nearly 1 in 20 conversations will feel extremely slow. Second, simply adding the stages is an oversimplification. A well-architected system overlaps these stages. For instance, the LLM can start processing a partial ASR transcript, and TTS can begin before the LLM has finished its full thought. A production study demonstrated a 41.8% latency reduction just by overlapping two stages (Source).
How Do You Measure Voice Agent Latency Accurately?
Accurate measurement is the foundation of optimization. You need to log precise timestamps at every boundary in your pipeline for every single conversational turn. Collecting this data allows you to calculate the duration of each stage and identify bottlenecks. A complete Voice AI Agent Architecture: A Complete Guide must include this level of detailed instrumentation.
Here are the critical timestamps you must capture:
user_speech_end_ts: When the VAD/endpointing logic confirms the user has stopped speaking. This is the starting line for end-to-end latency.asr_final_ts: When the final ASR transcript is available.agent_request_ts: When the agent logic begins processing (e.g., before any tool calls).llm_first_token_ts: When the first text token is received from the LLM.tts_first_audio_ts: When the first audio chunk is received from the TTS service.client_playback_start_ts: When the user's device actually starts playing audio. This is the finish line.
With these timestamps, you can calculate key performance indicators (KPIs). The following pseudo-code shows how to calculate end-to-end latency and the duration of a single stage.
# Illustrative calculations from logged timestamps
# End-to-end user-perceived latency
e2e_latency = client_playback_start_ts - user_speech_end_ts
# Latency of the LLM generation stage (TTFT)
llm_latency = llm_first_token_ts - agent_request_ts
# Latency of the TTS stage (TTFAB)
tts_latency = tts_first_audio_ts - llm_first_token_tsBy logging these values for thousands of turns, you can build a statistical model of your system's performance. This allows you to track not just the average (p50) latency but also the tail at p95 and p99, which is where the real problems often hide.
Common Mistakes That Ruin Your Latency Budget
Many teams struggle with voice agent latency because they fall into common traps. Avoiding these pitfalls is as important as choosing the right models.
- Measuring the Wrong Interval: Focusing on server-side "request-to-response" time ignores the user's reality. The only metric that matters is
end-of-speech-to-first-audio. - Ignoring the Tails: Designing for the average (median) latency guarantees a poor experience for a significant fraction of your users. A system with a 700ms median but a 3-second p95 latency will feel slow.
- Sequential Thinking: Building a pipeline where each step waits for the previous one to complete is a recipe for high latency. You must use streaming and parallelism to overlap ASR, LLM, and TTS processing.
- Overlooking Fixed Costs: Underestimating the impact of non-model delays is a frequent error. A 300ms endpointing delay and a 200ms database query can consume half of a sub-second budget before the LLM even starts.
- Optimizing the Wrong Component: Teams can spend weeks shaving 20ms off a fast TTS model when a slow, unoptimized API call is the real bottleneck. Accurate instrumentation prevents this by showing you exactly where the time goes.
Start Latency vs. Stop Latency: Two Sides of Responsiveness
A responsive agent isn't just one that starts talking quickly; it's also one that stops quickly. This introduces two distinct types of latency that both affect conversational quality (Source).
- Start Latency is what we've primarily discussed: the delay from the end of the user's utterance to the beginning of the agent's audible response.
- Stop Latency (or Barge-in Latency) is the delay from when a user interrupts the agent to when the agent's audio playback actually ceases. High stop latency makes an agent feel like it's talking over the user, which is a major source of frustration.
Both metrics must be measured and managed. An agent can have excellent start latency but poor stop latency, making it feel fast to reply but slow to listen. A complete latency strategy requires a separate budget for each.
Cascaded vs. Speech-to-Speech: Which is Faster?
Two main architectures are used for voice agents today, each with different latency characteristics. Understanding what an AI agent is at this architectural level is key.
A cascaded system is the traditional approach, using separate, specialized models for Speech-to-Text (ASR), the language model (LLM), and Text-to-Speech (TTS). A speech-to-speech (S2S) system uses a single, end-to-end model that takes audio in and produces audio out, bypassing the intermediate text step.
| Feature | Cascaded (ASR → LLM → TTS) | Speech-to-Speech (S2S) |
|---|---|---|
| Latency Source | Handoffs between components, plus each component's latency. | Single model inference. Potentially lower overhead. |
| Optimization | Can be optimized via streaming and parallelism between stages. | Optimized by making the single model faster (e.g., quantization). |
| Inspectability | High. Easy to debug the text transcript at each stage. | Low. It can be a "black box," making it harder to debug or add tools. |
| Tool Use | Straightforward. Agent can pause at the text stage to call APIs. | More complex. Requires specialized models or techniques. |
| Flexibility | High. Can swap out individual components (e.g., use a new TTS). | Low. The entire model is monolithic. |
There is no universally "faster" option. While S2S models theoretically reduce latency by eliminating handoffs, well-optimized cascaded systems that use aggressive streaming can be extremely fast. Today, cascaded systems are more common and flexible, especially for complex agents that need to use tools or follow business logic.
When Is a Higher Latency Acceptable?
While the goal is often to minimize latency, the lowest possible delay is not always necessary or even desirable. The acceptable latency depends heavily on the context of the conversation and the user's expectations.
For simple, transactional queries like "What time is it?" or "Turn on the lights," users expect an immediate response. A delay of more than a second can feel awkward.
However, for more complex tasks, a longer pause can feel natural. If you ask an agent to "summarize the last three emails from my manager and list the action items," you implicitly understand that this requires more work. A 2-3 second pause can signal that the agent is "thinking" and processing the request. In these cases, rushing to give an answer might feel less reassuring than a thoughtful pause. This is particularly relevant when deploying AI agents for finance or insurance where accuracy is critical.
The key is to match the agent's response time to the perceived complexity of the task. Some systems even use conversational fillers ("Okay, let me check that for you...") to manage user perception during longer delays, though this must be done carefully to avoid sounding repetitive (Source).
What to learn next
- Outbound AI Calling Compliance: A Technical Guide - A technical guide to outbound AI calling compliance. Learn to design a compliant workflow, structure a valid consent record, and navigate US
- Voice AI vs IVR: A Technical Comparison - A technical comparison of Voice AI vs IVR, explaining architectures, capabilities, and when to use each. Learn why they are not mutually exc
- Testing Voice Agents: A Complete Step-by-Step Guide - Learn how to test voice agents effectively with our step-by-step guide. We cover everything from scenario design and audio quality to regres
- Speech to Speech vs Cascaded Voice AI: Which is Better? - Learn the difference between speech to speech vs cascaded voice AI. Compare latency, control, and cost to choose the right architecture for
- Streaming TTS Latency: A Guide to Under 200ms Audio - Learn how to measure and optimize streaming TTS latency from end to end. This guide explains the full pipeline from text generation to first
- Voice AI Agent Architecture: A Complete Guide: Dive deeper into the components that make up a voice agent.
- AI Agents for Real Estate: A Practical Guide: See how these concepts apply in a specific industry.
Sources and further reading
- An End-to-End Turn-Taking Model for Spoken Dialog Systems
- Design and Evaluation of a Voice-Based Conversational E-Commerce Agent
- A Taxonomy of Real-World Conversational AI: An Exploratory Analysis of Production Agent Interactions
- Augmenting Voice-enabled Conversational Agents with Contextual Information
