Tutorials8 min readby Jose M. Cobian

How to Reduce ElevenLabs TTS Latency in Voice AI Calls (2026 Guide)

A practical guide to minimizing ElevenLabs text-to-speech latency in production voice AI — model selection, streaming optimization, connection pre-warming, and Twilio silence threshold tuning.

TL;DR — The short answer

  • 1

    eleven_flash_v2_5 is the correct default model for real-time conversational voice AI in 2026 — its p50 first-chunk latency of 75–150ms is 2–4x faster than turbo and 5–8x faster than multilingual_v2.

  • 2

    Streaming mode is not optional for production voice AI — non-streaming adds 400–800ms of unnecessary latency for typical response lengths and makes silence-detection-induced drops significantly more frequent.

  • 3

    Twilio's silence_timeout should be set to p95 TTS latency plus a 2-second buffer; a timeout below 5 seconds with any ElevenLabs model will produce measurable silent failure rates at production scale.

  • 4

    Connection pre-warming eliminates 80–200ms of cold-start overhead and is the single highest-ROI latency optimization that requires zero changes to your TTS pipeline or model selection.

ElevenLabs model benchmarks: flash vs turbo vs multilingual

Model selection is the highest-leverage latency decision in your ElevenLabs configuration. The three models that matter for production voice AI in 2026 differ meaningfully in both latency profile and quality characteristics.
eleven_flash_v2_5 is the lowest-latency option. Under normal API load with streaming enabled, it produces first audio chunks at 75–150ms p50 for inputs under 100 characters. At 200 characters, p50 rises to 110–190ms. This model is explicitly optimized for real-time streaming — ElevenLabs' internal architecture for flash prioritizes chunk output over quality consistency, and the tradeoff is appropriate for voice AI where comprehensibility matters more than studio-grade fidelity.
eleven_turbo_v2_5 runs at 200–350ms p50 for the same input sizes. The quality improvement over flash is audible on headphones in a quiet environment and largely imperceptible on a telephone call where the audio is compressed to 8kHz PCM by Twilio's media pipeline. Teams that migrated from turbo to flash consistently report no increase in caller complaint rate. The 150–200ms latency saving is meaningful — it represents 3–4% of the typical caller's patience budget before the interaction feels unresponsive.
eleven_multilingual_v2 is a fundamentally different architecture optimized for quality and language coverage rather than latency. P50 first-chunk latency is 400–700ms; at p95 under load it can exceed 1,200ms. This model should not be in the critical path of a real-time voice AI call. If you need multilingual support, test whether flash's multilingual capability (which is more limited but present) is sufficient for your language set before defaulting to multilingual_v2.
Benchmarks to run before committing to a model: use ElevenLabs' streaming API directly (not via Vapi or Retell, which add orchestration overhead) and measure time-from-request-submission to first-byte-received across 500 requests with input lengths representative of your actual AI agent responses. Sample 10th, 50th, 90th, and 95th percentile latencies separately — the tail behavior is what determines your silence timeout configuration, and a model that looks acceptable at p50 may be unacceptable at p95.

Streaming optimization: why most teams configure it wrong

ElevenLabs streaming mode is enabled by requesting the WebSocket or chunked HTTP API endpoints instead of the synchronous generation endpoint. Most teams that think they have streaming enabled are actually using a configuration that negates most of the benefit.
The most common misconfiguration: making synchronous HTTP POST requests to the ElevenLabs generation endpoint and buffering the entire response before passing it to Twilio. This is not streaming — it is sequential HTTP with network transfer in place of generation wait time. The effective latency is nearly identical to non-streaming because you are waiting for the full audio file to transfer before beginning playback.
True streaming requires either the WebSocket API (for low-latency bidirectional communication) or the streaming HTTP endpoint with chunked transfer encoding where you begin piping audio chunks to Twilio's media stream as each chunk arrives. The key implementation requirement: your server must be able to forward audio chunks to Twilio within the same network I/O loop that receives them from ElevenLabs — no buffering, no synchronous processing, no database writes in the hot path.
For Vapi and Retell integrations: both platforms handle ElevenLabs streaming internally. Verify in your Vapi dashboard or via the Vapi API that the voice.chunkPlan setting is configured appropriately for your latency target. Vapi's default chunk plan accumulates tokens before sending to ElevenLabs to improve prosody — this adds 100–300ms to first-chunk latency in exchange for more natural speech rhythm. For maximum speed, set the chunk plan to send on first token. For Retell, the equivalent setting is the response_delay parameter.
A practical test to verify you have genuine streaming configured: instrument your server to log the timestamp when the first audio byte is received from ElevenLabs and the timestamp when the first audio byte is forwarded to Twilio. The delta should be under 10ms if streaming is working correctly. A delta above 50ms indicates buffering somewhere in your pipeline.

Connection pre-warming: eliminating cold-start latency

Every ElevenLabs WebSocket connection goes through TLS handshake, HTTP upgrade, and initial frame exchange before it can accept a synthesis request. On a connection to ElevenLabs' API servers from a US-East server, this overhead is typically 80–140ms. From EU-West it runs 100–180ms. This overhead is incurred once per connection — if you establish the connection before you need it, the overhead is paid off the critical path.
The pre-warming pattern for Twilio inbound calls: when Twilio fires your TwiML webhook (the initial call arrival event), immediately initiate the ElevenLabs WebSocket connection in parallel with generating your TwiML response. By the time the caller hears your first prompt and the AI processes their response (typically 3–8 seconds), the connection is warm and the synthesis request goes straight to generation. The cold-start cost is eliminated on every call.
For outbound AI dialers built on Twilio Programmable Voice: pre-warm the connection when Twilio fires your call_initiated event, before the call is answered. The pre-warming completes during the ring cycle, meaning the connection is warm before the called party picks up.
Connection pool management for high-volume deployments (above 20 concurrent calls): maintain a pool of pre-warmed connections sized to your expected concurrency. Measure your p99 call duration to set pool connection TTL — closing connections that have been idle longer than your p99 call duration prevents accumulating stale connections that have been closed server-side. ElevenLabs does not publish a maximum connection idle time, but empirical observation suggests connections held idle for more than 60 seconds may be closed server-side without notification. Set your client-side ping interval to 45 seconds to keep connections alive.
The ROI on connection pre-warming is highest when your call volume is under 5 concurrent calls — at that scale, connection pooling cannot amortize the cold-start cost, and pre-warming per-call is the only available mechanism.

Twilio silence_timeout tuning for ElevenLabs response times

Twilio's silence detection mechanism terminates a or media stream session when it detects silence for longer than the configured threshold. The default configuration — 5 seconds in most contexts — was designed for DTMF collection from human callers who press a key within 2–3 seconds. It is not appropriate for voice AI deployments where the AI's response generation takes 250–1,200ms and the audio streaming adds additional latency.
The correct approach to silence_timeout configuration is empirical: measure your actual end-to-end latency distribution (LLM response time + TTS first-chunk time + streaming buffer to Twilio), then set the timeout to accommodate your p95 case with a 2-second safety margin.
For a typical eleven_flash_v2_5 deployment with a fast LLM (GPT-4o-mini or Claude Haiku, ~200ms response time): - LLM response: 150–350ms p50 - ElevenLabs flash first chunk: 75–150ms p50 - Twilio media stream buffer: 30–80ms - Total p50: 255–580ms - Total p95 (under load): 600–1,200ms - Recommended silence_timeout: 5–6 seconds
For eleven_turbo_v2_5 with a more capable LLM (GPT-4o, ~800ms response time): - Total p95: 1,200–2,000ms - Recommended silence_timeout: 6–7 seconds
For eleven_multilingual_v2 (not recommended for real-time, included for completeness): - Total p95: 2,000–3,500ms - Recommended silence_timeout: 8–10 seconds
The silence_timeout parameter is set in TwiML on the verb: . For media stream configurations, the equivalent is the inactivity_timeout parameter on the verb, though the parameter name varies by SDK version. Verify which parameter your Twilio helper library version exposes.
One common mistake: setting a very long silence_timeout to 'play it safe' and then observing that callers feel the conversation is unresponsive when the AI doesn't speak. A silence_timeout above 8 seconds creates noticeable pauses that degrade caller experience. The goal is a tight, data-driven value — not a conservative maximum.

Measuring p50/p95 latency in production: instrumentation patterns

Gut-feel latency estimates are useless for tuning silence timeouts or diagnosing degradation. The only path to accurate configuration is structured measurement from production traffic.
The three timestamps you must capture for every TTS generation call: 1. request_sent_at: the millisecond timestamp at which your server sends the ElevenLabs synthesis request. 2. first_chunk_at: the millisecond timestamp at which your server receives the first audio byte from ElevenLabs. 3. last_chunk_at: the millisecond timestamp at which ElevenLabs signals stream completion.
From these three timestamps: first_chunk_latency = first_chunk_at - request_sent_at. This is the metric you care about for silence timeout tuning. total_generation_time = last_chunk_at - request_sent_at is useful for cost analysis (it correlates with character count) but not for latency configuration.
Store these measurements with the associated call SID (for Twilio correlation) and model name (to compare models in production). A minimal schema: {call_sid, elevenlabs_session_id, model, input_char_count, first_chunk_latency_ms, total_generation_ms, timestamp}. Index on timestamp and model for efficient percentile queries.
Calculate percentiles weekly and compare to the prior week. A p95 that is rising week-over-week without changes to your model or input distribution indicates either increased API load on ElevenLabs' servers or growth in your average response length. Both are actionable: ElevenLabs load issues can be mitigated by switching to a lower-latency model or timing major campaigns to avoid peak periods; response length growth can be corrected in your LLM prompt.
For Vapi and Retell deployments, these platforms log their own latency metrics in call event webhooks — but they measure LLM-to-TTS pipeline time, not raw ElevenLabs generation time. To get the raw ElevenLabs number, you need a custom proxy or to pull from ElevenLabs' history API, which stores generation latency per request.

Advanced: request batching and input length optimization

ElevenLabs latency scales with input length — a 400-character input takes approximately 2.5x longer than a 100-character input at first-chunk latency, because the model must process more tokens before it can begin generating the first phoneme. This means that AI agent verbosity is a direct latency lever, and managing it is part of voice AI performance engineering.
In practice, the target input length for minimum latency is under 150 characters for the first response chunk. This is not always achievable — some responses are necessarily longer — but LLM prompt engineering to produce concise first-sentence responses before longer elaborations can meaningfully reduce the first-chunk latency for the opening of each turn. In sentence-streaming architectures (where the LLM generates a sentence, sends it to TTS, speaks it, then generates the next sentence), this is built-in. In full-response architectures, it requires prompt-level work.
Sentence streaming architecture for ElevenLabs: instead of waiting for the LLM to complete its full response, detect sentence boundaries in the LLM's output stream and begin synthesizing each sentence as it completes. The caller hears the first sentence within 75–200ms of the LLM generating its first period, rather than waiting for the full multi-sentence response to complete. This architecture effectively reduces the input-length-to-first-audio relationship to the length of the first sentence, regardless of total response length.
Implementation: pipe LLM streaming output through a sentence boundary detector (look for ., !, ? followed by whitespace), accumulate tokens into a sentence buffer, and flush to ElevenLabs when a boundary is detected. Queue subsequent sentences for synthesis while the first is playing. The complexity is in managing the queue — if the second sentence synthesizes faster than the first plays, you can begin streaming it immediately; if it is slower, you need a brief gap strategy (a pause, a filler phrase, or silence) to avoid jarring interruptions.

How Sherlock surfaces ElevenLabs latency spikes automatically

Even with optimal model selection, streaming configuration, and connection pre-warming, ElevenLabs latency spikes occur. API load events, ElevenLabs infrastructure incidents, and traffic distribution issues can push p95 latency above your silence timeout threshold without any configuration change on your side.
The detection problem: ElevenLabs does not publish granular real-time latency metrics on their status page. The first signal you typically receive is an increase in short-duration 'completed' calls in your Twilio logs — silence-detection-induced drops that appear 3–6 hours after the latency spike began. By that point, you have already had hundreds of failed caller interactions.
Sherlock monitors this correlation automatically: it pulls Twilio call duration distributions and ElevenLabs generation latency from the history API on a rolling basis. When p95 first-chunk latency crosses the threshold relative to your configured silence timeout — or when short-duration completed calls spike above baseline — Sherlock posts a case file in Slack identifying the specific time window, the affected calls, and the measured latency increase. The median time from latency spike to Slack alert is under 8 minutes. The alternative — waiting for a customer complaint or a billing anomaly — has a median detection time of 4–6 hours.
Connect your ElevenLabs account at [usesherlock.ai](https://usesherlock.ai/?utm_source=blog&utm_medium=content&utm_campaign=elevenlabs-latency-guide) to enable automatic latency monitoring alongside Twilio call quality tracking.

Explore Sherlock for your voice stack

TwilioElevenLabsVapiRetell AIBland AIGenesys

Frequently asked questions

What is normal ElevenLabs TTS latency in production?

Under normal API load, eleven_flash_v2_5 produces first-chunk latency of 75–150ms for inputs under 100 characters when streaming is enabled. eleven_turbo_v2_5 runs 200–350ms for the same input size. eleven_multilingual_v2, the highest-quality model, averages 400–700ms first-chunk latency. These are p50 figures — at p95 during peak load periods, add 30–60% to each. Your effective latency budget also includes the WebSocket roundtrip to your server (15–40ms typical on a well-located server) and Twilio's media processing overhead (30–80ms). A realistic end-to-end p50 for flash with streaming enabled is 250–350ms measured from the moment the LLM generates the last token to the moment audio starts playing on the caller's handset.

How do I choose the right ElevenLabs model for a voice AI call?

For real-time conversational AI where caller experience depends on fast response, eleven_flash_v2_5 is the correct default in 2026 — it has the lowest latency of any available model and sufficient quality for speech comprehension. Reserve eleven_turbo_v2_5 for use cases where the slight quality improvement is worth 150–200ms of additional first-chunk latency — for example, high-stakes enterprise calls where voice quality signals credibility. Avoid eleven_multilingual_v2 in real-time conversation paths unless you require non-English language support that flash does not cover; its latency profile makes it unsuitable for low-silence-threshold configurations. When your voice AI runs through Vapi or Retell, these platforms expose their own model selection settings — ensure you are setting ElevenLabs model selection at the platform level, not relying on platform defaults, which may lag behind ElevenLabs' latest model releases.

What silence timeout should I configure in Twilio for voice AI?

The correct silence timeout depends directly on your p95 end-to-end TTS latency. Measure p95 TTS latency for your specific model and input-length distribution over at least 1,000 calls. Your silence_timeout should be set to p95_latency + 2,000ms minimum — giving your AI agent time to generate and begin streaming audio even on a slow tail-latency call. A common production configuration for eleven_flash_v2_5 with streaming enabled is silence_timeout=7 seconds. Setting it below 5 seconds with any ElevenLabs model other than flash is likely to produce silence-detection-induced call drops on p95+ calls, which will show in your logs as 'completed' calls with anomalously short duration.

When should I use ElevenLabs streaming vs non-streaming?

Always use streaming for real-time voice AI calls. Non-streaming mode requires ElevenLabs to generate the complete audio file before returning any bytes — for a 200-character response this adds 400–800ms of latency on top of the generation time. Streaming mode begins returning audio chunks as soon as the first phoneme is synthesized, typically within 75–150ms of request submission for flash. The only case to consider non-streaming is when you need the complete audio duration before the call (for example, to mix it with background audio or to generate a time-synchronized transcript). For any configuration where the caller is waiting for the AI response in real time, streaming is mandatory for acceptable latency.

Does connection pre-warming actually reduce ElevenLabs latency?

Yes, measurably. ElevenLabs WebSocket connections include a TLS handshake and HTTP upgrade that adds 80–200ms to the first request on a cold connection. By establishing the WebSocket connection 500ms before you expect to need it — at the moment the inbound call arrives, for example, rather than at the moment the LLM first responds — you eliminate this cold-start overhead from the latency-critical path. Connection pre-warming is most impactful when your call volume is low enough that connections cannot be pooled (fewer than 20 concurrent calls). Above 20 concurrent calls, maintaining a warm connection pool sized to your concurrency level produces the same result more efficiently.

Share

Ready to investigate your own calls?

Connect Sherlock to your voice providers in under 2 minutes. Free to start — 100 credits, no credit card.

Start for free
← Back to the blog