The ten questions your operation must answer on demand
Question 1: What is your current call failure rate, broken down by provider — right now, not from yesterday's report?
Question 2: What is your average cost per converted call this week versus last week? Not cost per call — cost per converted call.
Question 3: What happened on your last ten failed calls? Not the error code — the full sequence of events across every provider, timestamped.
Question 4: Which agent configuration is your most expensive per call, and what is its conversion rate?
Question 5: What is your average TTS latency this month relative to your configured silence threshold?
Question 6: Which calls this week had transcript quality below your defined minimum standard?
Question 7: What is your on-call response time for voice AI incidents, and what information does the responding engineer have within the first five minutes?
Question 8: What is your current provider spend rate relative to your monthly budget? Which providers are on track to exceed budget?
Question 9: Which callers experienced more than one failed AI interaction in the last 30 days?
Question 10: Can you answer all nine questions above in under two minutes, from your phone, without a spreadsheet? If the answer to the tenth is no, the other nine are academic.
Why each question is on the list
Each of these ten questions corresponds to a failure mode that caused measurable, documented business damage for voice AI teams over the course of 2025. They are not theoretical — every one of them is on the list because a team somewhere did not have the answer and paid for it.
Failure rate by provider is on the list because aggregate failure rates conceal provider-specific problems. A 7% aggregate rate hiding a 15% ElevenLabs rate and a 2% Twilio rate requires a provider-specific intervention, not a system-level change. Without the breakdown, you cannot act correctly.
Cost per converted call — not cost per call — is on the list because the two metrics point in opposite directions often enough to make cost-per-call optimization actively dangerous. The question about the last ten failed calls is on the list because a team that cannot reconstruct what happened on a failed call cannot prevent it from recurring. The question about callers with multiple failures is on the list because repeat failures on the same account are the strongest predictor of churn and require proactive outreach — but only if you know who they are.
The four dimensions these ten questions cover
The ten questions map to four operational dimensions. Cost visibility (questions 2, 4, 8) ensures you understand where your provider spend is going at the call level, not just the invoice level. Failure attribution (questions 1, 3, 5) gives you the evidence needed to diagnose incidents to root cause without manual log correlation. Quality measurement (questions 6, 9) surfaces the quality signals that predict conversion outcomes weeks before they appear in sales numbers. Incident response readiness (questions 7, 10) measures whether your operational posture is actually functional under adversarial conditions.
A team with complete coverage across all four dimensions is not just well-monitored. It is well-positioned to make correct configuration decisions, because every decision has immediate, measurable feedback. Change the system prompt and see transcript quality score update within 24 hours. Switch TTS models and see the latency distribution change in the same day. Deploy a new agent configuration and see cost per converted call before the week is out.
Without this feedback loop, configuration decisions are made on intuition and validated by waiting for customer complaints. With it, every operational decision is an experiment with a measurable outcome.
What the inability to answer them costs you
If your team cannot answer these ten questions in under 30 seconds, you have an observability gap. Observability gaps compound at the rate of your call volume. Every week you run without per-provider failure attribution is a week where a provider degradation is being averaged into your aggregate and treated as background noise. Every week without cost-per-converted-call tracking is a week where your most expensive, least effective agent configuration continues operating without review.
The math is specific. A team running 1,000 calls per week with a 5% provider-specific failure rate that is hidden in the aggregate — visible only through per-provider attribution — is losing 50 calls per week to a preventable failure. At an average conversation value of €600, that is €30,000 per month in silent revenue loss. The observability gap that conceals it costs nothing to close and is costing €30,000 per month to maintain.
The good news: closing the observability gap does not require a six-month internal engineering project. It requires connecting a multi-provider observability tool to your existing providers — a setup that takes hours, not months. The evidence is there in your providers' logs. What every voice AI operation needs in 2026 is a detective capable of reading it.
Frequently asked questions
What does voice AI observability mean in practice?
Voice AI observability means being able to answer operational questions about your AI calling system in near-real-time, without manual log analysis. The practical test: can your team answer 'what is my current failure rate by provider?' and 'what happened on the last ten failed calls?' in under 30 seconds, from a mobile device, at 11 PM on a Friday? If not, you have an observability gap.
What is the minimum observability stack for a production voice AI deployment?
Minimum viable voice AI observability requires: real-time failure rate alerts by provider (not just aggregate), per-call cost tracking with budget threshold alerts, cross-provider call event correlation for incident investigation, and transcript availability for quality auditing. These four capabilities cover 85% of the operational questions that arise in production. Everything beyond this is optimisation.
How do you close a voice AI observability gap without a six-month project?
Connect a multi-provider observability tool to your existing providers — Twilio, ElevenLabs, Vapi, Retell, your CRM. Most teams can achieve full cross-provider correlation within 2–3 days of setup, not months. The long-path option is building custom ETL pipelines to aggregate logs from each provider into a BI tool — functional but expensive to build and maintain. The fast path is using a tool designed specifically for voice AI provider correlation.
Ready to investigate your own calls?
Connect Sherlock to your voice providers in under 2 minutes. Free to start — 100 credits, no credit card.