Voice AI Reliability Audit: What Breaks After the Demo

A Voice AI demo usually has excellent working conditions.

One person makes one call from a quiet room. The carrier answers quickly. The AI services are healthy. Nobody talks over the greeting. Nobody hangs up at exactly the wrong moment. The transcript looks reasonable, the bot responds and everyone agrees that the hard part is done.

Then production adds timing.

Customers interrupt. Voicemail greetings sound like people. Audio arrives late or not at all. A provider sends duplicate events. The speech service slows down while a campaign launches a burst of calls. The customer hangs up while the application still believes the AI is speaking.

The system may remain technically online through all of this. That does not make it reliable.

A useful Voice AI reliability audit asks a more demanding question:

When one part of the call is late, ambiguous or wrong, does the whole system still converge on a safe and explainable result?

Reliability is a call-path property

Voice AI is often described as an application connected to a telephony provider. Operationally, it is a chain of real-time systems:

caller
  ↕
carrier / SIP trunk
  ↕
SBC or SIP proxy
  ↕
media server
  ↕
STT -> orchestration -> LLM -> TTS
  ↕
business systems and call state

Kamailio may route and protect signalling. FreeSWITCH may own channels, bridge RTP and play generated audio. A speech provider turns audio into text. An orchestration service decides what should happen next. A TTS service generates the reply. A database records what the business believes happened.

Each component can be healthy while the customer experience is broken.

The speech-to-text service can return every transcript eventually, but too late for a natural conversation. FreeSWITCH can have two healthy channels while the application has lost ownership of the call. The recording can sound clean even though the customer heard gaps during playback. The API can return 200 after a hangup command while a late event moves the call back into an active state.

This is why an audit should follow complete calls rather than review components in isolation.

Start with the customer-visible contract

Before measuring infrastructure, write down what the call is supposed to do.

For example:

• begin the conversation only after the correct answer condition;
• respond without an awkward unexplained pause;
• stop speaking when the caller interrupts;
• avoid treating a human as voicemail;
• end cleanly when either side hangs up;
• record one final business outcome;
• degrade safely when an AI dependency is unavailable.

The acceptable thresholds depend on the use case. A collections bot, an appointment reminder and an inbound support agent do not have the same conversation rhythm or error cost.

The audit should therefore avoid one universal latency or accuracy target. It should define targets per call phase, campaign and business risk, then inspect distributions rather than a single average.

A useful reliability report separates at least:

customer experience
call-control correctness
media quality
AI dependency health
capacity and admission
observability and recovery

That keeps a fast transcript from hiding a broken hangup path, or a clean SIP trace from hiding an unusable conversation.

Audit latency as a budget

“Bot latency” is not one measurement.

A caller finishes speaking, but several things must happen before the response becomes audible:

speech end detected
  -> final or usable transcript
  -> orchestration decision
  -> model response
  -> first TTS audio
  -> media playback
  -> audio heard by caller

The total delay matters to the caller. The individual steps matter to the engineering team.

Instrument timestamps around each boundary and correlate them with one application call ID. Useful measurements include:

• answer to first audible bot audio;
• caller speech end to usable transcript;
• transcript to orchestration decision;
• decision to first TTS byte;
• first TTS byte to playback start;
• interruption detected to playback stopped;
• hangup event to durable terminal state.

Do not subtract timestamps from unrelated machines blindly. Clock drift can manufacture latency that did not exist. Measure local durations where possible, synchronize clocks and record when an event was created separately from when it was processed.

Percentiles and outliers are more useful than averages. A system can feel good on most calls while a smaller group experiences long silence because one dependency occasionally stalls. Segment the results by carrier, region, codec, media node, speech provider and call phase before deciding that the issue is random.

The business question is not merely “is the model fast?” It is “where is the customer waiting, and can the system explain why?”

Dropped audio needs evidence from both sides

Dropped audio is one of the easiest problems to misdiagnose because every layer observes a different media path.

Consider:

carrier RTP <-> FreeSWITCH <-> AI media stream
                    |
                    +-> recording

A server-side recording proves that audio was available where the recording was attached. It does not prove that the carrier received the same frames with the same timing. Likewise, an STT transcript proves that some audio reached the recognizer, not that playback toward the caller was clean.

The audit should test both directions and several call phases:

• caller audio toward STT;
• TTS audio toward the caller;
• early media before answer;
• normal media after answer;
• audio during and immediately after barge-in;
• transfers, hold or other topology changes;
• the final moments before hangup.

For a correlated test call, collect:

• SIP signalling and negotiated SDP;
• packet timing and loss near the media server;
• FreeSWITCH channel UUIDs and media statistics;
• STT/TTS stream lifecycle events;
• recordings from known observation points;
• browser getStats() data when WebRTC is involved;
• application timestamps for playback and transcript events.

Then ask where the media first becomes wrong.

If inbound RTP is well-paced but the STT stream has gaps, inspect the FreeSWITCH media fork, resampling and application backpressure. If TTS bytes arrive continuously but outbound RTP is irregular, inspect playback buffering and media-thread scheduling. If packet timing is clean at the server but the browser reports concealment, inspect the WebRTC leg rather than the carrier leg.

“Audio dropped” is a symptom. “Frames stopped between the FreeSWITCH media fork and the STT WebSocket while the channel remained active” is an actionable finding.

Barge-in is distributed cancellation

Barge-in sounds like a feature: when the caller speaks, stop the bot.

In practice it is a cancellation protocol across several components.

caller speech
  -> VAD or transcript signal
  -> orchestration interruption
  -> model generation cancelled
  -> TTS stream cancelled
  -> queued audio discarded
  -> FreeSWITCH playback stopped
  -> conversation state updated

Stopping only the currently audible playback is not enough. TTS audio may already be buffered in the application or media server. The model may continue generating a response that will later be treated as completed. A delayed playback-complete event may advance the conversation after the caller has changed the topic.

A reliability audit should test:

• interruption near the start, middle and end of bot speech;
• interruption as a TTS stream is being created;
• interruption just as playback naturally completes;
• short acknowledgements such as “yes” or “right”;
• background speech, line noise and echo;
• repeated interruption before the next response begins;
• a caller who starts speaking and immediately stops.

The expected result should be explicit. Which signal is strong enough to interrupt? Is partial speech accepted? Is the abandoned bot response removed from conversation history? Can a late PLAYBACK_STOP or TTS callback restart the old turn?

Barge-in quality is partly a tuning problem, but correctness is an ownership problem. Every generated turn needs an identity. Cancellation should apply to that turn idempotently, and stale completion events should not be allowed to advance the current conversation.

AMD is a business decision, not a magic label

Answering-machine detection is often presented as a binary result:

human
machine

Production audio is less cooperative.

There are carrier announcements, long greetings, short greetings, business switchboards, silence, background television, non-native speech and voicemail systems that answer without an immediate beep. A detector can also take long enough to decide that the correct label arrives after the conversation should already have started.

The audit should treat AMD as a decision with cost and uncertainty:

human
machine
unknown
no usable audio
provider announcement

The exact labels can differ, but hiding uncertainty usually produces worse automation.

Measure:

• time from answer to decision;
• percentage of answered calls receiving a decision;
• human-to-machine false positives;
• machine-to-human false negatives;
• unknown and timeout rates;
• behavior when the detector and call state disagree;
• behavior when the beep arrives after playback has started.

Evaluate the errors by campaign economics. A false machine result may discard a real sales opportunity. A false human result may waste AI capacity or play an awkward introduction into voicemail. Those costs are not equal, and the policy should reflect that.

AMD also needs labelled review data. Detector output compared with itself is not validation. Sample real calls, apply human labels and track the confusion matrix by carrier, destination, campaign and greeting type.

Most importantly, define what happens while the detector is uncertain. The system can wait, begin a reversible greeting or follow another campaign-specific policy. “The detector eventually returned something” is not a complete call strategy.

Call-state races are normal

Telephony events do not arrive in a polite sequence.

The caller and application can hang up together. FreeSWITCH may emit events from both legs. A command can time out in the API while succeeding in the media server. A provider can reject an originate before the expected channel exists. A playback completion can arrive after the call is already terminal.

If each event directly updates the database, the final state depends on timing.

A safer lifecycle is explicit:

requested -> originating -> ringing -> answered
answered  -> ai_active
ai_active -> terminating
*         -> terminal

Media actions can have their own states and identifiers:

queued -> playing -> interrupted
queued -> playing -> completed
*      -> cancelled

The exact state names are less important than the invariants:

• terminal calls cannot become active again;
• duplicate events are safe;
• a call has one current owner;
• each media command targets the correct channel and turn;
• agent or campaign capacity is released once;
• every originate attempt converges on a recorded outcome;
• late callbacks can be identified and ignored.

Audit this with event sequences, not only happy-path calls. Duplicate hangups. Delay the application’s event consumer. Return an API timeout after the command reaches FreeSWITCH. Terminate a call while TTS is starting. Restart an orchestration worker while calls remain active.

The question is whether every sequence ends in one consistent product state, not whether every callback ran.

Scaling means CPS as well as concurrency

“How many concurrent calls can it handle?” is useful for budgeting and incomplete for engineering.

Active calls consume resources over time:

• RTP ports and media buffers;
• FreeSWITCH channels;
• SIP dialogs;
• STT and TTS streams;
• WebSocket connections;
• memory, file descriptors and recording capacity.

Calls per second concentrate work at transitions:

• SIP transactions;
• carrier admission;
• channel creation;
• SDP negotiation;
• database writes;
• AI stream setup;
• orchestration callbacks;
• initial prompts and detector startup.

A platform can hold many stable calls and still fail during a launch burst. It can also accept calls quickly and then degrade as more conversations begin transcoding, recording or streaming to AI services.

The audit workload should include:

1. A gradual ramp to examine stable active-call behavior.
2. Bursts to exercise CPS, originate correlation and dependency admission.
3. Churn with short calls, rapid answers and hangups.
4. Soak traffic to expose leaks in channels, sockets, timers and records.
5. Mixed media profiles rather than one convenient codec path.
6. Slow or unavailable dependencies to verify degraded behavior.
7. Uneven traffic across carriers, regions or media nodes.

During these tests, measure correctness alongside throughput:

originate attempts, accepted and rejected
carrier responses by class
active calls and active channel legs
event queue depth and event processing age
RTP loss, jitter and packet timing by leg
STT/TTS connection setup and failures
database transition latency and conflicts
calls missing a terminal outcome
orphaned channels, streams and recordings

CPS limits can exist at the carrier, SIP proxy, media server, application, database and AI provider. Admission control should respect the narrowest available dependency instead of accepting a call and discovering the shortage through silence.

A controlled rejection is usually easier to recover from than a connected call with no working conversation.

Test recovery, not only failure

Chaos testing is useful only when the expected recovery is defined.

For each dependency, ask:

• What does the caller hear?
• What state does the business system record?
• Are retries bounded?
• Is capacity released?
• Can the operator identify the failed boundary?
• Does the system recover automatically when the dependency returns?

Disconnect an STT stream during caller speech. Delay TTS setup. Restart an orchestration worker. Make the database temporarily slow. Reject new calls at the carrier. Interrupt connectivity between the application and FreeSWITCH without dropping the media server itself.

The correct behavior may differ by product. Some systems should apologize and retry. Some should transfer to a human. Some should terminate and schedule a follow-up. Reliability does not require pretending that every dependency is available. It requires a deliberate, testable failure policy.

What the final audit should deliver

A useful audit does not end with a generic health score.

It should leave the team with:

• a diagram of the real signalling, media and AI paths;
• a shared call ID across application, SIP, media and AI logs;
• latency distributions split by stage and traffic segment;
• evidence for dropped-audio boundaries;
• barge-in tests covering cancellation races;
• labelled AMD results tied to business cost;
• call-state invariants and failed event sequences;
• a capacity model separating CPS from active calls;
• recovery behavior for each critical dependency;
• prioritized fixes with owners and verification steps.

Every finding should connect a technical boundary to a product consequence.

“Event processing lag increased” matters because agents remained unavailable after calls ended. “Playback cancellation was late” matters because callers had to speak over the bot. “AMD false positives increased on one route” matters because real customers were removed from the conversation before the AI began.

That translation is what makes a reliability audit useful to the business rather than only interesting to the infrastructure team.

The practical standard

A reliable Voice AI platform is not one that never encounters late packets, ambiguous speech, dependency failures or duplicate events. Those are normal conditions.

It is one that:

• notices the failing boundary;
• protects the conversation where possible;
• converges on a valid call state;
• releases resources predictably;
• records an explainable outcome;
• gives the team enough evidence to improve the next call.

The demo proves that the system can work.

The audit proves that it knows what to do when timing stops being polite.