A correct transcript and a successful tool call can still belong to a conversation that went badly. The system may respond during a thinking pause, continue talking over a correction, or confirm an action before the external service has reported its outcome.
These failures are often hidden by an ordinary-looking transcript. What matters is the timing of each event, what the user was able to hear, and which operations continued after the conversation changed. Noisy audio, different speaking styles, longer sessions, slow dependencies, and unexpected interruptions bring those problems into the same interaction.
Diagnosing them means identifying where the sequence first went wrong, how that decision affected later stages, and which component owns the resulting state.
This is the first of two articles examining where voice agents fail in production, with a focus on latency, turn-taking, interruptions, and conversation state.
A voice agent is a pipeline
A typical voice interaction can pass through several stages:
audio capture → speech detection and turn detection → speech recognition → dialogue/orchestration → model inference → tool calls → speech synthesis → playback
Network transport, buffering, session state, retries, cancellation, and fallback behavior sit around that sequence. Some architectures separate recognition, inference, and synthesis. Others use a realtime speech model, and hybrid implementations combine both approaches. A published engineering account of GPT-Live, for example, describes continuous audio processing with asynchronous reasoning and tool work.
The stages are rarely as sequential as the diagram suggests. A partial transcript can change while a response is prepared. Synthesis can start before the complete response exists. The user may speak while the previous response is playing, or a tool may finish after the instruction that initiated it has been revised.
That overlap is why a local improvement can create a downstream error. Starting the model earlier may reduce silence, but it also increases the chance of generating from an incomplete turn. Stopping playback can help with an interruption while leaving a booking request untouched.
Latency is cumulative
A user finishes a question and waits. The silence is often blamed on model inference, although the model may account for only a fraction of it. The system could still be deciding whether the user has stopped, waiting for a transcript, contacting a tool, synthesizing speech, or preparing the next audio packet for playback.
A useful latency trace separates those delays:
| Measurement | What it can reveal |
|---|---|
| End of speech → turn committed | Delay introduced by endpointing and turn detection |
| End of speech → final transcript | Speech recognition finalization delay |
| Model request → first output | Initial model inference delay |
| Tool start → tool result | External dependency latency |
| Speech synthesis request → first audio | Synthesis startup delay |
| End of speech → playback begins | User-visible response-onset delay |
Figure 1. How overlapping stages contribute to voice-agent response latency.
Production instrumentation can expose these distinctions. The observability hooks in LiveKit, for example, record an end_of_utterance_delay alongside other transcription and model metrics. A slow model first-output measurement points to a different investigation from a long gap between detected speech end and turn commitment.
Transport adds delay of its own. Network jitter can increase the time audio spends in a playback buffer, while packet loss can degrade audio received by the listener or recognition system. On WebRTC connections, jitter-buffer delay and packet-loss statistics help distinguish network problems from inference or synthesis delays. With audio carried over WebSockets, TCP retransmissions can also delay later packets behind a lost one.
Human conversational timing offers context without providing a universal service-level target. A review by Levinson and Torreira describes gaps between turns as being on the order of 200 milliseconds, although speech production typically takes more than 600 milliseconds. People often prepare replies before the previous speaker finishes, which helps explain why fully sequential agent pipelines can feel slow.
A short social reply, a database lookup, and a multi-step booking will still have different acceptable delays. Streaming can reduce response-onset time by overlapping stages. Speculative model generation may start from a final transcript before the turn is fully committed; some systems can prepare synthesis early as well. Both techniques risk wasted work when the user continues speaking or the transcript changes.
Average latency alone hides these tradeoffs. Track tail response times and attribute the slow interactions to individual stages, including the transport, tools, endpointing, and playback path.
Knowing when someone has finished speaking
Consider the request:
“Can you move my appointment to... Thursday afternoon?”
The pause after “to” contains no speech, but the turn is incomplete. Detecting speech activity and deciding that a speaker has finished are separate problems. VAD identifies likely speech and silence; endpointing estimates boundaries; higher-level turn detection may also consider linguistic, acoustic, and contextual information.
Figure 2. How a mid-sentence pause can cause premature turn detection or allow the user to finish speaking.
Several implementation choices are available. The turn-detection documentation for LiveKit describes semantic and acoustic turn detection alongside VAD-only, recognition-based endpointing, realtime-model detection, and manual control. Pipecat offers a different example through Smart Turn, an open source model that analyzes audio after VAD detects a pause. It estimates whether the turn is complete without requiring a transcript.
Research on Voice Activity Projection approaches the problem by predicting future speech activity and conversational events, including turn changes and backchannels. Turn-detection systems use different combinations of audio, wording, context, and time rather than assuming that any silence means the user is done.
A premature commitment can cut off a thinking pause or act on an instruction before an important qualifier arrives. Excessive waiting creates a gap after the user has finished. Background voices, noise, and short acknowledgments complicate both directions. A nearby speaker may keep activity detection open, while “yeah” during playback may be mistaken for an attempt to take the floor.
The correct pause policy can also change within a workflow. Reading an account number, confirming a date, and giving an address have different rhythms. One silence threshold may therefore produce good results for one task and poor results for another.
Full-duplex architectures can move turn-taking into the speech model rather than a separate detector. Discrete turns may still be needed for logs, tool coordination, and conversation history, even where audio is processed continuously. Evaluation should measure premature completion and unnecessary waiting, not only detector accuracy.
Interruptions and barge-in
Suppose the agent begins saying:
“Your appointment is booked for Tuesday at 10:30, and I’ve also sent a confirmation email.”
The user interrupts after “Your appointment is booked for Tuesday at 10:30...”. The model may already have generated the whole sentence, and speech synthesis may have produced the remainder, even though playback stopped after the first clause. Saving the full generated sentence in conversation history would imply that information had been presented when it had not.
The Realtime API guide from OpenAI provides a concrete example. With WebRTC and SIP, the server tracks output buffering and automatically truncates unplayed audio after an interruption. In a WebSocket client, playback is managed locally; the client stops the audio, measures how much played, and requests truncation of the remainder. The API removes transcript text associated with the unplayed portion from model context, although it cannot return a transcript aligned precisely to the audio that played.
Other frameworks handle the same state requirement. LiveKit truncates conversation history after interruptions, while Pipecat synchronizes text with speech playback so that only the spoken portion is committed. The details differ, but both need to keep generated output distinct from output presented to the user.
Figure 3. How an interruption affects audio playback and conversation history.
Reliable barge-in needs coordinated handling across several parts of the application:
1. Detect speech while the agent is talking.
2. Decide whether it is a real interruption rather than noise or a backchannel.
3. Stop or lower playback, and cancel output generation that is no longer useful.
4. Update conversation history to match the portion played.
5. Preserve the user’s new utterance and decide what it changes.
6. Determine what happens to external work already in progress.
Stopping speech may be enough when the agent is only explaining something. A booking, payment, or CRM update already sent to an external service has different consequences. Muting the agent does not undo that action.
False interruptions need recovery as well as detection. A cough or short acknowledgment can cut off a useful answer without introducing a new turn. The adaptive handling available in LiveKit uses an acoustic model to distinguish likely interruptions from backchannels, and its turn controls can resume speech after a false interruption. Pipecat offers a minimum-word setting that can prevent a short “okay” from stopping the agent. A threshold of more than one word can also block a genuine command such as “stop,” so that tradeoff needs testing.
Echo introduces another failure mode. If a microphone captures speech coming from the agent’s speaker, the system may interpret its own output as new user input. Acoustic echo cancellation can reduce this crosstalk, but performance should be tested with the devices and audio environments users are likely to have.
Recovery depends on knowing which audio was played, which work was canceled, and what remains valid after the new input. Those are playback, conversation-state, and workflow questions as much as audio questions.
Conversation state and orchestration
Even an accurate transcript may leave operational state unresolved. A user asks to change an appointment, and the agent offers a slot:
“I can move it to 15:30 on Friday. Should I confirm that?”
Before confirming, the user replies, “Morning would be better.” A model can infer from conversation history that the afternoon offer has been superseded. The application handling the booking should also know which proposal is no longer authorized.
State worth recording explicitly may include:
- The appointment or resource identifier.
- The exact slot awaiting confirmation and whether that confirmation remains valid.
- Whether an external action was requested, started, completed, or failed.
- Whether the result is known, including when a response timed out.
- Whether the user has been told the outcome.
- Whether a correction or interruption invalidated earlier instructions.
These are operational facts, and recovering them from natural-language history on every turn gives a model responsibility that a workflow engine or application state store can often handle more reliably. Voice interactions make the issue visible because playback, new speech, model generation, and API calls can all progress at once.
In its GPT-Live engineering account, OpenAI describes speculative conversation state alongside an authoritative account of what was said. The distinction applies beyond that architecture. Tentative information can help with responsiveness, but completed actions and confirmed parameters should rely on authoritative state.
Conversational context is still appropriate for preferences and dialogue. Resource IDs, approvals, transaction status, and side effects generally need a more durable record owned by the application or external service.
Production-readiness questions
The review questions below are intended to expose operational gaps before they become user-facing failures.
Conversation
- Can someone pause, correct themselves, or interrupt without losing the intended turn?
- What evidence causes the system to decide that the user has finished speaking?
- If an interruption is false, can playback and conversation state recover coherently?
Timing
- Can end-of-speech to first audible reply be separated into turn detection, recognition, inference, tools, synthesis, transport, and playback?
- Can engineers identify the stage responsible for slow tail interactions?
State and actions
- Does the system distinguish generated speech from speech that was played?
- Which conversation facts are stored as explicit application state?
From conversation to external actions
Turn detection, interruption handling, and conversation state can fail in ways a transcript does not reveal. Diagnosing those failures requires knowing when each event occurred, what the user heard, and which decisions remained valid.
The consequences become more significant when an agent acts on external systems. Part 2: tools, recovery, and observability will examine what happens when instructions change during tool execution, operations return uncertain results, or the system needs to recover from a failed interaction.
Turn-taking and interruptions are among the themes for AAIF’s Voice Agents Forum, a one-day event in San Francisco on November 5, 2026.
Share
Author
