A voice agent can handle a conversation correctly and still leave the wrong result in an external system. Tool calls can continue after an interruption, handoffs can fail, and apparently successful conversations can conceal uncertain outcomes.
Part 1: latency, turns, and interruptions covered timing, turn-taking, interruptions, and conversation state. This article examines what happens when agents act on external systems, how teams can investigate failures, and what reliable operation requires beyond the conversation itself.
Tool calls change the conversational problem
The user says “Book the 16:30.” A moment after the request begins, they add, “Actually, make it Wednesday.” The conversational correction is clear, but the first booking may already have reached the external service.
At the time of the interruption, an operation might be queued, running, completed, failed, or uncertain because its response was lost. A timeout cannot safely be treated as proof that nothing happened. Retrying without checking can produce two bookings.
Figure 1. Reconciling an uncertain operation outcome before retrying.
This is a familiar distributed-systems problem. RFC 9110 defines idempotent request methods in terms of the intended server effect of repeated identical requests and warns against automatic retries of non-idempotent operations without additional assurance. Voice introduces a fast-moving conversational layer around that existing risk.
Frameworks differ in what an interruption cancels. With LiveKit, tool code continues running by default after the agent is interrupted, although a late result may no longer reach the agent. Developers must propagate cancellation if they intend to stop the underlying work. Pipecat cancels in-flight function calls by default, but can be configured to let them continue asynchronously and return results to context later. Neither default guarantees that an operation already accepted by an external service will be stopped or reversed.
For actions that change external state, LiveKit also documents disallow_interruptions() as an option during operations such as scheduling an appointment or placing an order. Preventing interruption is one possible policy, not a substitute for knowing the operation’s outcome. A read-only lookup and a payment-related action justify different handling.
Stable operation IDs, deduplication, application-level idempotency, narrowly scoped confirmations, and reconciliation against the system of record all reduce the risk of duplicates or unintended changes. A confirmation for Friday at 15:30 should not silently authorize a different date or time after the user changes their mind.
The language used during a slow tool call also needs to reflect state. A brief “I’m checking that” may be appropriate while work is pending, whereas “It’s booked” requires confirmation. Applications should represent running, succeeded, failed, and unknown outcomes separately rather than asking the model to infer them from the absence of an error.
A conversation can appear resolved while the external system has recorded a different result. The debugging and recovery path must follow the external state, not only the transcript.
Human handoff is a system behavior
A caller requests a person, and the agent says it is transferring the conversation. Nobody answers. The dialogue suggests a successful handoff, but the transfer did not complete.
Transfer workflows need states such as requested, waiting, accepted, unavailable, and failed. They determine what the caller hears, whether the automated agent remains active, and who is responsible for work started before the handoff.
An implementation example appears in the Dialogflow CX guidance. Its live-agent-handoff response signals that the surrounding integration should transfer control, but the response alone does not change the session state. The integration still has to carry out the transfer and handle failure.
The receiving human also needs more than a transcript. Useful context includes the caller’s goal, verified identifiers, completed actions, pending operations, previous failures, current confirmation state, and relevant tool results. Otherwise the human must reconstruct the operational history while the caller waits.
Any operation still running needs an owner. Depending on the workflow, it might be canceled, allowed to finish, surfaced to the person taking over, or paused pending approval. If nobody is available, the system should have an explicit fallback, such as returning to the agent, arranging follow-up, or moving to another supported channel.
Evaluation has to test full conversations
Word recognition can be accurate while the conversation fails. A transcript may capture every word but omit the moment when the agent spoke over a pause. A booking tool may receive the correct arguments yet produce a duplicate after an uncertain retry. A final answer may be correct even though the user was interrupted repeatedly on the way there.
Speech recognition remains worth evaluating on its own. The NIST evaluation plan, for example, uses word error rate, calculated from substitutions, deletions, and insertions against a reference transcript. That metric can show recognition regressions but does not establish whether an interactive task completed correctly.
Conversation-level benchmarks cover different behaviors. Full-Duplex-Bench evaluates pause handling, backchanneling, turn-taking, and interruption management separately. A plain transcript without audio or timing data can hide failures in those areas. VoiceBench introduces variation across speakers, environments, and content.
The τ-Voice benchmark, published by Sierra researchers in March 2026, combines full-duplex conversations with grounded multi-turn tasks and external actions. Across 278 tasks, GPT-5 reasoning scored 85% on the text equivalents, while the voice agents tested at that time reached 31-51% in clean conditions and 26-38% with noise and diverse accents. In the authors’ qualitative analysis, 79-90% of failures were attributed to agent behavior rather than the evaluation setup.
In October 2026, Sierra published τ-i18n, which extends τ-Voice to five more languages and keeps the tasks and success criteria fixed across languages. Of the three agents tested, GPT-Live-1 had the highest task completion, at 79.9% across all six languages. Even so, Sierra recorded responsiveness failures, such as a missed turn or a failure to yield, in more than 80% of calls for GPT-Live-1 and Gemini 3.8 Live, against 16% for Grok Voice Think Fast 2.0. An agent can complete the task and still handle the conversation badly. Because the setups differ, the October scores are not directly comparable with the March results.
Exact spoken details are a separate problem. Sierra’s τ-Entity benchmark, published a day after τ-i18n, tests whether an agent can capture a single spoken detail, such as a name, phone number, address, or email. Pooled across the six languages, GPT-Live-1 captured the value correctly in 60.3% of regular calls, but passed only 34% of tasks when it also had to succeed in noise-heavy and speech-heavy conditions. In τ-Elicitation, a separate study from September 2026, Sierra researchers tested four voice agent configurations. When the agents chose their own verification strategy, 14-41% of tasks succeeded across all three test conditions. With a prescribed sequence of spelling, read-back, correction, and confirmation, success rose to 37-54%, and each call took 21-28 seconds longer. These are benchmark-specific findings, not a general estimate of production success rates.
Production evaluation should follow the application’s risks. Tests might include unusually long pauses, names and account numbers, background conversations, corrections made during tool execution, ambiguous confirmation, lost tool responses, and handoffs when no person is available. When the result is measurable, check tool arguments, the final database state, whether an action occurred exactly once, latency, and transfer status.
Human assessment remains useful for conversational qualities that metrics capture imperfectly, such as whether a pause feels awkward, speech is intelligible in a noisy environment, or an interruption is handled naturally. A complete test program needs component measurements, realistic conversational scenarios, and checks on the final external state.
Observability and debugging
Consider a trace whose transcript contains only:
User: “Book the 16:30.”
Agent: “Your booking is confirmed.”
The caller says they tried to interrupt and ended up with the wrong booking. The transcript does not establish when the interruption was detected, what audio was played, whether the first booking request completed, or what result the tool returned.
A useful investigation needs timestamps for speech start and end, turn commitment, transcript finalization, model output, tool arguments and results, synthesis, playback, and interruption detection. These events must share stable conversation, turn, or operation identifiers so that one sequence can be reconstructed across services.
Figure 2. How correlated logs reveal the sequence of events behind a failed booking.
OpenAI’s GPT-Live engineering report offers an example of how production failures can evade aggregate measurements. During shadow testing, a supporting component saturated earlier than expected, causing requests to accumulate and latency to increase. Long sessions revealed memory and persistence pressure; reconnects exercised state restoration; ordinary disconnects exposed shutdown races. Some metrics also combined different latency sources or hid unhealthy individual engines behind averages.
The particular architecture is less important than the debugging lesson. A single success rate or total latency figure can conceal a problem tied to one stage, geography, session length, or order of events.
Observability creates a privacy tradeoff as well. Recording transcripts, audio, prompts, and tool payloads can help reproduce failures, but may also duplicate sensitive information. An OpenTelemetry walkthrough describes instrumentation that emits model names, token counts, and durations while omitting prompt content and tool arguments by default.
For many incidents, event ordering, durations, identifiers, error codes, and state transitions are enough. Richer content can be collected or accessed only where the use case, retention policy, and data controls justify it.
Privacy, security, and compliance
A user reads out an account number. The agent transcribes it and sends it to a tool. Depending on the instrumentation, the same value may now be present in the transcript, the model context, a tool payload, a recording, and a trace store. Ordinary debugging can therefore create several copies of information that the application should limit.
A privacy review should follow the data along its full path:
audio → transcript → model context → tool input/output → telemetry → retained recording or history
At each stage, the system owner needs to know what is captured, why it is required, where processing takes place, how long the data remains available, and who can access it. Retention and access controls should reflect the task and applicable jurisdiction.
Voice also calls for careful biometric terminology. A recording of speech does not automatically amount to special-category biometric processing in every legal context. The UK ICO guidance treats biometric data as special category data when it is processed to uniquely identify someone. Other laws frame particular identifiers more directly. For instance, Illinois’ Biometric Information Privacy Act includes a “voiceprint” within its definition of a biometric identifier.
Transparency obligations may apply to the conversation and its generated audio. Since August 2, 2026, Article 50 of the EU AI Act has required providers of certain AI systems that interact directly with people to ensure the people concerned are informed they are interacting with AI, unless that is obvious in context or an exception applies. The information must be provided clearly no later than the first interaction. Article 50(2) separately requires providers of systems generating synthetic audio, among other content, to mark outputs in a machine-readable format so they can be detected as AI-generated, subject to the legislation’s exceptions and to what is technically feasible. For systems placed on the market before August 2, 2026, the marking and detection obligations under Article 50(2) apply from December 2, 2026.
The obligations are not identical across healthcare, financial services, employment, public-sector deployments, and general consumer use. A deployment-specific review is more useful than assuming that one disclosure or retention policy covers every application.
Security controls also need to follow the actions available through tools. Fluent speech should not substitute for authentication or authorization. Permissions, confirmations, audit trails, and checks on sensitive operations belong at the application and tool boundaries that enforce them.
Production-readiness questions
The review questions below are intended to expose operational gaps before they become user-facing failures.
State and actions
- If the user changes intent while a tool runs, what happens to the underlying operation?
- Can a timed-out action have an unknown outcome, and is that resolved before retrying?
- Does each confirmation cover the exact action and parameters to be executed?
Evaluation and operations
- Do tests include pauses, corrections, noise, interruptions, tool failures, retries, and handoffs?
- Can an engineer reconstruct a failed interaction across audio, turns, tools, state, and playback?
- Do reproduced production failures become regression cases?
Handoff, privacy, and control
- What completed and pending actions reach the human who takes over?
- What happens when the transfer or another dependency is unavailable?
- What audio, transcripts, tool data, and telemetry are retained, and who can access them?
- Which actions require stronger authentication, authorization, or confirmation?
A slow reply can begin with endpointing, an interruption can leave a tool running, and an uncertain retry can create a duplicate external action. Yet the transcript may still read like a normal exchange. With enough timing, playback, and state evidence, teams can isolate the failing component and test whether a change fixes the interaction.
Evaluation and human handoff are among the themes for AAIF’s Voice Agents Forum, a one-day event in San Francisco on November 5, 2026.
Share
Author
