How Voice Agents Work (and Learn to Stop Talking)
From audio to actions: the components, interruptions and handoffs behind a useful AI conversation, explained with the calls a law firm actually receives.
By VoiceLaw Editorial Team
The hardest feature may be good manners
A caller says, "Can I move my consultation to Thursday?" The assistant starts answering. "Actually, Friday," the caller adds.
A useful voice agent has to hear the correction, stop the obsolete reply and work with the updated request. A beautiful synthetic voice confirming the wrong day is still the wrong day, just with excellent diction.
Voice agents combine audio processing, conversational decisions and connections to real systems. Their quality depends on that whole loop. Getting a model to say a sentence is the beginning of the engineering, not the finished product, and in a law firm the stakes of getting the loop right are higher than most.
The familiar pipeline: listen, decide, speak
One common architecture has three main model stages. Automatic speech recognition, or ASR, converts speech into text. A language model uses the transcript and conversation context to generate a response or request a tool. Text-to-speech, or TTS, converts the response into audio.
This separation lets builders inspect a transcript and choose components for each stage. It also creates interfaces where information can be lost or delayed. A transcription mistake can become a confident answer to a question nobody asked. In legal intake, a misheard surname is not a cosmetic error; it can mean a conflict check runs against the wrong person.
LiveKit's agent documentation describes both this pipeline approach and realtime speech-to-speech models, in which a model can process and generate audio directly instead of requiring a separate text stage for every exchange. The architectures offer different trade-offs in control and latency; neither removes the need for careful testing.
Conversation has a timing problem
A person pausing to remember an address, or to steady their voice while describing a separation, has not necessarily finished speaking. Voice activity detection helps identify speech and silence, while turn detection estimates when the person has completed a turn.
Reply too quickly and the agent interrupts. Wait too long and the caller wonders whether the line has dropped. Streaming allows parts of the pipeline to progress before a whole turn is complete, but the actual timing depends on the components and the deployment.
Interruption handling, often called barge-in, lets incoming speech stop an agent's response. The application must also keep track of what the caller actually heard. Otherwise the next response may assume it already explained something, such as the fact that it is an AI assistant, that was cut off halfway through.
A voice needs hands to do useful work
Saying "your consultation is booked" does not book a consultation. The agent needs a tool connected to the firm's calendar, with an operation the application permits.
A sensible flow looks up available slots for the right lawyer and consultation type, confirms the chosen time with the caller, submits the booking and checks the result before reporting success. If the calendar service fails, the agent should explain that and offer a callback, not improvise a confirmation.
Tool permissions, argument validation and duplicate protection belong in application code. A conversation model can propose an action; the backend must decide whether that specific action is valid. Keep the available tools small and clear enough to inspect and test. For a law firm, that usually means a short list: check availability, book, request a callback, transfer, and send the intake summary.
Give the conversation a safe exit
A handoff is part of the product. Define when a person should take over: the caller asks, the agent cannot resolve an ambiguity, the request falls outside its supported work, or the call matches the firm's definition of urgent.
Pass along a concise summary and the information already collected, so the caller does not have to restart the story. If a live transfer is unavailable, offer a concrete next step the system can actually deliver, with a realistic time frame.
Be clear that the caller is interacting with an automated assistant. In legal intake, also be clear about what it will not do: give legal advice, assess a case or accept a matter. Decide what audio and transcripts need to be retained, who can access them and how confidential information is handled. These choices shape both the caller's experience and the system's design from the start.
Test the messy calls
Try background noise, different speaking speeds, unfamiliar names, long pauses and corrections in the middle of a sentence. Test a failed tool call and a caller who changes their mind after a proposed action. In a law firm, also test the caller who insists on advice, the caller who starts telling the whole confidential story before a conflict check, and the caller who is clearly distressed.
Measure the time from the end of the caller's turn to the first audible reply, alongside task completion and incorrect actions. A fast greeting cannot make up for a wrong booking.
The goal is a conversation that moves the task forward and makes recovery easy. Sometimes the most intelligent thing a voice agent can do is ask one clear question. Sometimes it is stop talking. Humans are still working on that feature too.
Sources and further reading
LiveKit: Voice agent architectures
https://docs.livekit.io/agents/
LiveKit: Turn detection and interruptions
https://docs.livekit.io/agents/logic/turns/
LiveKit: Tool definition and use
https://docs.livekit.io/agents/logic/tools/
LiveKit: Transfers and human handoffs
https://docs.livekit.io/telephony/features/transfers/