Talking to an AI while it works: the problem lies in the silences

I am not satisfied with the voice solutions I use. I want to talk to my agents, including from my phone, while they work on different tasks and projects. Answering a simple question is one thing. Handling a request that involves retrieving context, looking up information and thinking, while we keep talking, is another.

That is where the silences appear. Along with the phrases that try to fill them: “one moment”, “right away”, “I am working on it”. A natural-sounding voice is not enough for me. I want the conversation to remain useful while the system has work pending, and I want it to listen when I need to correct it.

When I talk to Hermes on my computer, it works well. I mean Hermes Agent, from Nous Research.[24] But if I switch sessions, sometimes the switch works and sometimes it does not. In my use, that conversation only works at the computer, not on my phone. To connect it to my phone, I use Hermes Conduit, a free, open-source client for iPhone and iPad that connects to your own Hermes installation.[25] I encounter many errors: sometimes it does not send the message, cuts out or fails to connect. That is my experience with that setup, not a general assessment of these products.

The time between two sentences

Human conversation moves at a demanding pace. A study of questions and answers in ten languages found an average gap of 208 milliseconds between turns: roughly two tenths of a second. This is a descriptive reference, with variations across cultures and types of response; it is not a requirement to answer every question within that time.[1]

AI has several things to work out before responding. First, it must receive the sound. Then it must decide whether we have finished. A pause to find a word can seem like the end of a sentence. If it waits too long, it feels slow; if it rushes, it cuts us off.

In many systems, a three-step chain follows: transcribe the speech, prepare a response with a language model and turn it back into sound. Some of this work can begin while we are speaking. But the network, processing and playback still take time.[2]

Native voice models reduce that separation between components. Moshi, presented by Kyutai in 2024, reported a practical latency of 200 milliseconds under its experimental conditions. That figure does not mean it can resolve a case or complete a search in two tenths of a second.[3]

To judge whether this is useful to me, I need to distinguish three moments: when the machine acknowledges the request, when it provides the first useful information and when it finishes the task. An immediate “I am looking into it” improves the first. The other two remain pending.

A voice available while another component works

The receptionist and worker analogy helps explain a solution that already exists. One component listens, clarifies and talks; another handles what takes longer. There need not be two providers or two characters: separating responsibilities is enough.

OpenAI documents this division in GPT-Live, generally available through its API since September 2026. The conversation can continue while a background model or agent uses tools. The application decides whether that work continues when the user interrupts.[4][5]

LiveKit also publishes a pattern for delegation to a subagent. Its documentation acknowledges specific risks: the conversational agent might improvise an answer before receiving the result, or subsequently fail to communicate it correctly.[6]

The receptionist needs to know what is happening. If the worker is still handling the previous request and we switch projects, the answer that arrives may be correct for a question we are no longer asking.

This is easy to understand in a law firm. The person answering a call can keep the client informed while someone else looks into the matter. But they need to distinguish “I have requested it”, “it is being reviewed” and “it has been checked”. AI needs the same discipline.

“One moment” helps, until it stops helping

Fillers are no accident. ElevenLabs allows a phrase to be configured when a response takes time. On 15 October, OpenAI announced more brief expressions during searches in its voice mode.[7][8]

Research offers a nuanced answer. A 2025 study with 54 participants and agents in virtual reality found that combining gestures and verbal expressions improved perceptions of the wait, without statistically significant improvements in the other general measures assessed. It did not isolate the effect of voice alone.[9]

The useful distinction is between keeping someone company and putting on an act. “The query is still pending” reports a situation, if the system knows it. “Almost done” requires knowing that there really is little left to do. Hold music or the sound of typing does not demonstrate activity.

Nor should “thinking aloud” be confused with providing access to what happens inside the model. For getting work done, a brief explanation of the next step and the obstacle encountered is usually more useful than a running account of supposed thoughts.

Interrupting the voice does not always stop the action

“Yes, I am following” and “no, stop” may both be spoken while the AI is talking, but they serve different purposes. The first signals attention; the second tries to change what is happening. A detector that reacts only to sound can confuse them.

That is why systems combine voice detection, interpretation of when a turn ends and the ability to listen while speaking. This last capability is called full-duplex. It is an important advance, but it does not guarantee that every pause or correction will be understood correctly.[3][10]

There are also several things to stop: sound still waiting to play, the response being generated and external work. Deepgram explicitly warns that discarding a response does not reverse an operation that the external system has already carried out.[11]

Imagine a request to send something, followed by “wait, not that recipient”. The AI falling silent does not prove that sending has been cancelled. The right response depends on the actual state: stopped in time, already completed or still unconfirmed.

My principle is simple: stop talking and listen first. Then, depending on what I say, stop the task or carry on. Interrupting an explanation to add a detail is not the same as cancelling the request. Nor do I want to have to wait until it finishes speaking before I can clarify that.

To apply this principle, the system needs to know the state of each task: pending, running, completed or with a result still unconfirmed. That information must reach whoever is handling the conversation. As Deepgram's warning shows, it cannot say “cancelled” simply because it has stopped speaking; it must check what could be stopped and what has already happened.[11]

What products already offer

In October 2026, the landscape goes well beyond dictating and hearing a reply. ChatGPT documents simultaneous conversation and delegation; Google presents Gemini 3.8 Live with reasoning and tools running in parallel. Claude offers voice with tools on the web, desktop and mobile, still in beta.[8][10][12][13]

Alexa+ focuses on everyday conversations and actions; Amazon continues to offer early access in Spain. Siri AI arrived in beta in September, with a waiting list and restrictions by device, region and language. Expansion announcements should not be confused with features available to everyone.[14][15]

OpenAI also documents calls to its Dots: users can talk to the agent about its work and hang up without that work necessarily stopping.[16]

In my experience, ChatGPT's Dot works better. But I find few voice options, I have to call, and often it does not pick up. Besides, it is ChatGPT: these are not my agents. That distinction matters for what I want. I want to talk to the bots I have assigned different tasks and projects to, rather than judge only whether an isolated conversation is pleasant.

What I describe about Hermes, the mobile connection and Dot is my experience of using them. Manufacturers' published capabilities help explain what is on offer, but they do not turn that experience into an objective comparison. That would require testing the same tasks, language, connection and interruptions. A pleasant demonstration does not prove that a system manages several projects well or communicates all its results.

The conversation also has legal conditions

The more human a voice sounds, the more relevant it becomes to know who is responding. In the EU, Article 50 of the AI Act requires providers to design these systems to inform people that they are interacting with AI, unless this is obvious in the context specified by the legislation. The information must be provided no later than the start of the interaction. This obligation has applied since 2 August 2026.[17][18]

Norway requires a qualification. In August, its government was still preparing to incorporate the regulation and envisaged presenting the legislation in spring 2027. That intention does not establish that it has entered into force in Norway. Nor does it rule out certain activities of a Norwegian company falling within the EU's territorial scope.[19][20]

Informing someone that an AI is speaking does not automatically authorise recording. Under the GDPR, the lawful basis, purpose, information provided to the data subject and retention must be determined. Consent may be necessary depending on the circumstances; it is not the only possible basis for all voice processing.[21]

A recording may contain personal data and reveal specially protected information. That does not mean every recording falls under the special regime for biometric data: what matters is whether the voice undergoes specific technical processing aimed at uniquely identifying or authenticating a person.[21][22]

A law firm must also consider professional secrecy. It needs to know who receives audio and transcripts, what they retain, which third parties they use and under what safeguards. The CCBE's technical guide stresses the need to examine those flows and contracts. “We do not use your data for training” does not, on its own, answer all those questions.[23]

What remains to be resolved

That is why I am building my own solution: an internal app that connects to my computer so I can talk to my five main bots. Each has a particular voice and personality and handles different tasks and projects. That is the system I am building; I am not saying that the problems with connections, waiting or interruptions have already been solved.

What I require of it is the ability to finish a sentence, keep talking while work is pending and receive accurate information about that work. If I interrupt, it must first stop talking and listen; then decide, based on what I say, whether to stop or continue. To do that, it needs to distinguish what it has done from what it has merely attempted.

The research helps make those requirements precise. It does not guarantee that my app will meet them better. But it does make clear what I want to check: that I can talk to my agents, retain control of their tasks and understand what is happening when a response takes time.