NotesFileEditViewWindowHelp
July 29, 2026 at 9:25 AM

[Blog] your voice agents are half-deaf

this is how we fixed the really annoying issue of voice agents not being able to tell when the user's audio isn't clear. this is how your agent can ask "can you please repeat that" or "i could not quite catch that" just like a human would :)

also live on medium (please leave a like man).

first, a 60s primer on voice agents

Simply put, a voice agent is an AI that talks to you on a phone call. Under the hood it's three models relay-racing (passing the baton i mean):

STT (speech-to-text) listens to your audio and turns it into text → LLM (brain) reads that text and generates a reply → TTS (text-to-speech) speaks the reply back to the user.

Important nuance for us: the LLM never hears your voice. It's like a person locked in a room replying to a live chat transcript of your call. If the transcriber (STT) doesn't type your words, then as far as the brain (LLM) is concerned you never spoke. That single design fact is the root of everything in this post.

what we observed in live customer calls

We observed big silent gaps in our calls whenever the STT could not transcribe a user audio chunk. Whenever the STT model misses something, the LLM is usually blind (or deaf) to it. The agent only replies to what it hears from STT.

Here's what i observed in live production calls. The 10s silence-timer fallback gets triggered and the agent asks "are you still there?" because it missed the user's "okay". Really bad ux, and honestly just annoying given i could do very little about fixing the foundational models to be better.

t 0.0s : Agent asks "Do you want to confirm your order"
t 2.0s : User says "Okay" (STT model missed this)
-- Dead silence --
t 10s  : Agent asks "Are you still there?"
t 12s  : User is annoyed and cuts the call

Important observations:

  • This is observed far more in multilingual use-cases, especially Indian languages. At Nurix our clients have customers across different regions/languages/backgrounds. It got especially highlighted when we worked with one of the biggest online retailers in India on customer support calls. A big chunk of their callers were from tier 2/3 cities where generally speaking phones don't have the best microphones, audio quality is low, and background noise is high. This was causing a lot of drop-offs and (pseudo) high-latency turns.
  • It usually happens when users reply with single words like "haan", "yes", "okay". These are exactly the words that move a conversation forward, and when missed, are a pain. Later you'll see why this observation matters.

what goes on under the hood that causes this?

In a voice agent everything is cascaded: STT → LLM → TTS. Each has its own quirks and failure modes. But after listening to thousands of bad calls, i slowly realised STT was the biggest culprit, and because it's the first touchpoint it cascades the failure downstream and disempowers every model after it. The LLM reasons over a wrong transcript, the TTS speaks a wrong answer, and the whole conversation goes off track.

Two things kept showing up while debugging prod calls:

case 1: STT completely missed the user turn.

case 2: it transcribed something with low confidence but never finalised it.

Let me explain the second case better. Streaming STT doesn't transcribe everything at once, it guesses as you speak, sending a stream of "rough drafts" (interims) that get revised until the model is confident enough to commit one as final. So you get a bunch of chunks with is_final = true and is_final = false:

"can"                     is_final = false
"can you"                 is_final = false
"can you speak"           is_final = false
"Can you speak in English?"  is_final = true

The agent only trusts finals.

how do humans handle this?

We took a step back and asked what we (humans) do in this situation. When you're on a phone call and you can't understand what the other person said, how do you figure out that you missed something? What are the signals?

We don't have a transcript in our head, but we heard a voice. We caught a fragment of something we can't make into words, and now the other person has gone quiet, clearly waiting for us to respond. So we do the most natural thing: "sorry, can you say that again?" or "there's a lot of noise on your end, i couldn't catch that."

Trivially easy for humans. We wanted our agents to do the same thing.

But that's not how voice agents usually work, and the reason is simple: in most voice AI stacks the LLM only sees the output of STT. If STT fails, the LLM is just deaf. It doesn't get a bad transcript that it could question, every transcript it gets it treats as 100% correct. Nothing in between. We needed to fix this.

the signals we already had

I listed down every signal flowing through the pipeline that could be useful:

  • final transcript (is_final = true): the only thing the LLM normally consumes.
  • interim transcripts (is_final = false): intermediate, low-confidence output. The STT heard something, it's just not sure what yet.
  • VAD (voice activity detection): flips the user state between listening and speaking. Completely independent of STT output, raw user audio goes in. Generally used for turn-taking.

When does a voice agent know to start speaking? It replies once it gets an EOU (end of utterance) signal, which is usually just a combination of two things: a final transcript with is_final = true plus the user state at that moment being listening (controlled by VAD). Both have to be true to conclude the user's turn is over, and then whatever STT transcribed gets handed to the LLM.

When i looked at the missed-transcript instances, there was a clear pattern of these signals telling me the user said something, and we were just ignoring it.

finally, how we fixed it

Two patterns showed up in the signals:

  1. VAD detects user activity for a meaningful time but STT transcribes nothing. Usually when the voice is loud enough but the audio is garbled, noisy, or the user spoke gibberish.
  2. A few orphaned interim transcripts that never get finalised. Usually when the user's voice is too low, or they say a single short word without proper pronunciation ("haan", "yes", "okay") that fails to trigger VAD activity but STT still picks it up as an interim.

Once you see the patterns, the direction of the solution is clear. We built a module that subscribes to those event streams and runs two small state machines, one per pattern:

on user speech START:
    snapshot start time + agent state, reset flags
    cancel any pending timers (user is retrying, let them)

on interim transcript:
    got_any_stt = true
    start ORPHAN WATCHDOG (first interim of this session only)

on final transcript:
    resolved - cancel everything

on user speech END:
    if agent was LISTENING when speech started
       and duration >= ~1s
       and zero STT arrived:
        start GRACE TIMER

on agent starting to speak:
    cancel everything (something else is answering)

GRACE TIMER expires     -> confirmed no-STT miss
ORPHAN WATCHDOG expires -> confirmed orphaned-interim miss
    -> re-verify, rate-limit, then ask the user to repeat

Detection was honestly the easy part. Most of the real engineering went into not firing when we shouldn't. An agent that asks "can you repeat that?" when you didn't say anything is worse than an agent that occasionally goes silent. So the module is mostly a list of reasons to stay quiet:

  • Only trigger if the agent was listening when the speech started. Our data showed misses during the agent's own speaking state were false alarms around two-thirds of the time — echo, crosstalk.
  • Minimum speech duration. Half the zero-STT sessions are sub-500ms coughs and hmms. Below the threshold we deliberately do nothing. Tune the threshold to taste.
  • A grace period before prompting. Late STT sometimes arrives seconds after the speech ends, and a lot of the time the user just repeats themselves on their own. Both cancel the timer, so PLEASE add a grace timer.

results

We enabled this behind a feature flag for production agents with Hindi-heavy, noisy-audio use cases.

On the turns that actually had a miss, the improvement is structural. Those turns used to resolve through the default silence timer, 10+ seconds of dead air before the agent said anything. Now they resolve inside the 1–3 second detection window.

Overall perceived turn latency went down too. We didn't actually reduce the latency of any model here, we just saved a LOT of seconds by preemptively detecting a transcription miss and reclarifying. Win-win.

  • median perceived latency: down 12%
  • p90 perceived latency: down 43%

takeaways

  • "foundational models will fail. resilience lives in the orchestration layer." We didn't fix the STT model, we made its failures observable and recoverable.
  • instrumentation is king. The signals you need probably already exist in your pipeline, you're just discarding them. Please don't. Please. In our case VAD and interims were flowing through every call, unused.
  • users forgive an agent that asks. they don't forgive an agent that ignores them.
  • measure passively everywhere before intervening anywhere. We instrumented these signals in production for a while before actually shipping the fix.
  • false-positive discipline is super important. Most of the code is conditions for staying quiet lmao.

Earlier the LLM was deaf to everything outside its transcript. Now it can actually hear what's going on in the call. It knows when it didn't hear you, and it does what any of us would do on a bad line, it asks you to say it again.

Huge shout-out to the goat Pushkar Patel for helping me out with the implementation.