While they're thinking, it's almost like they're deaf and blind. They cannot perceive anything else about what's going on. -
It baffles me that most voice agents still make decisions from a transcript.
Every call is converted from audio to text before the agent's "thinking layer" sees it. That conversion is lossy. Only the words survive. The hesitation, the "umm", the falling pitch at the end, the background noise, the half-turn to talk to someone else: all of it is gone.
So the agent guesses. It cuts people off mid-thought, waits too long after they've finished, and misses the anger behind a polite sentence.
I think this is the biggest gap in voice AI today.
This article is my take on the status quo and explores the following questions:
- Can an agent decide well if it never hears the call?
- Why hasn't a web of LLMs closed the gap?
- Why not use speech-to-speech models?
- What would a "thinking layer" that actually listens look like?
Where is the market today?
https://x.com/jordan_dearsley/status/2085602471918064064
For years, people predicted that voice would become the universal way to talk to software. In 2026, that prediction is starting to come true. AI is handling billions of business calls every year, thanks to companies like @ElevenLabs and @DeepgramAI, and to more than 800 voice AI products.
But like the early phase of any tech wave, adoption has outpaced quality. Only about one in five enterprises is very satisfied with their agents in production. So most agents stick to what is affordable and dependable: outbound calls that follow a script, or inbound requests simple enough to predict.
Nearly all voice agents are cascaded: speech-to-text (STT), then an LLM, then text-to-speech (TTS). Engineers love this design because they can pick the best model for each job, read every intermediate step, and keep tight control over what the agent says and does.
While STT and TTS have come a long way in latency and cost, the weak part is the thinking layer.
What a transcript throws away
The thinking layer's job is simple. Know what state the conversation is in, and choose the action that moves it to the right next state: speak, act, hand over, or keep listening.
Your code always knows which step of the workflow a call is on. What it doesn't know is what's happening with the person on the other end. That's the part the thinking layer has to read every turn:
- Has the user finished, or just paused?
- What language are they speaking?
- Are they curious, or angry about bad service?
- Is that a TV in the background, or children playing?
- Are they talking to me, or to their partner?
- What do they want: a refund, a new account, a status check?
Two audio clips with the same transcript, "Yes, go ahead." One is spoken to the agent, the other to someone else in the room. Only the audio tells them apart.
The bold ones above tell us how something was said, not only what was said. A transcript keeps the words and drops the pitch, the pace, the pause and the room.
Take the first question. We all know a person is about to finish before the silence arrives: the pitch falls, the pace slows, the voice trails off. Models that hear the audio learn to use exactly these cues (Ekstedt & Skantze, 2022). But models that only read transcripts can't, because the prosody was never written down (Rajaa, 2026).
Same with the fifth question. When Apple tested whether speech was meant for its assistant, adding the audio to the transcript cut errors by up to 39% compared with the transcript alone.
The best decisions use the audio and the text together. Sadly, today's agents only get the text.
Transcript only: 74.8% of turns cut in early. Audio: 7.8%. Uniphore, arXiv 2609.11066.
Why the web of LLMs hits a wall
Engineers chose the greedy solution to build the "thinking layer". They ran toward the easiest path with the maximum immediate return: hire the top frontier LLM of the day and hand it one giant manual on what to do. That was the whole of 2023-24.
There are a couple of problems here.
- First, teams realised LLMs are unpredictable and slow, andthey lose instructions in long, dense prompts.
- Second, a single LLM call isn't enough to understand an utterance.
So the next logical step was a web of faster, cheaper LLMs: each for different tasks with an orchestrator on top. The demos were fantastic. Buyers were impressed and the revenue started flowing.
This is the frontier stack today.
But in production, this architecture showed two cracks:
- Every piece still reads the same transcript. Adding more LLMs can't bring back the pause or the tone that the transcript already dropped.
- The pieces wait on each other. Each call adds delay. Recognising the turn, deciding what to do and doing the tasks take up to 80% of response time.
A cascaded voice agent: caller audio goes to speech-to-text, then several LLM calls, then text-to-speech. Pitch, pauses, tone, background and addressee are dropped at speech-to-text.
The industry split the thinking layer by tasks. It should have split it by input: decisions needing audio signals should go to an audio-native model, and the LLM should work only on text.
Why not skip the transcript altogether?
Speech-to-speech (STS) models started to show real signs of life this year. The appeal is clear: lower latency, better conversation understanding, and less engineering, by compressing STT, LLM and TTS into a single model.
The biggest downside is lack of control and verified data. You can't pin down what an STS model will say, and tool calls and code execution don't come naturally to a model that thinks in audio.
I think STS is still far from reality in the short term. Even if it accelerates, it will first suit simpler use cases like FAQ answering. Builders will stay on the cascade, partly out of inertia, and mostly because they need its control and flexibility.
That's why I'm building models that make the cascaded stack easier to control, and feasible to deploy on both cost and latency.
The layer that listens
I believe there is a way to control agents and remove the bloat: decision models.
Look closely at GPT-Live and you will find a decision engine at its core. Many times a second, it decides whether to speak, keep listening, pause, interrupt or call a tool. The cascade needs the same engine, as a component you can see and control.
A decision model takes the current state and a bounded set of options (speak or wait, escalate or continue, which intent) and returns one choice with a probability, faster and far cheaper than an LLM. @typesafeai surprised people with how well this works on text.
I'm proposing:
- The decision model listens to the call and makes the bounded decisions.
- Your software keeps workflow state, permissions and tools.
- The LLM does what it's best at: writing the reply, once the decision about what to do has been made.
A decision model takes audio, the conversation so far and a set of options, and returns a probability for each option: refund_status 0.91, cancel_order 0.06, escalate 0.03 (example).
The confidence score is what makes this work. Ask an LLM how sure it is, and it will usually say "very", even when it's wrong (Xiong et al., 2024). A classifier's scores can be tuned until they mean what they say: when a classifier says it's sure, it usually is (Guo et al., 2017).
Our early experiments with design partners show that 30 to 40% of what an LLM does inside an agent is exactly this kind of work: routing, triage, choosing between bounded options.
But today's decision models work on text or images. TypeSafe AI reports that every model, Jev included, makes worse decisions when another modality is added alongside text, and Jev has no documented audio support.
You can't bolt audio onto a text decision model. Building one that listens takes three things:
- Real conversational audio labelled with the right decision at each moment
- A model that decides using the audio and the text together
- Confidence scores that hold up across accents, noise and bad phone lines.
The right decision model for voice is the one that listens, an audio-native decision layer.
What changes when the decision layer can hear
You unlock the following use cases with an audio-native decision model:
- Agents you can debug. Many teams now design an agent as a graph of states: greet, verify identity, handle the refund, escalate. Each turn comes down to two decisions: which state is the conversation really in, and which state should it move to next? Get either one wrong and the whole call drifts.
With a decision model at each step, every transition becomes a small, explainable unit: what was heard, which options existed, what was chosen, and how confident it was. When a call goes wrong, you don't debug a giant prompt. You find the one cog that slipped and fix that piece.
- Answers ready before the caller finishes. People reveal what they want before they finish the sentence. "I'm calling about the order I placed last week..." is already an order lookup. A text-first agent waits for the user to stop, waits for the final transcript, decides which tool to call, and only then calls it. Every step adds to the silence.
Timeline of a caller saying "I'm calling about my order from last week". A text-first agent waits for the end, then transcribes, decides and calls get_order. A listening router fires get_order mid-sentence and replies as soon as the caller stops, saving time.
A decision model that listens works as a router that preempts the call. It recognises the intent mid-sentence and fires the lookup while the user is still talking, so the order details are ready the moment they stop.
Researchers at Meta and CMU showed the payoff: predicting tool queries in parallel with the user's speech cut tool latency by 20% and made answers more accurate.
- Personal agents that feel instant. Personal agents hear commands via voice notes all day, in kitchens, cars and busy streets. Most of what they hear needs one quick call: is this a task, a reminder, a passing thought, or not meant for me at all?
Sending every utterance to a frontier LLM is slow and expensive. A small model that listens makes that call in a fraction of the time, and only wakes the LLM when there's real work to do. The agent feels instant, because the first decision never waits on the big model.
https://x.com/signulll/status/2077444321243013306
The future of voice AI
Voice was supposed to be the universal interface. Today it only works when the conversation is predictable.
But an unscripted conversation is not unbounded. At every turn, the sensible moves are few: speak, wait, ask, act, hand over. What nobody can script is the order. You don't need to script a conversation to control it. You need to make each decision well, from what was actually heard.
That's the shift I'm betting on: decide, then prompt. A listening model makes the calls, your software keeps the state, and the LLM writes the words. Every decision is visible, fast and measurable. When that works, voice stops being a channel for simple calls and becomes how people talk to software.
I am building that decision layer at @tazolabs, and I will share more about our models in the next few days.
If you build voice agents, I'd love to hear which decisions your agent struggles with today. And if you resonate with this vision, come build with us at Tazo Labs.
What's the one decision your voice agent gets wrong most often?
