The Hard Part of Voice AI Begins After the Demo
Lessons from production voice AI on latency, orchestration, cost, and reliability
Date
July 16, 2026
Author
Rishi Kumar
Overview
Voice AI has reached a point where building a working agent is no longer the biggest challenge. With advances in speech recognition, language models, and text-to-speech, most teams can assemble a system that holds a natural conversation in a relatively short time.
The moment those systems move into production, the engineering challenge changes completely.
Production deployments introduce problems that prototypes rarely expose. Conversations become unpredictable, latency starts affecting user experience, infrastructure costs increase with usage, and multiple services need to work together without disrupting the flow of a call. What appears seamless in a controlled setting becomes far more complex when thousands of real customers are involved.
These challenges are shaping how organisations approach voice AI today. The discussion is gradually shifting from building capable agents to building systems that remain reliable, adaptable, and cost-effective in production.
In a recent conversation with Maitreya Wagh, Founder of Bolna AI, we explored many of these engineering challenges through the lens of production voice AI. His experiences reflect questions that engineering teams across the industry are trying to solve as voice AI moves into large-scale business operations.
This article builds on those insights to examine the practical challenges behind scaling voice AI, including latency, orchestration, cost optimisation, monitoring, and the operational decisions required to make these systems work reliably in the real world.
Scaling Changes the Engineering Problem
Most voice AI demonstrations follow a familiar pattern. A user asks a question, the system understands it, responds naturally, and completes a task in a matter of seconds. Under controlled conditions, the interaction feels remarkably close to a human conversation.
Once deployed, those assumptions disappear.
Real customers speak over the agent, pause mid-sentence, switch languages, use unfamiliar terminology, and ask questions that were never anticipated during testing. At the same time, businesses expect the system to respond instantly, remain accurate, integrate with existing workflows, and keep operating as call volumes increase.
During our discussion, Maitreya Wagh summarised this challenge simply:
"The biggest problem we're trying to solve is scaling."
That challenge goes well beyond supporting more simultaneous calls. Production voice AI must solve several problems at once.
Maintaining low response latency
Handling interruptions naturally
Controlling infrastructure and inference costs
Coordinating multiple services without failure
Measuring whether conversations actually achieve business outcomes
Unlike traditional enterprise software, customers experience all of these systems as one continuous conversation. They never see the speech recognition model, language model, telephony platform, or business workflow operating behind the scenes. If any one of those components slows down or fails, the entire interaction feels unreliable.
This is one reason operating at enterprise scale demands a different engineering approach from chatbots or workflow automation. Every additional customer, every extra second of latency, and every integration introduces another layer of complexity that must be managed without affecting the experience.
The same pattern is becoming visible across enterprise AI adoption more broadly. According to the Stanford AI Index 2026, organisations are rapidly moving beyond experimentation, placing greater emphasis on deployment, operational reliability, and measurable business outcomes rather than model capability alone.
For engineering teams, the takeaway is becoming increasingly clear. Building an agent is now one milestone in a much larger journey. Latency stops being a performance metric and becomes part of the customer experience.
Latency Separates Prototypes from Production
Most voice AI systems perform well in controlled demonstrations.
A customer asks a question. The system responds almost instantly. The conversation flows naturally, the task gets completed, and everyone in the room walks away believing the hard part is over.
Once real customers enter the picture, the conversation changes completely.
Unlike demo users, they interrupt. They hesitate halfway through a sentence. They speak with different accents, change topics unexpectedly, and rarely follow the path the system was designed for. The conversation becomes messy because that's how people naturally communicate.
This is where latency stops being a technical metric and starts becoming a customer experience problem.
The delay a caller notices is rarely caused by one system. A voice interaction moves through several stages before a response is heard. Speech is converted into text. The request is interpreted. A response is generated. That response is converted back into speech before it finally reaches the caller through the telephony network. Each step takes only a fraction of a second, but those fractions accumulate surprisingly quickly.
Customers don't see any of this happening. They only notice whether the conversation feels effortless.
If the response comes a second too late, it feels uncertain. If the system starts talking while the customer is still speaking, it feels impatient. When both happen repeatedly, trust disappears long before the technology has a chance to prove itself.
One observation from our conversation captured the challenge well:
"You want the sort of layer that gives you access to the best at that point of time for each of your calls."
It sounds like a discussion about models, but it is really a discussion about flexibility.
Production systems cannot assume that today's fastest speech engine or strongest language model will still be the best choice a year from now. Different conversations also demand different priorities. A customer support interaction may value context and accuracy above everything else. An outbound campaign handling thousands of calls may place greater importance on speed and operating cost. Designing around a single provider or a fixed stack leaves very little room to adapt when those priorities change.
Timing presents another challenge that is much harder to measure.
People interrupt each other constantly. We pause to think, restart sentences, correct ourselves, and occasionally speak at the same time without even noticing. Humans instinctively understand these signals. Software has to learn them.
Should the agent stop talking because the caller interrupted? Was that a genuine interruption or just a short pause before the sentence continued? Respond too early and the conversation feels unnatural. Wait too long and the interaction loses momentum.
Those decisions become harder as deployments grow. The same platform may support customer service in the morning, appointment booking in the afternoon, and collections or sales by evening. Every workflow has different expectations, but the customer judges them all using the same standard: did the conversation feel natural?
That is why latency is rarely solved by making one model faster. It is solved by designing an entire system that can respond consistently under real-world conditions. Once voice AI reaches production, milliseconds are no longer just engineering measurements. They become part of the customer experience itself.
Orchestration Is Where Voice AI Stops Being Just AI
Ask anyone building voice AI what makes a good demo and the answers are usually similar. Low latency. Natural speech. Good voice quality.
None of those become the biggest problem once the system reaches production.
The first real hurdle is much less visible. It's getting a conversation to move through a business without falling apart.
Take something as routine as changing an address on an insurance policy. The customer hears one conversation. The system doesn't.
Before the request is completed, it may need to verify identity, retrieve policy information, check whether the request meets business rules, update the policy administration system, record the interaction for compliance, and confirm the change back to the customer. Every one of those steps depends on a different system.
If any one of them slows down or fails, the customer doesn't blame the CRM or the policy database. They blame the voice agent.
During our discussion, Maitreya Wagh summed this up in one line.
"It's a bit of both. It's like internally routing the calls, multi-agent and multi-workflow orchestration."
The word orchestration often sounds like an engineering term. In reality, it's closer to operational coordination.
A customer support call shouldn't follow the same path as a collections call. A research interview shouldn't consume the same resources as a balance enquiry. Some conversations need deeper reasoning. Others need enterprise data. Some need another workflow entirely. The platform has to decide that while the customer is still talking.
That's a very different problem from generating text.
This is also why voice AI architecture is becoming increasingly modular.
Very few enterprise teams expect to keep the same speech model, language model or voice provider for years. New models arrive constantly, costs change, and performance improves. Locking an entire deployment to one provider may simplify the first release, but it makes every future upgrade harder.
The organisations preparing for long-term deployment are solving a different problem. They're building systems where individual components can be replaced without disrupting the conversation or rewriting the workflow.
Open source projects have made it possible to build capable voice agents remarkably quickly. Operating them inside a business is still where most of the engineering effort sits. Reliability, governance, integrations and workflow coordination usually become bigger challenges than the conversation itself.
That's why orchestration is starting to matter far beyond engineering teams. It's becoming an architectural decision that affects how quickly an organisation can adopt new technology, expand into new use cases and scale operations without rebuilding everything from scratch.
The Real Cost of Voice AI Isn't the Model
The first question most organisations ask about voice AI is whether it works.
The question that follows is far more difficult: can it keep working without becoming too expensive?
That shift usually happens once a pilot moves into production. Early testing usually feels predictable. Response times remain consistent, infrastructure stays stable, and operating costs appear manageable. The true economics only become visible as usage grows. Nothing stresses an architecture quite like scale. A few hundred conversations rarely expose weaknesses. Thousands almost always do.
It's also where many assumptions begin to fall apart.
During our conversation, Maitreya Wagh observed that cost optimisation isn't something teams solve after building a voice agent. It is shaped by the architectural decisions made from the very beginning.
That becomes easier to understand when you look beyond the language model.
Every conversation passes through several layers before a customer hears a response. Speech has to be recognised, interpreted, processed, converted back into natural speech, and delivered through telephony infrastructure. In many enterprise environments, the system is also retrieving information from internal databases, updating records, or triggering business workflows in the background. Each layer adds its own operational cost.
Looking at one component in isolation rarely tells the full story.
The same applies to model selection. Not every interaction needs the most capable or the most expensive model. A caller checking a policy status has very different requirements from someone discussing a complex claim or completing identity verification. Treating every conversation the same may simplify the architecture, but it also makes large-scale deployments unnecessarily expensive.
The trade-off is rarely between quality and cost alone. It's about deciding where higher capability genuinely improves the customer experience and where it doesn't.
The same balancing act extends beyond models.
Reducing inference costs may expose limitations in telephony infrastructure. Improving response speed can increase cloud utilisation as more conversations run simultaneously. Scaling one part of the system often shifts pressure somewhere else, which is why production engineering becomes a continuous process of optimisation rather than a one-time exercise.
Long-term success depends less on the cost of individual models and more on the economics of the entire platform. Customers never experience speech recognition, text-to-speech, cloud infrastructure, or APIs as separate technologies. They experience a single interaction, and that interaction is only as efficient as the system supporting it.
For organisations planning long-term deployments, that is the more useful way to think about economics. The goal isn't simply to reduce model costs. It's to build a voice platform that continues to deliver reliable conversations while remaining commercially sustainable as adoption grows.
Reliable Voice AI Depends on the System, Not Just the Models
Choosing the right speech model or language model is often treated as the biggest technical decision in voice AI. In reality, production systems rarely succeed or fail because of one model alone.
They succeed because every moving part continues working together when conversations become unpredictable.
A single customer interaction may pass through speech recognition, a language model, text-to-speech, telephony infrastructure, authentication services, internal databases, and business applications before a task is completed. If any one of those components slows down or becomes unavailable, the customer doesn't see which service failed. They simply experience a conversation that no longer works as expected.
During our discussion, Maitreya Wagh explained that production systems increasingly rely on multiple agents and workflows operating together rather than a single AI model handling everything.
"Multiple things that happen during the call itself are different agents connected to one another."
That reflects a broader shift in how enterprise voice systems are being designed.
Instead of expecting one large model to manage every task, organisations are breaking conversations into smaller responsibilities. One service may verify identity, another may retrieve information from enterprise systems, while another manages the conversation itself. This makes the system easier to maintain, but it also introduces a new challenge. Every additional dependency becomes another point where failures can occur if the overall architecture is not designed carefully.
The same thinking applies to the underlying AI models.
Voice AI is evolving too quickly for organisations to assume that today's preferred provider will remain the best option throughout the life of a deployment. Better speech models, faster language models, and improved voice synthesis systems continue to emerge. Building around a single provider may simplify the first deployment, but it often makes future improvements more difficult.
As Maitreya put it,
"You want the sort of layer that gives you access to the best at that point of time for each of your calls."
That flexibility is becoming an important part of production architecture. It allows organisations to improve individual parts of the system without rebuilding the entire application every time the technology landscape changes.
For enterprise teams, reliability is no longer measured only by model quality. It depends on whether the complete system can continue delivering consistent conversations as business requirements, customer expectations, and AI capabilities evolve. The strongest production deployments are designed to absorb change rather than resist it.
Monitoring Determines Whether Voice AI Actually Delivers Value
Building a voice AI system that can answer calls is relatively straightforward compared to understanding whether those conversations are actually successful.
Once voice AI moves into production, organisations need answers that go beyond uptime and response times. Did customers complete the task they called for? Where did conversations break down? At what point did customers interrupt the agent, ask to speak to a human, or abandon the interaction altogether? Without those answers, improving the system becomes largely a matter of guesswork.
During our discussion, Maitreya Wagh highlighted that monitoring is one of the areas that deserves far more attention as voice AI deployments mature.
"We should be focusing very aggressively on monitoring and evaluating."
The challenge is that monitoring a voice AI system is fundamentally different from monitoring traditional software. Infrastructure metrics remain important, but they only tell part of the story. A system may have low latency, stable uptime, and accurate speech recognition, yet still deliver poor customer outcomes if conversations fail to reach a meaningful resolution.
For production deployments, technical performance and business performance need to be measured together. Engineering teams may track response latency, interruption rates, speech recognition accuracy, or failed API calls. Business teams are more likely to focus on task completion, first-call resolution, customer satisfaction, compliance, or conversion rates. Looking at only one side creates an incomplete picture of how the system is performing.
This broader approach is reflected in the NIST AI Risk Management Framework, which encourages organisations to continuously monitor AI systems within the context in which they are used rather than relying only on technical performance. For enterprise voice AI, that means evaluating whether the system is consistently delivering the outcomes it was designed to achieve while remaining reliable as customer behaviour and business requirements evolve.
Another important consideration is that no two organisations define success in exactly the same way. A bank may measure whether identity verification was completed securely. An insurer may focus on claims resolution and compliance. A healthcare provider may care more about appointment completion and patient experience. The technology may be similar, but the measures of success are determined by the business objective.
As voice AI becomes part of day-to-day operations, monitoring is no longer just an operational task. It becomes the feedback loop that determines where systems improve, where workflows need redesigning, and where human intervention is still required. Organisations that invest in continuous evaluation will be better positioned to refine their voice AI over time instead of relying on assumptions made during the initial deployment.
Voice AI Is Entering Its Specialisation Phase
Most technology markets evolve in similar ways. During the early years, companies often try to build every layer of the stack themselves. As the market matures, those roles become more specialised. Some organisations focus on core infrastructure, others build platforms, while others solve specific business problems on top of those foundations.
Voice AI seems to be following a similar path.
During our conversation, Maitreya Wagh summed up this shift with a simple observation:
"Everyone's doing everything right now, but soon companies are going to start figuring out what they do best."
That change is already beginning to take shape. Instead of building speech recognition, language models, orchestration, telephony infrastructure and enterprise applications under one roof, many organisations are starting to concentrate on the areas where they can create the most value. The result is a broader ecosystem where specialised technologies work together rather than competing to do everything.
For enterprise buyers, this changes how voice AI should be evaluated. Choosing a platform is becoming less about finding a single vendor that offers every capability and more about selecting technologies that integrate well, remain flexible, and can evolve as business needs change. The ability to replace or upgrade one part of the stack without rebuilding the entire system is becoming a practical advantage rather than a technical preference.
The conversation also pointed towards another shift. Much of today's attention remains focused on high-volume customer service and outbound calling, but some of the most valuable opportunities may lie elsewhere. Research interviews, financial advisory, insurance consultations and healthcare conversations require systems that can maintain context, reason across longer discussions and support more thoughtful interactions. Those use cases demand a different level of maturity than simply answering routine customer queries.
As voice AI continues to evolve, the organisations creating long-term value may not be the ones trying to build every layer themselves. They are more likely to be the ones that understand where they can contribute most and build dependable systems around that expertise.
From Demonstrations to Dependable Systems
Voice AI has moved well beyond proving that machines can hold natural conversations. The bigger challenge today is making those conversations dependable enough for businesses to trust them in everyday operations.
The lessons explored throughout this discussion reflect a broader shift taking place across the industry. Building a capable voice agent is only one part of the journey. The harder work begins when organisations need that system to perform consistently across thousands of real conversations while balancing latency, operational cost, reliability, governance and measurable business outcomes.
Those challenges cannot be solved by better models alone. They require careful engineering, thoughtful system design and continuous evaluation after deployment. As voice AI becomes part of core business operations, success will increasingly depend on the quality of the production environment surrounding the models rather than the models themselves.
The conversation with Maitreya Wagh offered one perspective on these production challenges, but the lessons extend far beyond a single company. They reflect the questions that engineering teams, product leaders and enterprise decision-makers across the industry are now working to answer.
At SubVerse AI, we continue exploring these conversations with founders, engineers and technology leaders building AI systems for real-world deployment. If you're interested in practical insights into enterprise AI and production engineering, follow InsurTech Insiders: The Insurance AI Community for more conversations, analysis and lessons from teams solving these challenges at scale.
More Blogs
Stay ahead with the newest advancements in AI automation. Discover productimprovements, feature releases,
