Summary
Venky B, founder and CEO of Plivo, delivers a conference talk at AI Engineer 2026 on the five most common failure modes developers encounter when taking voice AI agents from proof of concept to production. Drawing on Plivo’s experience processing over a billion voice calls monthly across their full-stack AI agent platform — which includes a programmable speech pipeline, a no-code AI Agent Studio, and their own SIP trunking layer — the talk cuts past the hype to address what actually breaks in the real world.
The most discussed failure mode is latency, specifically time-to-first-audio from the moment a user stops speaking. Venky walks through the full cascaded pipeline (STT, LLM, TTS, turn detection) and practical mitigation strategies at each layer. On model selection, he recommends mixture-of-experts models for generic use cases and a minimum of 8–12 billion parameter models for fine-tuning on domain-specific applications like healthcare, prioritizing fast token generation and reliable tool-calling. A key insight: even state-of-the-art transcription engines hit 4–6% word error rate on clean eval sets, but real-world noisy calls with accents and domain vocabulary routinely push that into double digits.
Builders using orchestration frameworks like LiveKit or Pipecat will find direct relevance here, as Venky’s lessons come from actual customer deployments rather than controlled benchmarks, covering both the hybrid model strategy (small conversational model plus larger tool-calling model) and how to design for transcription brittleness from the start.
📺 Source: AI Engineer · Published September 15, 2026
🏷️ Format: Keynote Launch
