Summary
Bohan Li from EliseAI presents the architecture behind their real-time voice agent system at AI Engineer 2026, drawing an analogy to self-driving car stacks: perception (transcription), planning (LLM), and controls (TTS). The talk details several engineering techniques designed to maintain frontier-model intelligence without sacrificing the low latency that voice interactions demand.
The centerpiece is a “streaming speculative transcriber” that layers a fast streaming model like Flux beneath a slower, more accurate batch transcription layer (Scribe V2). The corrective layer cancels or fires based on whether incoming audio context supersedes the prior hypothesis — producing both speed and accuracy without forcing a tradeoff. On the LLM side, EliseAI uses background agents to pre-execute tool calls in parallel with early response generation, pushing results into the main agent’s context so it can respond without additional inference round trips. For text-to-speech, they introduce a prefix cache that reuses previously generated audio for repeated or predictable word sequences, effectively hiding generation latency by playing cached audio while the unique tail of the response is still being synthesized.
Together, these techniques allow EliseAI to run frontier-class LLMs in real-time voice applications — a combination typically considered impractical given inference speed constraints. The talk is highly concrete, walking through specific detection and cancellation scenarios with timing context, making it useful reference material for engineers designing cascaded voice pipelines.
📺 Source: AI Engineer · Published September 15, 2026
🏷️ Format: Keynote Launch
