Summary
Kanesh Manuja, principal engineer at Twilio, delivers a production-focused deep dive into LLM gateway architecture, drawing on lessons from operating AI infrastructure at enterprise scale. The central framing: an LLM gateway must simultaneously optimize availability, latency, guardrails, and cost — and during any degradation event, teams must consciously choose which to sacrifice.
On availability, Manuja covers per-request sequential and parallel fallback strategies, circuit-breaker patterns adapted specifically for slow and expensive LLM calls, and a frequently overlooked failure mode: teams provision and test primary providers rigorously but under-invest in fallback capacity — which is the last line of defense and should carry even more headroom. On latency, he argues against gateway-wide P99 metrics for mixed workloads (embedding requests, chat, reasoning models all have radically different latency profiles) and recommends tracking P99 per model per route. Reasoning models are called out specifically: the same prompt can take anywhere from 2 to 60 seconds in production, making timeout configuration per model class critical.
The talk also covers streaming’s impact on fallback flexibility (once you’ve committed to a provider mid-stream, you can’t switch), the nuances of cross-provider normalization as tool-calling schemas and token limits still diverge despite OpenAI-compatible format convergence, and guardrails as an unreliable dependency requiring explicit fail-open versus fail-close policies. Practical and battle-tested guidance for any team building or operating AI middleware at scale.
📺 Source: AI Engineer · Published August 28, 2026
🏷️ Format: Deep Dive
