Summary
Nick Ung and Aka from Lyft’s data science team present their end-to-end evaluation framework for production AI agents at the AI Engineer conference, drawing on more than a year of building and iterating on Lyft’s customer support AI system. The talk walks through the complete eval lifecycle: a rigorous offline evaluation phase before any agent reaches live users, online evaluation and tracing in production, and a human-in-the-loop pipeline that surfaces failure modes back to the development team.
A central finding is that generic pre-built metrics — helpfulness, completeness, conversation naturalness — are not actionable because they produce scores with no clear connection to launch decisions. Lyft replaced these with custom Lyft-specific graders tied directly to launch gates, ensuring every evaluation score either permits or blocks deployment. They also fine-tuned a user simulator on real Lyft customer verbatim to make simulated conversations harder and more representative, deliberately accepting lower eval scores in exchange for higher production fidelity. User personas — including “bypasser” users who immediately escalate to a human agent and “refund seekers” — further ground the simulator in realistic behavior, taking inspiration from Microsoft’s UserALEN research.
The presentation is particularly useful for teams moving beyond toy eval setups: it covers LLM-as-judge design, multi-turn conversation simulation, grader calibration, and the principle that an eval too easy to pass offers no real production signal. A practical reference for any team deploying customer-facing AI agents at scale.
📺 Source: AI Engineer · Published July 19, 2026
🏷️ Format: Deep Dive
