Summary
Datadog AI lead Diane Huang Lin — formerly of Alexa, Vicarious (now Google DeepMind), and Zscaler — presents a systematic treatment of one of the most frustrating problems in production AI systems: the same model, given the same input, returning semantically different outputs across runs. The talk goes beyond attributing this to simple stochasticity and instead identifies the structural cause: data points that fall near a decision boundary, where even human experts disagree.
Lin illustrates the real-world stakes through a cybersecurity triage scenario — an AI agent labeling security alerts as malicious or benign — where flip-flopping verdicts erode customer trust and cost deals in competitive evaluations. She then introduces a principled solution borrowed from classical machine learning: active learning. Rather than checking all model outputs in production (which defeats the purpose of automation), active learning surfaces only the ambiguous cases — those where the model’s confidence is lowest or where repeated runs produce disagreement — and routes them for human review and label correction.
The talk covers how the traditional active learning pipeline maps onto LLM agents, why uncertainty scoring from LLMs tends to be unreliable (models often don’t know what they don’t know), and why query-by-committee — running the same input multiple times and flagging disagreements — is a more practical selection strategy. Experimental results are presented to validate the approach’s feasibility in real deployments. Engineers building agents for high-stakes classification tasks will find the framing and methodology directly applicable.
📺 Source: AI Engineer · Published July 20, 2026
🏷️ Format: Deep Dive
