Summary
Rustem Feyzkhanov, head of the AI platform team at Snorkel AI, presents the case that private benchmarks built from production traces are the only reliable way to evaluate, release, and continuously improve enterprise AI agents. Public benchmarks like SWE-bench, TerminalBench, and CoBench orient teams to frontier capability but test narrow domains and report only pass rate — leaving out cost, latency, and retry counts that govern real deployment decisions.
The methodology Feyzkhanov describes converts live production traces into repeatable offline simulations: extract tasks and environment state from real agent runs, reconstruct them in sidecar containers (databases, API services, MCP tools, file snapshots), inject a simulated user driven by an LLM prompt, then run different agent configurations against the same environment in parallel. This apples-to-apples comparison lets teams test not just model swaps but prompt changes, tool availability, and full-stack configuration — with Snorkel running millions of such simulations monthly.
The talk gives detailed attention to the verifier layer, explaining how deterministic output checks, LLM-as-judge calls, and agent-as-judge evaluations can be layered to assess not just final answers but intermediate planning quality and tool-call correctness. Feyzkhanov frames the benchmark as serving three lifecycle roles: proving an agent works before first release, acting as a regression gate for ongoing changes, and generating fine-tuning training data from production traces. The approach is presented as an engineering discipline, not a research exercise.
📺 Source: AI Engineer · Published July 25, 2026
🏷️ Format: Deep Dive
