Summary
Engineers from DoorDash’s GenAI platform team — Nachiket Paranjape and Swaroop Chitlur Haridas — share how they built a cross-functional evaluation platform to help product teams balance accuracy, latency, and cost across LLM and agentic applications. The platform serves diverse internal customers including consumer discovery, personalization ML, and multi-agent systems teams, each requiring fundamentally different evaluation approaches: session-level quality judgments, scaled human labeling, and trajectory-based agent evals.
The platform evolved through three design philosophies: UI-first (empowering non-engineers like strategy-and-operations staff and PMs to annotate data), API-first (unblocking engineering teams to build their own tooling), and workflow-first (enabling even non-technical stakeholders to run operations via coding agents). A particularly practical insight: rather than building bespoke annotation UIs for every use case, DoorDash stabilized their underlying table APIs and empowered partner teams to vibe-code their own interfaces on top — with Claude Code and similar tools producing clean, functional annotation UIs for image labeling, manual testing, and other workflows.
The talk outlines the full eval lifecycle: capturing traces and sessions, sampling down to reviewable sets, annotating with domain expert input, creating golden datasets, and calibrating LLM judges. The central argument is that AI evaluation is a team sport requiring strategy, product, operations, and engineering to collaborate — and that platform teams succeed by building primitives that scale across that collaboration rather than trying to centralize every decision.
📺 Source: AI Engineer · Published August 28, 2026
🏷️ Format: Workflow Case Study
