Summary
In this workshop from Anthropic’s Claude channel, an Anthropic engineer walks through the complete process of designing, building, and acting on evals for a slide generation agent — a concrete, end-to-end example of what the team calls “evals for taste” and the hill-climbing approach to iterative agent improvement.
The session opens by distinguishing generic public benchmarks — SWE-bench, Terminal Bench, BrowseComp, OSWorld, ARC-AGI 2 — from the custom evals that product developers actually need. Public benchmarks measure broad model capability across standardized tasks, but if you’re building a slide generation agent or any domain-specific system, those benchmarks say little about your specific quality dimensions. The speaker’s recommendation is direct: build your own evals, benchmark models against your use case, and use the right model for each job.
The bulk of the session covers designing concrete graders for slide output. Programmatic checks include emoji count, total slide count (verifying the agent follows the five-slide instruction), presence of images, font size minimums, and shape density as a proxy for clutter. LLM-judge graders score color contrast, layout quality, and text density on a 0–5 scale using a structured system prompt. The speaker emphasizes a core principle throughout: if you cannot act on a grader’s output — if a failing score doesn’t point to a specific change in your agent or prompts — that grader should not be in your eval suite. The session closes by connecting evals back to the development loop: measure, identify weaknesses, modify the agent, re-run, and track improvement over successive iterations.
📺 Source: Claude · Published May 23, 2026
🏷️ Format: Deep Dive
