Descriptions:
Phil Hetzel, head of solutions engineering at Braintrust, delivers a conference talk breaking down exactly why agent observability is a fundamentally different engineering problem from the traditional observability most teams already have in place. The talk is framed around the limits of established tools like Grafana and Datadog, which were designed to answer one question: is the system up, and is it performing within technical SLAs? That scope, Hetzel argues, is categorically insufficient for AI agents.
The core distinctions he works through include non-determinism (agents produce highly variable outputs unlike deterministic application code paths), the structural complexity of agent traces (nested spans mixing model calls, tool calls, and large volumes of unstructured text), and a dual read-pattern challenge: the platform must simultaneously support real-time trace streaming for live monitoring and analytical SQL-style querying for evaluation pipelines. Hetzel explains that Braintrust built a purpose-built database from the ground up to handle these requirements — incorporating a write-ahead log for instant trace visibility, optimized indexing for filter queries, and a forked version of Tantivy, an open-source full-text search framework, to enable text-based trace queries such as retrieving every session that referenced a specific word or phrase.
The talk is grounded in Hetzel’s 12 years of consulting experience, including leading the global Databricks practice at Slalom Consulting, and is aimed at AI engineers and platform teams deciding how to instrument production agent systems and understand where traditional observability tooling falls short.
📺 Source: AI Engineer · Published May 28, 2026
🏷️ Format: Deep Dive
