Descriptions:
Ramp, the finance automation platform used by more than 50,000 businesses, presents detailed lessons from three years of building AI agents for expense management, accounts payable, and financial operations. The central architectural insight is a pivot the company made in early 2026: rather than maintaining dozens of specialized agents (the team had accumulated four different approaches to the same problem), they consolidated around a single agent with many skills, driven by a new conversational interface called the “omnichat” deployed across every surface of the product.
The most technically specific section covers the policy agent, which evaluates employee expense transactions against company-specific policies. To build a reliable ground-truth dataset, Ramp ran weekly cross-functional labeling sessions, then built a custom lightweight labeling tool using Claude Code and Streamlit — a one-shot build that non-engineers can modify and that deploys in seconds. This ground-truth dataset enabled rapid iteration cycles, early customer buy-in as design partners, and meaningful eval coverage before broad rollout. The team also discusses using Opus 4 (and anticipating improvements with Opus 4.6) for this internal tooling.
The talk closes with infrastructure and culture changes required to operate at this pace: moving to a single conversational UX, defining cross-functional alignment on what “correctness” means for AI outputs, and getting non-technical stakeholders to engage consistently with the labeling and eval process. Engineering leaders building AI products in data-sensitive enterprise environments will find the discussion of evals, ground-truth construction, and agent consolidation especially actionable.
📺 Source: The Pragmatic Engineer · Published March 09, 2026
🏷️ Format: Workflow Case Study
