Summary
Andrew Qu, Chief of Software at Vercel, traces the full iterative journey of how Vercel built its internal data science agent — from a copy-paste SQL experiment to a file-system-native agent — and how those learnings shaped Vercel’s public AI SDK.
The problem: Vercel’s lean data team was bottlenecked answering ad-hoc analytics questions from marketing, sales, and finance. Qu started by dumping a Snowflake schema into a system prompt and manually running the generated SQL, just to test model capability. Each subsequent version added structure — decomposing the data science workflow into phases (question processing, schema exploration, SQL execution, reporting) and mapping those to agent sub-tasks. Despite reaching 30% eval pass rates, early user testing revealed the agent failed on unanticipated real questions, and hand-labeling more scenarios didn’t scale.
The inflection point came when Claude Code with Opus 4.5 launched. Qu observed that a minimal tool set — list file, read file, run bash — combined with a file system containing the full semantic layer outperformed the hand-engineered agent dramatically. The team rebuilt their agent to mirror this file-system-native pattern in a cloud sandbox, and eval scores roughly doubled. The talk distills this into a practical principle: agents perform better when given tools they are already well-trained on and a rich file system to explore, rather than a prescriptive custom tool API. Vercel’s AI SDK now embeds these lessons for external developers.
📺 Source: AI Engineer · Published September 14, 2026
🏷️ Format: Workflow Case Study
