Descriptions:
The cheapest configuration they tested was the one sending the most tokens. Across 11 presets run against their open source AI tutor, doing nothing at all to the context beat every compaction technique on recall, cost, and latency at once, and their own production defaults scored worse than leaving the history alone. Prompt caching is why. With 97% of tokens served from cache, and cached tokens up to 50 times cheaper on some APIs, compaction has to shrink a context by more than 50 times before it pays for itself, because rewriting the context invalidates the cache. Louis-François Bouchard’s framing is that summarization is potentially a trap.
Omar Solano walks through the architecture and the first run, including a knowledge base browsing tool they built, measured, and found returned identical recall while running 50% slower. Samridhi Vaid extends it: keeping the full history recovered specific details 95% of the time against 32% after summarizing, and distinctive facts survived to 800,000 tokens without visible rot. Hardware changes the answer, though. Capped locally at a 32k window, keeping everything stops being possible, and a larger parameter count does not buy a larger context window. Dense retrieval fell to 0% recall on facts buried at 400k tokens where BM25 still found them every time. The rule they land on is to name the constraint you actually have before reaching for compaction, rather than compacting by default.
Speaker info:
Louis-François Bouchard (Towards AI):
– https://x.com/Whats_AI
– https://www.linkedin.com/in/whats-ai/
– https://www.louisbouchard.ai
Omar Solano (Towards AI):
– https://x.com/omar_solano1
– https://www.linkedin.com/in/omar-solano1
Samridhi Vaid (Towards AI):
– https://x.com/samridhivaid
– https://www.linkedin.com/in/samridhivaid/
Project:
– https://github.com/towardsai/ai-tutor-app
Timestamps:
0:00 – The problem is the context, not the model
1:30 – The AI tutor, and its five requirements
5:21 – Two root problems: a finite window, a stateless model
7:52 – Context rot, cost, and latency
9:13 – The compaction toolkit, with and without an LLM
12:58 – Offloading to files, the LLM wiki, progressive disclosure
16:47 – Prompt caching, and why compaction can backfire
19:20 – When to clear, compact, and optimize for cache hits
21:53 – The tutor’s architecture, a single ReAct agent
25:49 – Hybrid search over an 8 million token corpus
28:28 – Letting the agent browse the knowledge base
31:01 – The browse tool measured: same recall, 50% slower
36:23 – The experiment setup: presets, tasks, harness
42:55 – Results: doing nothing wins on all three fronts
48:13 – Should you ever compact?
49:32 – DeepSeek, and a 50 times cache discount
50:53 – Memory: 95% against 32% after summarizing
54:38 – Cost at scale, and going local
57:09 – Local limits: bigger models, same window
58:25 – Where dense retrieval fails and BM25 holds
1:01:01 – What they finally chose
