Descriptions:
Annabell Schäfer, growth engineer at Langfuse — the open-source AI observability and evaluation platform — presents a structured experiment on self-optimizing AI loops at the AI Engineer conference. Her central argument is that automatic self-improvement only works reliably when developers invest early in defining clear, domain-specific target functions and high-quality evaluators, rather than assuming a general LLM can judge its own outputs accurately.
The experiment used arXiv paper classification as a testbed: a GPT-4o nano classifier was paired with a Claude Sonnet 4.8 optimizer running through Claude Code, which proposed iterative prompt updates referencing OpenAI’s GPT-5 prompting guide. Using a 200-item fit set, 100-item validation set, and 300-item held-out test set, the loop improved accuracy from a 68% baseline to 83% after three to four iterations — a 15-percentage-point gain — and the improvement generalized to the unseen test set at 80.2%.
The most instructive finding is that the biggest jump came in the very first iteration: the optimizer added a general classification approach, disambiguation rules between similar labels, and few-shot examples for frequently misclassified pairs — not what Schäfer expected (label descriptions). Her broader takeaway for AI application teams is that those who invest in building out a genuine “target function” grounded in domain expertise are consistently the ones who ship improving applications with confidence, while teams without it plateau or regress.
📺 Source: AI Engineer · Published July 18, 2026
🏷️ Format: Deep Dive
