Summary
Cole Medin explains why open-weight models like Kimi, Qwen, GLM, and DeepSeek still fall short of frontier models for serious AI coding, even when their benchmark numbers look competitive. He points to Terminal Bench 4.0, where Kimi K 3.1 Pro Max edges out GPT-6 Astra and Claude Opus 5.5, yet says real-world results often disappoint.
His central finding is a split by task length. On shorter tasks, including complex ones that need deeper reasoning, open models perform about as well as frontier models at a fraction of the cost, which makes them a strong fit for many business automations and individual agent workflow steps. Medin cites the view that open-weight models could handle most enterprise agentic steps, especially where sensitive data favors running models locally. On longer-running tasks driving a full harness, he says they break down.
To test this, Medin ran a controlled experiment using his AI software factory, a workflow with planning, implementation, review, and play-testing stages. He had Claude (Opus 5.5 and Sonnet 5.5), GPT-6 Astra, and Kimi K 3.1 Pro each build the same video game, called Northstar, then compared the results. He closes with practical ways to bring open models into a workflow without relying on them for everything.
📺 Source: Cole Medin · Published November 05, 2026
🏷️ Format: Comparison
