Descriptions:
Fahd Mirza covers SpatialClaw, a new training-free spatial reasoning agent from researchers at NVIDIA and KAIST that significantly outperforms prior approaches by giving the AI a persistent Python kernel rather than forcing it to write a complete program upfront or call tools sequentially through fixed JSON.
The core insight is that an agent’s capability is bounded not by which tools are available but by how they can be composed. SpatialClaw runs a five-stage loop: a planning step analyzes the question without images, a vision-language model writes a Python cell with its reasoning and stated goal, the cell is AST-checked and executed in a persistent kernel (where Matplotlib, SciPy, and SAM 3 perception tools are preloaded), and all output — including errors, variable states, and images from the show() function — feeds back as the next observation. This lets the agent catch its own mistakes mid-run. In a concrete example, SpatialClaw correctly computed a 0.9439m closest-point distance (ground truth: 0.9m) by switching from median-based centroids to SciPy’s KDTree mid-execution — something both single-pass code generation and JSON-tool approaches failed at.
Across 20 spatial benchmarks, SpatialClaw beats the previous best agent by 11.2 percentage points with zero benchmark-specific tuning, and gains hold across every backbone tested. Hardware requirements are substantial — at minimum two H100 or A100 GPUs to run the SAM 3 and Depth Anything 3 perception servers alongside the main VLM — making this a research-grade project. NVIDIA has open-sourced the full codebase on GitHub.
📺 Source: Fahd Mirza · Published July 04, 2026
🏷️ Format: Deep Dive
