Summary
Simran Arora, principal scientist at Together AI and incoming Caltech professor, presents original research asking a pointed question: can today’s frontier language models actually write high-performance multi-GPU kernels? The talk, delivered at AI Engineer, begins with a hardware-grounded primer — tracing how GPU bottlenecks have shifted from single-GPU memory access (largely solved by FlashAttention and memory-efficient architectures like Mamba) to multi-GPU communication across NVLink fabrics and NVSwitch.
Arora introduces Parallel Kernel Bench, a new benchmark built by the Together AI frontier performance team to evaluate LLMs specifically on multi-GPU kernel generation. The benchmark surfaces a persistent gap: most existing baselines — including frameworks built on NCCL like Megatron-LM and Triton Distributed — achieve below 50% of their theoretical communication-aware roofline bound. Triton Distributed, she notes, was tuned on H800 GPUs and fails to adapt efficiently to H100 architectures.
The team’s own approach involved first hand-building “Parallel Kittens,” a minimal library of multi-GPU primitives, to ground the research in real performance trade-offs before asking models to generate novel kernels. The conclusion is sobering: despite strong benchmark claims around reasoning and code generation, frontier models do not reliably understand the fundamental principles governing multi-GPU kernel design. The talk is essential viewing for ML infrastructure engineers and anyone evaluating AI-assisted kernel development tools like Triton, Mojo, or ThunderKittens.
📺 Source: AI Engineer · Published August 27, 2026
🏷️ Format: Keynote Launch
