Summary
Bloomberg Tech interviews Richard Ho, OpenAI’s Vice President of Hardware, about Jalapeno — the company’s first custom inference chip. OpenAI published benchmark results using the open-source InferenceX framework showing Jalapeno outperforms NVIDIA’s GB300 system on both throughput and latency simultaneously. Ho says the chip achieves 1.8–4x better performance per watt compared to existing alternatives, which is expected to substantially reduce per-token inference costs when it enters production. The benchmarks compared against GB300 rather than NVIDIA’s newer Vera Rubin because published, optimized GB300 numbers were already available.
Jalapeno uses HBM4 memory — one of the first chips to go to volume production with HBM4 — and features a novel architecture designed around reducing data movement between memory and compute cores. Ho explains that this design was informed by working directly with OpenAI’s research team to identify where LLM inference bottlenecks actually occur, and that the resulting memory-core affinity model is architecturally distinct from both GPUs and TPUs. Ultra-low latency is identified as critical for agentic workloads, where fast response times directly affect agent loop speed.
Ho explains that OpenAI deliberately focused its custom silicon program on inference rather than training, viewing NVIDIA as a strong training partner while building internal optionality on the inference side. Jalapeno will sit alongside NVIDIA and Cerebras chips in OpenAI’s fleet, with the company routing specific model workloads to whichever chip best matches their cost and latency profile. The chip was brought from delivery to benchmarked production in a matter of months, which Ho cites as evidence of both programmability and AI-assisted kernel optimization.
📺 Source: Bloomberg Tech · Published August 25, 2026
🏷️ Format: Interview
