Descriptions:
Fahd Mirza walks through a complete local installation and live test of NVIDIA’s newly released Nemotron 3.5 Lightning — a 30-billion-parameter mixture-of-experts model with only 3 billion parameters active per token, built on a hybrid Mamba 2 and attention architecture with a 1-million-token context window. Released under the Open MWD 1.1 license with weights, training data, and recipes all publicly available, Lightning is explicitly positioned not as a frontier reasoning model but as an “execution layer” for long-running agentic workflows: high-volume tool calls, output validation, and result formatting at a fraction of the compute cost of full-size frontier models.
Mirza serves the model locally using vLLM on an Ubuntu server, walking through specific configuration flags including the Mamba backend FlashInfer kernel, FP16 memory storage for Mamba state (cutting VRAM roughly in half), and stochastic rounding to prevent quantization error accumulation across long sequences. Total VRAM consumption lands near 75 GB, with options to reduce via KV cache tuning. He then tests the model against a real broken FastAPI port-management application — without hinting where the bug is — and Nemotron identifies and fixes a draft-clearance logic error in its first pass, with impressive speed.
NVIDIA’s own benchmarks show Lightning trailing frontier models (including Opus 5 and Kimi) on overall intelligence indices but winning on instruction-following and long-context tasks, a positioning Mirza validates as honest and commercially sensible for agentic pipelines needing reliable, fast, cost-effective execution rather than frontier reasoning.
📺 Source: Fahd Mirza · Published August 11, 2026
🏷️ Format: Hands On Build
