Summary
Fahd Mirza demonstrates llama.cpp’s built-in router mode, a native feature that enables instant model hot-swapping without third-party frontends like Ollama, LM Studio, or Open Web UI. The tutorial walks through four key server flags: `–models-dir` (points to a folder of GGUF files), `–models-auto-load` (registers all models at startup without consuming VRAM), `–models-preset` (an INI configuration file for per-model settings), and `–models-max` (controls how many models can reside in VRAM simultaneously).
The demo runs on Ubuntu with an NVIDIA RTX A6000 (48GB VRAM), using two quantized GGUF models from GGML-org on Hugging Face: a 3.6B A3B1 and an MTP 27B. Mirza walks through the INI file structure in detail — a global `[static]` section defines defaults like thread count, flash attention, and context size, while individual named model sections can selectively override those defaults. The model name in each section must exactly match the GGUF filename without its extension.
The end result is a lean setup where the llama.cpp server registers all available models on startup but loads them into VRAM only on first request, automatically unloading the prior model when switching. This eliminates duplicate model storage, removes the abstraction layers that third-party tools introduce, and keeps everything operating directly inside the llama.cpp server binary — no additional containers or processes required.
📺 Source: Fahd Mirza · Published May 21, 2026
🏷️ Format: Tutorial Demo
