Summary
Harshul Jain, senior software engineer at Audible and author of an open-source LLM inference handbook, teams up with AI researcher Tanmay Sah for a two-hour workshop delivered at the AI Engineer conference. The session is structured as a ground-up curriculum: starting from what LLM inference actually is, walking through the economic pressures driving optimization demand (the $23 billion inference market, OpenAI’s GPT-3 training cost vs. recurring inference costs), and building toward practical GPU selection and model optimization strategies.
The hands-on portions use Jupyter notebooks hosted on Modb with free RTX 6000 GPU access, walking through a capacity calculator that helps engineers fix latency or throughput as the primary constraint before selecting hardware. The workshop covers both model-level optimizations — quantization, distillation, architectural choices — and serving-level optimizations including batching strategies and engine selection, with a benchmark report and decision chart for choosing among available serving frameworks.
The content targets beginner-to-intermediate ML engineers who may understand transformer architecture but haven’t yet dealt with production inference economics. All slides, notebooks, and the benchmark report are available in the public GitHub repository referenced during the session, making this a reusable reference for teams standing up their first inference infrastructure or auditing existing deployments for cost efficiency.
📺 Source: AI Engineer · Published September 08, 2026
🏷️ Format: Deep Dive
