Summary
At an AI Engineer conference, Nan Jiang from Modal presents a technical architecture for running reinforcement learning post-training workloads across geographically distributed, heterogeneous GPU clusters rather than within a single RDMA-connected datacenter. The talk targets a structural bottleneck in RL training: the standard loop couples the trainer and rollout fleet to the same high-bandwidth fabric, which is precisely the scarce, expensive, and inelastic compute configuration. Modal’s insight is that rollout workers — which generate trajectories, call environments or tools, and ship data back to the trainer — require no cross-island all-reduce operations, making them eligible to run on scattered, autoscaled capacity across providers and regions.
The proposed architecture keeps backpropagation and collective communication inside a tightly coupled trainer cluster while exporting rollout “islands” to cheaper, more available compute. Each island is a coherent serving endpoint that holds one policy version; the global interface reduces to policy weights in and trajectories out. Jiang then tackles the weight synchronization problem: how often must rollout workers receive updated weights? A key finding involves BF16 quantization and the Adam optimizer. At typical RL post-training learning rates, per-step weight updates are roughly 1,000× smaller than the BF16 rounding boundary, meaning the quantized weights served by rollout engines remain unchanged across many trainer steps — a phenomenon Jiang calls “Adam absorption.”
This sparsity means only weight diffs, not full FP32 checkpoints, need to be shipped across the network, making cross-datacenter RL practically viable. The talk includes worked numerical examples with specific learning rate bounds and BF16 epsilon values, grounding the architecture in measurable hardware constraints.
📺 Source: AI Engineer · Published August 10, 2026
🏷️ Format: Keynote Launch
