Recent Articles
6/21 - The Serving Layer: Triton, vLLM, KServe, Ray Serve, and SGLang
A production-focused guide to the serving layer: triton, vllm, kserve, ray serve, and sglang, with architecture, capacity math, failure analysis, and operational controls.
5/21 - Training Across the Fleet: DDP, FSDP, DeepSpeed, and ZeRO
A production-focused guide to training across the fleet: ddp, fsdp, deepspeed, and zero, with architecture, capacity math, failure analysis, and operational controls.
4/21 - The Memory of a Conversation: KV, Prefix Reuse, Speculation, and Throughput
A production-focused guide to the memory of a conversation: kv, prefix reuse, speculation, and throughput, with architecture, capacity math, failure analysis, and operational controls.
3/21 - The Inference Engine Room: vLLM, TensorRT-LLM, SGLang, and llama.cpp
A production-focused guide to the inference engine room: vllm, tensorrt-llm, sglang, and llama.cpp, with architecture, capacity math, failure analysis, and operational controls.
2/21 - Smaller Numbers, Faster Models: Quantization and Batching
How INT8, FP8, 4-bit formats, static batching, dynamic batching, and continuous batching change memory, kernels, quality, and latency.
1/21 - Inside the GPU: From SMs to HBM Without the Hand-Waving
A systems-level guide to GPU execution, SMs, warps, VRAM capacity, CUDA's memory hierarchy, occupancy, and the data movement that governs AI performance.
20/20 - Expert Parallelism: Routing Tokens Through a City of Specialists
How MoE routers, top-k expert selection, all-to-all communication, capacity, grouped GEMM, and load balancing shape expert-parallel inference.
19/20 - Prefill-Decode Disaggregation: Two Worker Pools, One Token Stream
A deep guide to separating prefill and decode workers, KV transfer, independent scaling, routing, RDMA, conditional disaggregation, and SLOs.
18/20 - Chunked Prefill: How to Stop One Long Prompt from Freezing Everyone Else
How chunked prefill divides long prompts, co-schedules them with decode, controls TTFT and TPOT interference, and chooses token budgets.
17/20 - Continuous Batching: The GPU Schedule That Never Stands Still
How iteration-level scheduling adds and removes LLM requests between decode steps, improving utilization while preserving fairness and latency.