Recent Articles
16/20 - Streaming Generation: The First Token Is a Product Decision
How to design token streaming from model to browser with SSE or gRPC, buffering, cancellation, backpressure, usage accounting, and terminal events.
15/20 - Memory Offloading: Trading Bandwidth for Capacity
A systems guide to moving weights and KV cache between GPU, CPU, and NVMe, including prefetch, pinning, overlap, and latency tradeoffs.
14/20 - Dynamic Batching: Waiting Microseconds to Save Milliseconds
How dynamic batchers collect live requests, select batch sizes, manage queue delay, priorities, shapes, and throughput-latency tradeoffs.
13/20 - Graph Optimization: Teaching ONNX and TensorRT to See the Whole Model
How constant folding, node elimination, fusion, layout selection, precision propagation, and engine building optimize inference graphs.
12/20 - Sequence Parallelism: Divide the Tokens, Not the Meaning
A careful guide to sequence parallelism, its relationship to tensor and context parallelism, communication patterns, and long-context inference.
11/20 - Pipeline Parallelism: Turning Model Depth into an Assembly Line
How pipeline parallelism splits transformer layers across GPUs, where pipeline bubbles come from, and when it helps or hurts LLM inference.
10/20 - Tensor Parallelism: Splitting One Layer Across Many GPUs
A rigorous but approachable guide to column and row tensor parallelism, collectives, topology, latency, and production sizing for LLM inference.
9/20 - Quantized Kernels: Why a 4-Bit Model Is Not Automatically Fast
How packing, dequantization, scaling, fused GEMM, group size, and hardware support determine whether quantized LLMs deliver real speedups.
8/20 - Mixed Precision Inference: Spend Bits Where They Matter
A practical mental model for FP32, BF16, FP16, FP8, FP4, accumulation precision, scaling, calibration, and mixed-precision validation.
The Human Control Plane: What a VP Operations Must Build in an AI Company
A practical operating guide for the VP Operations role in an AI company: cadence, metrics, talent, cost discipline, customer trust, governance, and execution at speed.