-
Cutting inference cost: caching, routing, and small-model fallbacks
Every request to a large language model has a cost. For a small prototype, that cost may seem insignificant. But production AI applications and chatbots can process millions of requests, often with le
-
Speculative decoding and other tricks that make inference 2x faster
A modern large language model can process thousands of input tokens in parallel, yet generating its response still involves a surprisingly inefficient process: producing one token at a time.
-
Ensembling at inference time: voting, stacking, and when it’s worth the latency
A model that achieves 92% accuracy in 2 milliseconds may be more valuable in production than an ensemble that achieves 93% accuracy in 40 milliseconds. But what if the ensemble reaches 95%? And what i
-
Quantization in practice: running a capable model on one consumer GPU
Running a capable language model locally is mostly a memory-management problem.
-
XGBoost in production: GPU training, out-of-core data, model persistence, and fast inference
Productionizing gradient-boosted trees is not one decision. It is a chain of decisions about compute, memory, serialization, and request handling:
-
ONNX: one model format to run everywhere, from server to browser
Modern AI deployment rarely happens in the same environment where a model was trained.
-
Context windows, KV-cache, and why long prompts get expensive
Modern large language models can accept remarkably long context windows. It’s tempting to assume that if a model supports 128K, 200K, or even 1M tokens, you should simply keep appending more informati
-
Intro to scaling ML inference
Scaling machine learning inference efficiently is as critical as training a good model. As models grow larger and more complex, the challenge shifts from accuracy to throughput, latency, and cost optimization. This post introduces practical strategies, architectures, and tools used in 2025 to scale ML inference across CPUs, GPUs, and distributed environments.