-
Model cascades and routing: answer cheap when you can, expensive when you must
Large language models have changed how software answers questions, writes content, summarizes documents, and automates workflows. But production AI systems rarely have a single “best” model choice for
-
vLLM and PagedAttention: why LLM serving throughput jumped 10x
A language model can generate only one next token per sequence at a time. That sounds like an inherently serial workload, and at the level of one request it largely is.
-
Serving many models on one GPU: Triton, model repositories, and hot-swapping
A GPU is not a Kubernetes pod. Treating every model as if it deserves an entire accelerator can leave expensive hardware mostly idle, especially when you serve small classifiers, regressors, ranking m
-
Dynamic batching: the single biggest throughput lever in model serving
A model server processing one request at a time pays fixed costs for every execution:
-
ONNX: one model format to run everywhere, from server to browser
Modern AI deployment rarely happens in the same environment where a model was trained.
-
From pickle to production: wrapping a model in a real API that won’t fall over
Training a machine learning model is only half the job. The other half is making it available reliably, safely, and predictably.