-
ONNX: one model format to run everywhere, from server to browser
Modern AI deployment rarely happens in the same environment where a model was trained.
-
Context windows, KV-cache, and why long prompts get expensive
Modern large language models can accept remarkably long context windows. It’s tempting to assume that if a model supports 128K, 200K, or even 1M tokens, you should simply keep appending more informati
-
Intro to scaling ML inference
Scaling machine learning inference efficiently is as critical as training a good model. As models grow larger and more complex, the challenge shifts from accuracy to throughput, latency, and cost optimization. This post introduces practical strategies, architectures, and tools used in 2025 to scale ML inference across CPUs, GPUs, and distributed environments.