-
Speculative decoding and other tricks that make inference 2x faster
A modern large language model can process thousands of input tokens in parallel, yet generating its response still involves a surprisingly inefficient process: producing one token at a time.