-
vLLM and PagedAttention: why LLM serving throughput jumped 10x
A language model can generate only one next token per sequence at a time. That sounds like an inherently serial workload, and at the level of one request it largely is.
A language model can generate only one next token per sequence at a time. That sounds like an inherently serial workload, and at the level of one request it largely is.