LEARN · MULTI-MODEL INFERENCE & SERVING
Why batch size 1 wastes expensive hardware
A model server processing one request at a time pays fixed costs for every execution:
-
Queue scheduling and framework dispatch
-
CPU-to-accelerator coordination
-
Kernel launches
-
Input and output handling
-
Device synchronization
-
Model-weight and activation movement
The tensor operation may finish quickly, yet the server still pays those costs repeatedly. Small workloads can leave the accelerator waiting for the host to prepare the next unit of work.
Dynamic batching changes the unit of execution. The server briefly collects compatible requests, stacks their tensors, and executes one larger model call:
Request A ─┐ Request B ─┼──> Batch [A, B, C, D] ──> One model execution Request C ─┤ Request D ─┘
The goal is not to build the largest possible batch. It is to build the largest safe batch available without violating the latency budget.
For many models, processing 16 items together takes far less time than processing one item 16 times. The exact improvement depends on the model, accelerator, runtime, tensor shapes, precision and memory pressure.
Dynamic batching versus static batching
Static batching requires the caller to assemble a batch before sending it. That works well for offline jobs, but creates problems for online APIs:
-
Requests arrive at unpredictable times.
-
Independent clients cannot coordinate.
-
A quiet client may never fill a fixed batch.
-
Waiting indefinitely for a full batch creates unbounded latency.
Dynamic batching moves batch formation into the model server.
A typical scheduler uses two controls:
-
Maximum batch size: the largest execution permitted by model support and available memory.
-
Maximum queue delay: the longest the oldest request may wait for batch partners.
Its core behavior looks like this:
batch = [await queue.get()]
deadline = loop.time() + max_queue_delay
while len(batch) < max_batch_size:
remaining = deadline - loop.time()
if remaining <= 0:
break
try:
request = await asyncio.wait_for(
queue.get(),
timeout=remaining,
)
except asyncio.TimeoutError:
break
batch.append(request)
await run_model(batch)
Under heavy load, the queue fills quickly and batches reach their maximum size without intentional waiting. Under light load, the timeout prevents a lone request from being trapped in the queue.
Current NVIDIA Triton guidance recommends enabling the default dynamic batcher first, measuring it, and then increasing the maximum batch size or adding queue delay only while latency remains within budget. It also advises against preferred_batch_size for most models unless particular TensorRT optimization profiles are materially faster.
Why throughput can rise faster than latency
Suppose one item takes 4 milliseconds:
Batch size 1: 1 item / 4 ms = 250 items/second
Now suppose a batch of 16 takes 11 milliseconds:
Batch size 16: 16 items / 11 ms ≈ 1,455 items/second
The execution latency increased from 4 to 11 milliseconds, but throughput rose almost sixfold.
The useful condition is:
T_batch(n) < n × T_batch(1)
Batching helps while one batched execution is cheaper than the equivalent number of separate executions.
The benefit eventually plateaus because of:
-
Compute or memory-bandwidth saturation
-
Accelerator-memory limits
-
Padding from uneven tensor or sequence lengths
-
Longer queueing and execution times
-
KV-cache pressure in generative models
-
Backend-specific kernel and optimization-profile behavior
Recent GPU-level analysis found that large-batch LLM inference can remain memory-bound even when conventional wisdom suggests it should have become compute-bound. In the evaluated attention workloads, more than half of computation cycles could be stalled by memory access. A high batch size is therefore not proof that the GPU’s arithmetic units are efficiently occupied.
Runnable demo: batch size 1 versus dynamic batching
The following Python 3.11+ program simulates an accelerator-backed server.
It models two common properties:
-
Every model execution has fixed overhead.
-
Additional items increase execution time sublinearly until the simulated device saturates.
The same 800-request workload is run twice:
-
Batch size 1 with no collection window
-
Dynamic batches of up to 16 with a 2-millisecond window
from __future__ import annotations import asyncio import statistics import time from dataclasses import dataclass @dataclass(slots=True) class Request: enqueued_at: float done: asyncio.Future[float] class SimulatedModelServer: def __init__( self, max_batch_size: int, max_queue_delay_ms: float, ) -> None: if max_batch_size < 1: raise ValueError("max_batch_size must be at least 1") if max_queue_delay_ms < 0: raise ValueError("max_queue_delay_ms cannot be negative") self.max_batch_size = max_batch_size self.max_queue_delay_s = max_queue_delay_ms / 1_000 self.queue: asyncio.Queue[Request] = asyncio.Queue() self.worker_task: asyncio.Task[None] | None = None self.batch_sizes: list[int] = [] async def start(self) -> None: if self.worker_task is not None: raise RuntimeError("server is already running") self.worker_task = asyncio.create_task(self._worker()) async def infer(self) -> float: loop = asyncio.get_running_loop() request = Request( enqueued_at=loop.time(), done=loop.create_future(), ) await self.queue.put(request) completed_at = await request.done return (completed_at - request.enqueued_at) * 1_000 async def close(self) -> None: if self.worker_task is None: return self.worker_task.cancel() try: await self.worker_task except asyncio.CancelledError: pass self.worker_task = None async def _worker(self) -> None: loop = asyncio.get_running_loop() while True: first = await self.queue.get() batch = [first] deadline = loop.time() + self.max_queue_delay_s while len(batch) < self.max_batch_size: remaining = deadline - loop.time() if remaining <= 0: break try: request = await asyncio.wait_for( self.queue.get(), timeout=remaining, ) except asyncio.TimeoutError: break batch.append(request) self.batch_sizes.append(len(batch)) # Simulated accelerator execution: # 3 ms fixed overhead plus sublinear batch work. batch_time_s = 0.003 + 0.001 * (len(batch) ** 0.72) await asyncio.sleep(batch_time_s) completed_at = loop.time() for request in batch: if not request.done.cancelled(): request.done.set_result(completed_at) self.queue.task_done() def percentile(values: list[float], fraction: float) -> float: ordered = sorted(values) index = round((len(ordered) - 1) * fraction) return ordered[index] async def benchmark( name: str, max_batch_size: int, max_queue_delay_ms: float, request_count: int = 800, concurrency: int = 64, ) -> dict[str, float | str]: server = SimulatedModelServer( max_batch_size=max_batch_size, max_queue_delay_ms=max_queue_delay_ms, ) await server.start() semaphore = asyncio.Semaphore(concurrency) async def send_one() -> float: async with semaphore: return await server.infer() started_at = time.perf_counter() try: latencies_ms = await asyncio.gather( *(send_one() for _ in range(request_count)) ) finally: await server.close() elapsed_s = time.perf_counter() - started_at return { "mode": name, "throughput": request_count / elapsed_s, "p50_ms": percentile(latencies_ms, 0.50), "p95_ms": percentile(latencies_ms, 0.95), "mean_batch": statistics.mean(server.batch_sizes), "max_batch": float(max(server.batch_sizes)), } async def main() -> None: configurations = [ ("batch_size_1", 1, 0.0), ("dynamic_batching", 16, 2.0), ] print( f"{'mode':<20}" f"{'req/s':>10}" f"{'p50 ms':>12}" f"{'p95 ms':>12}" f"{'mean batch':>14}" f"{'max batch':>12}" ) for configuration in configurations: result = await benchmark(*configuration) print( f"{result['mode']:<20}" f"{result['throughput']:>10.1f}" f"{result['p50_ms']:>12.1f}" f"{result['p95_ms']:>12.1f}" f"{result['mean_batch']:>14.1f}" f"{result['max_batch']:>12.0f}" ) if __name__ == "__main__": asyncio.run(main())
Save it as dynamic_batching_demo.py, then run:
python dynamic_batching_demo.py
A representative run looks like this:
mode req/s p50 ms p95 ms mean batch max batch batch_size_1 238.9 266.7 269.6 1.0 1 dynamic_batching 1383.8 45.5 45.7 16.0 16
Dynamic batching delivers roughly 5.8 times the throughput in this run.
More surprisingly, latency is also lower. The 2-millisecond collection window adds a small delay, but the batch-size-1 server builds a much longer execution queue. Under sustained load, avoiding that queue can outweigh the batching delay.
This is a scheduler simulation, not a GPU benchmark. To adapt it, replace the simulated asyncio.sleep() with your real batched model call while preserving concurrency control, elapsed-time measurement and percentile reporting.
A production Triton starting point
A stateless ONNX classifier that supports batches up to 16 could use:
name: "classifier" backend: "onnxruntime" max_batch_size: 16 input [ { name: "input" data_type: TYPE_FP32 dims: [768] } ] output [ { name: "scores" data_type: TYPE_FP32 dims: [1000] } ] dynamic_batching { max_queue_delay_microseconds: 2000 }
Two milliseconds is not a universal recommendation. Treat it as a hypothesis to test.
A safer first experiment is:
dynamic_batching {}
That configuration batches requests already waiting in the scheduler without deliberately delaying the oldest request. Triton’s current batcher documentation recommends measuring this default behavior before adding delay.
Also verify that the model’s inputs and outputs actually support a leading batch dimension. Setting max_batch_size cannot make a fixed-shape model batch-compatible.
Dynamic batching is not continuous batching
Request-level dynamic batching works well for ordinary stateless inference, where one execution completes an entire request.
Autoregressive language models behave differently. Consider four generations:
A: 12 output tokens B: 40 output tokens C: 6 output tokens D: 85 output tokens
A fixed batch can leave finished slots idle while request D continues.
Continuous, iterative or in-flight batching rebuilds the active set during generation:
Step 1: [A, B, C, D] Step 7: [A, B, D, E] # C finished; E entered Step 13: [B, D, E, F] # A finished; F entered
Current vLLM documentation lists continuous batching, chunked prefill and prefix caching as core serving capabilities. Current TensorRT-LLM documentation describes in-flight batching as scheduling dynamically at each LLM step.
Use:
-
Dynamic request batching for stateless classifiers, encoders, detectors and similar models.
-
Continuous or in-flight batching for iterative generation.
-
Sequence-aware scheduling when calls carry persistent state and must remain on the same model instance.
One freshness warning: Hugging Face now describes Text Generation Inference as being in maintenance mode and recommends engines including vLLM and SGLang for new development.
Measure the batcher, not just the GPU
Do not approve a configuration based on average latency or a single utilization graph.
Measure:
-
Requests, items or tokens per second
-
p50, p95 and p99 end-to-end latency
-
Time spent waiting in the scheduler
-
Model execution time
-
Actual batch-size distribution
-
GPU utilization, memory use and power
-
Rejections, cancellations and timeouts
-
Cost per successful request
For Triton, average effective batch size can be derived as:
average batch size = nv_inference_count / nv_inference_exec_count
Triton also exposes cumulative request, queue and model-compute duration metrics through its Prometheus endpoint.
Test more than an always-full queue:
-
Constant concurrency
-
Random arrivals
-
Sudden bursts
-
Quiet periods
-
Mixed tensor shapes
-
Mixed prompt and output lengths
-
Real cancellation and timeout behavior
A configuration can look excellent at saturation yet add unnecessary delay during normal traffic.
Cherry on the cake: 81.66% decoding utilization
A recent BucketServe evaluation used four NVIDIA A100 40 GB GPUs, Llama-2 13B and mixed short- and long-sequence workloads. Its batch-aware system reported 81.66% average decoding GPU utilization, compared with 67.88% for DistServe and 60.86% for UELLM on the same evaluation. It also reported up to 3.58 times the throughput of UELLM under high load.
That is a gain of more than 20 percentage points over the lowest-utilization baseline.
It is not a pure “flip one batching flag” comparison: BucketServe combines adaptive length bucketing, memory-aware batch sizing and continuous decoding. That caveat is the lesson. Real serving gains often come from forming compatible, efficient batches, not merely making batches larger.
A practical tuning sequence
-
Benchmark batch size 1 at realistic concurrency.
-
Enable batching with no intentional delay.
-
Increase the maximum batch size until throughput plateaus, memory becomes unsafe or tail latency approaches its limit.
-
Add small queue delays only when fuller batches produce measurable gains.
-
Test bursty and low-volume periods.
-
Bucket variable-length inputs when padding becomes significant.
-
For LLMs, watch KV-cache occupancy, preemption, recomputation, TTFT and inter-token latency.
-
Repeat the benchmark after model, runtime, driver or hardware changes.
Run the demo, replace its simulated execution with your real model call, and compare batch size 1 against dynamic batching before adding another replica or purchasing another accelerator.