Large Language Model Serving and Inference: Latency, Throughput and Scaling

TTFT (Time To First Token) is the latency from when the client sends the request to when the first token is received. It includes queue time and prefill time—the time to process the entire input prompt in a single forward pass. TTFT is dominated by input length: longer prompts mean more tokens to process in prefill, which increases TTFT.

Inter-token latency is the time between consecutive generated tokens during the decoding phase. It is dominated by the model's forward pass time for a single token step, which depends on model size, batch size and GPU compute. Inter-token latency is relatively stable during generation but can increase if the batch grows (more concurrent requests sharing GPU compute).

Throughput is the total number of tokens generated per second across all concurrent requests. Throughput and per-request latency have a fundamental trade-off: larger batches increase throughput (better GPU utilisation) but may increase per-request latency (more time per token step). The serving system must balance these based on SLOs.

Continuous batching (also called in-flight batching or iteration-level batching) dynamically adds new requests to and removes completed requests from a batch at every token generation step. Unlike static batching (which waits to fill a batch before processing), continuous batching starts processing a request immediately and can join new requests mid-generation. This eliminates head-of-line blocking where short requests wait behind long ones.

The KV cache stores the key and value tensors computed during attention for all previous tokens. Without the KV cache, each new token would require recomputing attention over all previous tokens from scratch—an O(n²) cost. With the KV cache, each new token only requires computing attention for that token against the cached keys and values—an O(n) cost. The KV cache is the primary consumer of GPU memory during inference.

Memory management of the KV cache is critical. vLLM uses PagedAttention, which allocates KV cache memory in blocks (like virtual memory paging) rather than contiguously, reducing fragmentation and enabling higher batch sizes. The KV cache grows with context length and batch size—longer contexts and more concurrent requests both increase memory pressure.

Quantization reduces the precision of model weights (and sometimes activations) to reduce memory usage and increase inference speed. Common formats include INT8, FP8 and 4-bit (NF4, GPTQ, AWQ). Quantization reduces VRAM requirements—4-bit quantization can reduce model memory by approximately 4× compared to fp16. The quality cost depends on the quantization method and model size; larger models are generally more robust to quantization.

Speculative decoding uses a smaller, faster draft model to generate candidate tokens, which the larger target model verifies in a single forward pass. If the draft model's tokens are correct, they are accepted at the cost of one forward pass instead of N. If they are incorrect, the target model generates the correct token. Speculative decoding can significantly reduce TTFT and inter-token latency for workloads where a good draft model is available.

Both techniques involve trade-offs. Quantization reduces memory and may increase throughput but can affect output quality—always evaluate on your workload before deploying. Speculative decoding adds complexity (managing two models) and only helps when the draft model is sufficiently aligned with the target model.