Inference fundamentals
Inference has two phases: prefill processes the prompt in parallel, decode generates tokens one at a time. Most performance work targets one or the other.
Stable: this rarely changes. Reviewed 17 Oct 2026.
Key ideas
- TTFT (time to first token) is driven by prefill and queueing; TPOT/ITL by decode speed.
- The KV cache stores attention keys and values for past tokens; it is the main memory cost of long contexts.
- Continuous batching packs many requests together to raise GPU utilisation.
- Speculative decoding uses a small draft model to propose tokens that the large model verifies in one pass.
- Decode is memory-bandwidth bound, so smaller weights (quantisation) speed it up.
Common mistakes
- Reporting only average latency; track p95 and p99 TTFT and total time.