Decoding and sampling

The model outputs a probability distribution for the next token; the decoding strategy decides which token is picked.

Stable: this rarely changes. Reviewed 17 Oct 2026.

Key ideas

Temperature sharpens (low) or flattens (high) the distribution. Use 0 to 0.3 for extraction and classification, higher for ideation.
Top-p keeps the smallest token set whose probability sums to p; top-k keeps the k most likely tokens.
Temperature 0 is not fully deterministic on most hosted APIs because of batching and floating-point effects.
Stop sequences and max tokens bound output; logprobs expose confidence for routing and calibration.
Reasoning models often fix or ignore sampling parameters; check provider docs before tuning them.