Temperature (Language Model)

Temperature is a sampling parameter that determines how deterministic or random a language model's output is. At every generation step, the model produces a probability distribution over all possible next tokens. Temperature scales this distribution before the sampling step.

At temperature zero (or very close to zero), the model always selects the highest-probability token at each step, producing deterministic, consistent output. This is useful for factual tasks, code generation, and any context where reproducibility matters. At temperature one, the model samples proportionally from the distribution as it naturally appears. At temperatures above one, the distribution is flattened, making lower-probability tokens more likely and producing more diverse and creative outputs at the cost of coherence and accuracy.

Formally, temperature is applied by dividing the model's logits (unnormalized log-probabilities) by the temperature value before applying the softmax function. A temperature below one sharpens the distribution (makes the highest-probability token even more likely), and a temperature above one flattens it.

Temperature is distinct from top-p sampling (nucleus sampling), which truncates the sampling distribution to the smallest set of tokens whose cumulative probability exceeds a threshold p. Both parameters control output diversity, and they are often used together. A low temperature with top-p sampling is a common configuration for production AI assistants: it reduces randomness while preventing the model from always selecting the single highest-probability token.

In AI evaluation, understanding temperature explains why the same prompt produces different outputs across runs and why evaluation tasks focus on a specific output rather than the model's general capability.