Sampling: temperature & top-p

Lesson 2 of 8 · 20 min · part of the AI Prompt Engineer certification

At each step the model outputs a score for every token in its vocabulary. These scores are turned into probabilities, and then one token is sampled. Sampling parameters let you control how adventurous that choice is.

• Temperature rescales the probabilities. Low values (0–0.3) make the most likely token dominate — consistent, focused output. High values (1.0+) flatten the distribution — more diverse and creative, but more prone to errors.
• Top-p (nucleus sampling) keeps only the smallest set of tokens whose probabilities add up to p, then samples from those. It cuts off the long tail of unlikely tokens.
• Max tokens caps the length of the response (and your cost). If output stops mid-sentence, you probably hit this limit.

Interactive exercise: Pick the right setting