What Is Top-P Sampling (Nucleus Sampling)?

When a language model generates text, it produces a probability distribution over all possible next tokens. Top-P sampling dynamically adjusts how many tokens are considered at each step by including tokens from most probable to least probable until their cumulative probability reaches the threshold P (typically between 0 and 1). For example, with P=0.9, the model considers only the tokens that together account for 90% of the probability mass, ignoring the long tail of unlikely tokens. This approach adapts to the context: when the model is very confident about the next word, only a few tokens might be needed to reach P, producing focused output. When the model is uncertain, more tokens are included, allowing for creative diversity. Top-P is often preferred over top-K sampling because it naturally adapts to the shape of the probability distribution rather than using a fixed number of candidates. It is one of the most commonly exposed parameters in AI APIs alongside temperature, and the two are often used together to fine-tune the balance between creativity and coherence in generated text.