What Is Greedy Decoding?

Greedy decoding is the simplest approach to generating text from a language model. At each step, the model computes probabilities for all possible next tokens and selects the one with the highest probability (the argmax). This process repeats until the model produces a stop token or reaches a maximum length. Because there is no randomness involved, greedy decoding always produces the same output for the same input, making it fully deterministic. This is equivalent to setting temperature to 0 in most AI APIs. Greedy decoding is fast and computationally efficient since it only needs to track a single sequence. However, it has notable limitations: it can get stuck in repetitive loops, produce bland or generic text, and miss globally optimal sequences because the locally best token at each step does not always lead to the best overall output. Despite these drawbacks, greedy decoding is well-suited for tasks requiring consistency and precision, such as code generation, structured data extraction, and factual question answering where creativity is undesirable. For creative or conversational applications, sampling-based methods like top-P or temperature scaling are generally preferred.