What Is Autoregressive Model?
An autoregressive model is a type of generative model that produces output sequentially, one element at a time, where each newly generated element is conditioned on all previously generated elements, creating a chain of dependent predictions.
How Autoregressive Model Works
Autoregressive models are the backbone of most modern text-generating AI. When GPT produces a response, it generates one token at a time, feeding each newly generated token back into the model as input for the next prediction. This sequential process is why you see text appearing word by word in ChatGPT. The model calculates probability distributions over possible next tokens and samples from them (influenced by temperature). While this approach is computationally sequential at generation time, it produces highly coherent and contextually consistent outputs. Autoregressive generation is used not only in language models but also in image generation (pixel by pixel or patch by patch) and audio synthesis.
Real-World Examples
ChatGPT generating a response one word at a time, each word influenced by all the words generated before it
A code completion tool predicting the next line of code based on all preceding lines in the file
An autoregressive image model generating an image pixel by pixel from top-left to bottom-right