Language Models are Few-Shot Learners

Abstract

We demonstrate that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art fine-tuning approaches. We train GPT-3, an autoregressive language model with 175 billion parameters, and test its performance in the few-shot setting. GPT-3 achieves strong performance on many NLP datasets without any gradient updates or fine-tuning.

Key Findings

• 1Demonstrated that 175B parameter models exhibit strong few-shot learning abilities
• 2Showed that in-context learning emerges at sufficient scale
• 3Achieved competitive results without any gradient updates or fine-tuning
• 4Revealed scaling laws: bigger models show qualitatively different capabilities
• 5Introduced the concept of prompting as a new paradigm for using LLMs

Impact & Significance