The Llama 3 Herd of Models

Abstract

We present Llama 3, a herd of language models that natively support multilinguality, coding, reasoning, and tool usage. Our largest model is a dense Transformer with 405B parameters and a context window of up to 128K tokens. We find that Llama 3 delivers comparable quality to leading language models such as GPT-4 across a range of tasks.

Key Findings

• 1Scaled to 405B parameters with a 128K context window
• 2Achieved performance comparable to GPT-4 across many benchmarks
• 3Native support for multilingual, coding, reasoning, and tool use
• 4Trained on over 15 trillion tokens of data
• 5Released as open source with extensive documentation

Impact & Significance