MBPP (Mostly Basic Python Problems)
MBPP consists of around 1,000 crowd-sourced Python programming problems designed to be solvable by entry-level programmers. Each problem includes a task description, code solution, and three automated test cases.
Metrics
Pass@1 (%) on ~1,000 Python problems
Created By
Google Research
Paper
View paper →Website
Visit website →Top Model Scores
| Rank | Model | Score | Date |
|---|---|---|---|
| 1 | Claude Opus 4.6 | 93.8% | 2026-02 |
| 2 | GPT-5.2 | 93.2% | 2026-03 |
| 3 | Gemini 3 Ultra | 91.7% | 2026-01 |
| 4 | DeepSeek Coder V3 | 90.4% | 2026-01 |
| 5 | Grok 4 | 89.6% | 2026-02 |
Related Code Benchmarks
HumanEval
HumanEval evaluates the functional correctness of code generated by language models. It consists of 164 hand-written programming problems with function signatures, docstrings, and unit tests, measuring pass@1 and pass@k rates.
Top: Claude Opus 4.6 — 96.3%
SWE-bench Verified
SWE-bench Verified evaluates AI systems on real-world software engineering tasks drawn from GitHub issues in popular Python repositories. Models must understand codebases, diagnose issues, and generate correct patches.
Top: Claude Opus 4.6 + Agentless — 62.4%
CodeContests
CodeContests is a competitive programming benchmark drawn from Codeforces, CodeChef, and other platforms. It tests algorithmic problem-solving with problems requiring complex data structures, dynamic programming, and mathematical reasoning.
Top: GPT-5.2 — 43.2%
HumanEval+
HumanEval+ augments the original HumanEval benchmark with 80x more test cases per problem, providing a more rigorous evaluation of code correctness. Many models that score well on HumanEval see significant drops on HumanEval+.
Top: Claude Opus 4.6 — 90.2%