NL2Repo-Bench leaderboard
As of November 8, 2026, DeepSeek V4.1 Flash (DeepSeek) leads NL2Repo-Bench at 68.2%, ahead of DeepSeek V4 Pro at 61.5% and MiniMax M3.1 Flash at 61.2%. 8 systems ranked. Every score here is published by the model's own lab; each row links to the page it comes from.
What is NL2Repo-Bench? NL2Repo-Bench tests whether a coding agent can build a complete, installable Python library from scratch, given nothing but a natural-language requirements document. Made by M-A-P and ByteDance Seed with university partners, December 2025.
Top models on NL2Repo-Bench, November 2026
| # | Model | Score | Price / 1M | Context | Source |
|---|---|---|---|---|---|
| 1 | DeepSeek V4.1 Flash DeepSeek · listed as “DeepSeek V4.1 Flash (1017)” | as reported | $0.3 / $1.2 | 1M | Vendor-reported DeepSeek |
| 2 | DeepSeek V4 Pro DeepSeek | score | $1.32 / $3.96 | 1M | Vendor-reported DeepSeek |
| 3 | MiniMax M3.1 Flash MiniMax | as reported | $0.15 / $0.6 | 1M | Vendor-reported MiniMax |
| 4 | GLM-5.3 Zhipu (Z.ai / GLM) | as reported | $1.4 / $4.4 | 1M | Vendor-reported Z.ai (Zhipu) |
| 5 | Qwen3.8 Max Alibaba (Qwen) | as reported | $2 / $6 | 1M | Vendor-reported Alibaba (Qwen) |
| 6 | DeepSeek V4 Flash Deprecated DeepSeek | score | $0.44 / $1.32 | 1M | Vendor-reported DeepSeek |
| 7 | GLM-5.2 Zhipu (Z.ai / GLM) | as reported | $1.4 / $4.4 | 1M | Vendor-reported Z.ai (Zhipu) |
| 8 | Qwen3.7 Max Alibaba (Qwen) | as reported | $2.5 / $7.5 | 1M | Vendor-reported Alibaba (Qwen) |
NL2Repo-Bench: score against price
Can you trust NL2Repo-Bench scores?
No independent leaderboard publishes NL2Repo-Bench results for these models yet, so every number on this page is the lab's own report of its own model. Labs choose the harness, the effort level and the subset they report, and they rarely publish failures. Read the setting under each score before comparing two rows.
- The agent framework changes the score: the paper's main table uses OpenHands, and the same model scores differently in Claude Code or Cursor.
- Python libraries only, graded by the upstream tests, so it measures rebuilding existing open-source projects.
- There is no official leaderboard; labs report their own runs.
How NL2Repo-Bench works
The task
The agent receives a long requirements document and an empty workspace and has to design and write the whole library.
The score
The generated repository runs against the original project's own test suite; the main score is the average share of tests passed.
Test set
104 tasks, as published by the authors.
Made by
M-A-P and ByteDance Seed with university partners, first published December 2025.
Official sources
- github.com/multimodal-art-projection/NL2RepoBench
- Paper: arxiv.org/abs/2512.12730
- Definition read on 2026-11-08
Sources and freshness
9 published scores from 5 sources. Each was read on its source page; the oldest reading on this page is from November 8, 2026. Prices beside each score come from our model price index and move when a vendor reprices.
- Vendor-reported DeepSeek
- Vendor-reported Alibaba (Qwen)
- Vendor-reported Z.ai (Zhipu)
- Vendor-reported DeepSeek
- Vendor-reported MiniMax
NL2Repo-Bench FAQ
Answers derived from the 9 published scores on this page and the current price index.