LLM Benchmarks papers, 2023–2026
Capability benchmarks
Scope: NeurIPS, ICLR, ICML and CVPR, 2023–2026. 544 papers are tagged with this topic.
Key findings
- LLM Benchmarks went from 2 papers in 2023 to 307 in 2026, roughly 154x in three years.
- NeurIPS publishes the most LLM Benchmarks work: 256 papers from 2023 to 2026, 47% of the four-venue total.
- The fastest-growing overlap is with AI Agents: papers tagged with both rose from 1 in 2023 to 116 in 2026.
- Among the 5 topics under Evaluation and Benchmarks, it ranks #1 by 2026 paper count; the fastest grower in the group, LLM-as-Judge, is up 27x versus 154x here.
Papers per venue
Counts by year and venue
| Year | NeurIPS | ICLR | ICML | CVPR | Total |
|---|---|---|---|---|---|
| 2023 | 2 | 0 | 0 | 0 | 2 |
| 2024 | 24 | 13 | 21 | 3 | 61 |
| 2025 | 92 | 50 | 24 | 8 | 174 |
| 2026 | 138 | 103 | 60 | 6 | 307 |
Representative papers
Spread across years and venues so every edition shows up
- NeurIPS 2026A Free Lunch in LLM Compression: Revisiting Retraining after Pruning
- ICML 2026Controlled LLM Training on Spectral Sphere
- ICLR 2026Hey, That's My Model! Introducing Chain & Hash, An LLM Fingerprinting Technique
- CVPR 20263DrawAgent Teaching LLM to Draw in 3D with Early Contrastive
- NeurIPS 2025ACCO: Accumulate While You Communicate for Communication-Overlapped Sharded LLM Training
- ICML 2025Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM’s Reasoning Capability
- ICLR 2025Catastrophic Failure of LLM Unlearning via Quantization
- CVPR 2025Period-LLM: Extending the Periodic Capability of Multimodal Large Language Model
- NeurIPS 2024Large Language Models' Expert-level Global History Knowledge Benchmark (HiST-LLM)
- ICML 2024Characterizing ResNet's Universal Approximation Capability
- ICLR 2024Faithful Explanations of Black-box NLP Models Using LLM-generated Counterfactuals
- CVPR 2024Koala: Key Frame-Conditioned Long Video-LLM
- NeurIPS 2023Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing
- NeurIPS 2026Accelerating LLM Pre-Training through Flat-Direction Dynamics Enhancement
- ICML 2026LLM Priors for ERM over Programs
- ICLR 2026How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretraining
- CVPR 20263D-VCD Hallucination Mitigation in 3D-LLM Embodied Agents through Visual Contrastive
- NeurIPS 2025BrainEC-LLM: Brain Effective Connectivity Estimation by Multiscale Mixing LLM
- ICML 2025Position: Medical Large Language Model Benchmarks Should Prioritize Construct Validity
- ICLR 2025Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Rates
All LLM Benchmarks papers
3 papers, page 1 of 1
Ask about this trend from Claude or Cursor
Connect the Paperlist MCP and your assistant can pull the numbers on this page while you draft a survey, pick a direction or plan a submission: papers per year and venue, growth, the fastest-rising sub-themes and representative papers, shaped around your question.
Copy this prompt into your assistant
Use paperlist to show how LLM Benchmarks papers grew at NeurIPS from 2023 to 2026, and which year jumped the most.
Tool the assistant calls: get_topic_stats {"topic": "llm-benchmarks"}
Set up Paperlist MCP in two minutes (Claude Code, Cursor, Codex, ChatGPT)
FAQ
- How many LLM Benchmarks papers were published at NeurIPS, ICLR, ICML and CVPR from 2023 to 2026?
- 544 in total: 2 in 2023, 61 in 2024, 174 in 2025, 307 in 2026.
- How many LLM Benchmarks papers are at NeurIPS 2026?
- NeurIPS 2026 has 138 LLM Benchmarks papers, versus 92 the year before.
- How many LLM Benchmarks papers are at ICLR 2026?
- ICLR 2026 has 103 LLM Benchmarks papers, versus 50 the year before.
- How many LLM Benchmarks papers are at ICML 2026?
- ICML 2026 has 60 LLM Benchmarks papers, versus 24 the year before.
- How many LLM Benchmarks papers are at CVPR 2026?
- CVPR 2026 has 6 LLM Benchmarks papers, versus 8 the year before.
- Which conference publishes the most LLM Benchmarks papers?
- NeurIPS, with 256 papers from 2023 to 2026.