LLM Benchmarks papers, 2023–2026

Key findings

• LLM Benchmarks went from 2 papers in 2023 to 307 in 2026, roughly 154x in three years.
• NeurIPS publishes the most LLM Benchmarks work: 256 papers from 2023 to 2026, 47% of the four-venue total.
• The fastest-growing overlap is with AI Agents: papers tagged with both rose from 1 in 2023 to 116 in 2026.
• Among the 5 topics under Evaluation and Benchmarks, it ranks #1 by 2026 paper count; the fastest grower in the group, LLM-as-Judge, is up 27x versus 154x here.

Representative papers

Spread across years and venues so every edition shows up

• NeurIPS 2026A Free Lunch in LLM Compression: Revisiting Retraining after Pruning
• ICML 2026Controlled LLM Training on Spectral Sphere
• ICLR 2026Hey, That's My Model! Introducing Chain & Hash, An LLM Fingerprinting Technique
• CVPR 20263DrawAgent Teaching LLM to Draw in 3D with Early Contrastive
• NeurIPS 2025ACCO: Accumulate While You Communicate for Communication-Overlapped Sharded LLM Training
• ICML 2025Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM’s Reasoning Capability
• ICLR 2025Catastrophic Failure of LLM Unlearning via Quantization
• CVPR 2025Period-LLM: Extending the Periodic Capability of Multimodal Large Language Model
• NeurIPS 2024Large Language Models' Expert-level Global History Knowledge Benchmark (HiST-LLM)
• ICML 2024Characterizing ResNet's Universal Approximation Capability
• ICLR 2024Faithful Explanations of Black-box NLP Models Using LLM-generated Counterfactuals
• CVPR 2024Koala: Key Frame-Conditioned Long Video-LLM
• NeurIPS 2023Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing
• NeurIPS 2026Accelerating LLM Pre-Training through Flat-Direction Dynamics Enhancement
• ICML 2026LLM Priors for ERM over Programs
• ICLR 2026How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretraining
• CVPR 20263D-VCD Hallucination Mitigation in 3D-LLM Embodied Agents through Visual Contrastive
• NeurIPS 2025BrainEC-LLM: Brain Effective Connectivity Estimation by Multiscale Mixing LLM
• ICML 2025Position: Medical Large Language Model Benchmarks Should Prioritize Construct Validity
• ICLR 2025Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Rates

All LLM Benchmarks papers

50 papers, page 1 of 1

1. ICLR 2025AI Sandbagging: Language Models can Strategically Underperform on Evaluations
2. ICLR 2025An Empirical Analysis of Uncertainty in Large Language Model Evaluations
3. ICLR 2025Better than Your Teacher: LLM Agents that learn from Privileged AI Feedback
4. ICLR 2025BingoGuard: LLM Content Moderation Tools with Risk Levels
5. ICLR 2025Catastrophic Failure of LLM Unlearning via Quantization
6. ICLR 2025Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Rates
7. ICLR 2025Combatting Dimensional Collapse in LLM Pre-Training Data via Submodular File Selection
8. ICLR 2025Controlled LLM Decoding via Discrete Auto-regressive Biasing
9. ICLR 2025Cut the Crap: An Economical Communication Pipeline for LLM-based Multi-Agent Systems
10. ICLR 2025Determine-Then-Ensemble: Necessity of Top-k Union for Large Language Model Ensembling
11. ICLR 2025Diversity Empowers Intelligence: Integrating Expertise of Software Engineering Agents
12. ICLR 2025Do LLMs have Consistent Values?
13. ICLR 2025Do LLMs Recognize Your Preferences? Evaluating Personalized Preference Following in LLMs
14. ICLR 2025DON’T STOP ME NOW: EMBEDDING BASED SCHEDULING FOR LLMS
15. ICLR 2025ELICIT: LLM Augmentation Via External In-context Capability
16. ICLR 2025Faster Cascades via Speculative Decoding
17. ICLR 2025ForecastBench: A Dynamic Benchmark of AI Forecasting Capabilities
18. ICLR 2025Fourier Head: Helping Large Language Models Learn Complex Probability Distributions
19. ICLR 2025From Commands to Prompts: LLM-based Semantic File System for AIOS
20. ICLR 2025GameArena: Evaluating LLM Reasoning through Live Computer Games
21. ICLR 2025Herald: A Natural Language Annotated Lean 4 Dataset
22. ICLR 2025HShare: Fast LLM Decoding by Hierarchical Key-Value Sharing
23. ICLR 2025Improving Large Language Model Planning with Action Sequence Similarity
24. ICLR 2025Instruct-SkillMix: A Powerful Pipeline for LLM Instruction Tuning
25. ICLR 2025Instructional Segment Embedding: Improving LLM Safety with Instruction Hierarchy
26. ICLR 2025Language Models Are Implicitly Continuous
27. ICLR 2025Large Language Models can Become Strong Self-Detoxifiers
28. ICLR 2025Law of the Weakest Link: Cross Capabilities of Large Language Models
29. ICLR 2025Lawma: The Power of Specialization for Legal Annotation
30. ICLR 2025LiveBench: A Challenging, Contamination-Limited LLM Benchmark
31. ICLR 2025LLM Unlearning via Loss Adjustment with Only Forget Data
32. ICLR 2025Logical Consistency of Large Language Models in Fact-Checking
33. ICLR 2025MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning
34. ICLR 2025Omni-MATH: A Universal Olympiad Level Mathematic Benchmark for Large Language Models
35. ICLR 2025On Calibration of LLM-based Guard Models for Reliable Content Moderation
36. ICLR 2025Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Models
37. ICLR 2025PhyloLM: Inferring the Phylogeny of Large Language Models and Predicting their Performances in Benchmarks
38. ICLR 2025RevisEval: Improving LLM-as-a-Judge via Response-Adapted References
39. ICLR 2025Robotouille: An Asynchronous Planning Benchmark for LLM Agents
40. ICLR 2025SeedLM: Compressing LLM Weights into Seeds of Pseudo-Random Generators
41. ICLR 2025Selective Attention Improves Transformer
42. ICLR 2025SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal
43. ICLR 2025SpinQuant: LLM Quantization with Learned Rotations
44. ICLR 2025SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget
45. ICLR 2025StringLLM: Understanding the String Processing Capability of Large Language Models
46. ICLR 2025Style Outweighs Substance: Failure Modes of LLM Judges in Alignment Benchmarking
47. ICLR 2025TODO: Enhancing LLM Alignment with Ternary Preferences
48. ICLR 2025ToolACE: Winning the Points of LLM Function Calling
49. ICLR 2025Towards a Theoretical Understanding of Synthetic Data in LLM Post-Training: A Reverse-Bottleneck Perspective
50. ICLR 2025Trajectory-LLM: A Language-based Data Generator for Trajectory Prediction in Autonomous Driving

FAQ

How many LLM Benchmarks papers were published at NeurIPS, ICLR, ICML and CVPR from 2023 to 2026?

544 in total: 2 in 2023, 61 in 2024, 174 in 2025, 307 in 2026.

How many LLM Benchmarks papers are at NeurIPS 2026?

NeurIPS 2026 has 138 LLM Benchmarks papers, versus 92 the year before.

How many LLM Benchmarks papers are at ICLR 2026?

ICLR 2026 has 103 LLM Benchmarks papers, versus 50 the year before.

How many LLM Benchmarks papers are at ICML 2026?

ICML 2026 has 60 LLM Benchmarks papers, versus 24 the year before.

How many LLM Benchmarks papers are at CVPR 2026?

CVPR 2026 has 6 LLM Benchmarks papers, versus 8 the year before.

Which conference publishes the most LLM Benchmarks papers?

NeurIPS, with 256 papers from 2023 to 2026.