LLM Benchmarks papers, 2023–2026

Key findings

• LLM Benchmarks went from 2 papers in 2023 to 307 in 2026, roughly 154x in three years.
• NeurIPS publishes the most LLM Benchmarks work: 256 papers from 2023 to 2026, 47% of the four-venue total.
• The fastest-growing overlap is with AI Agents: papers tagged with both rose from 1 in 2023 to 116 in 2026.
• Among the 5 topics under Evaluation and Benchmarks, it ranks #1 by 2026 paper count; the fastest grower in the group, LLM-as-Judge, is up 27x versus 154x here.

Representative papers

Spread across years and venues so every edition shows up

• NeurIPS 2026A Free Lunch in LLM Compression: Revisiting Retraining after Pruning
• ICML 2026Controlled LLM Training on Spectral Sphere
• ICLR 2026Hey, That's My Model! Introducing Chain & Hash, An LLM Fingerprinting Technique
• CVPR 20263DrawAgent Teaching LLM to Draw in 3D with Early Contrastive
• NeurIPS 2025ACCO: Accumulate While You Communicate for Communication-Overlapped Sharded LLM Training
• ICML 2025Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM’s Reasoning Capability
• ICLR 2025Catastrophic Failure of LLM Unlearning via Quantization
• CVPR 2025Period-LLM: Extending the Periodic Capability of Multimodal Large Language Model
• NeurIPS 2024Large Language Models' Expert-level Global History Knowledge Benchmark (HiST-LLM)
• ICML 2024Characterizing ResNet's Universal Approximation Capability
• ICLR 2024Faithful Explanations of Black-box NLP Models Using LLM-generated Counterfactuals
• CVPR 2024Koala: Key Frame-Conditioned Long Video-LLM
• NeurIPS 2023Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing
• NeurIPS 2026Accelerating LLM Pre-Training through Flat-Direction Dynamics Enhancement
• ICML 2026LLM Priors for ERM over Programs
• ICLR 2026How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretraining
• CVPR 20263D-VCD Hallucination Mitigation in 3D-LLM Embodied Agents through Visual Contrastive
• NeurIPS 2025BrainEC-LLM: Brain Effective Connectivity Estimation by Multiscale Mixing LLM
• ICML 2025Position: Medical Large Language Model Benchmarks Should Prioritize Construct Validity
• ICLR 2025Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Rates

All LLM Benchmarks papers

256 papers, page 3 of 6

101. NeurIPS 2026SchemeArena: Factorized Stress Testing of Scheming in LLM Agents
102. NeurIPS 2026SCTI: Self-Calibrated Trident Identification of Black-Box LLM Watermarks
103. NeurIPS 2026Sequential Behavioral Watermarking for LLM Agents
104. NeurIPS 2026SFT-then-RL Outperforms Mixed-Policy Methods for LLM Reasoning
105. NeurIPS 2026SGEvolve: Semantic Gradient Guidance for LLM-Driven Evolutionary Search
106. NeurIPS 2026SkillMaster: Toward Autonomous Skill Mastery in LLM Agents
107. NeurIPS 2026Soteria: Formally Verified Planning with Runtime Enforcement for Safe LLM Agents
108. NeurIPS 2026Spatial-IQ: Deconstructing Spatial Intelligence via Hierarchical Capability Tests
109. NeurIPS 2026Specialists Hold, Generalists Discount: Asymmetric Equilibrium in LLM Routing Auctions
110. NeurIPS 2026Strategic Evaluation: Incentivizing AI Capability Coverage with Private Benchmarks
111. NeurIPS 2026STRIDE: Learnable Stepwise Language Feedback for LLM Reasoning
112. NeurIPS 2026Strong Teacher Not Needed? On Distillation in LLM Pretraining
113. NeurIPS 2026SudoBench: A Contextual Authorization Benchmark for LLM Agents
114. NeurIPS 2026SVDP: Training-Free Contextual Sparsity Predictors for Fast LLM Inference
115. NeurIPS 2026TextSeal: A Localized LLM Watermark for Provenance & Distillation Protection
116. NeurIPS 2026The Geometry of Forgetting: Temporal Knowledge Drift as an Independent Axis in LLM Representations
117. NeurIPS 2026The Illusion of Diversity: Aligning LLM Exploration via Effective Entropy
118. NeurIPS 2026The Labeling Problem in Hallucination Detection Benchmarks: An Empirical Evaluation
119. NeurIPS 2026The Scaling Paradox of Tool-Calling LLM Agents Under Realistic MCP Faults
120. NeurIPS 2026The Text Uncanny Valley: Non-Monotonic Performance Degradation in LLM Information Retrieval
121. NeurIPS 2026Tokens-per-Parameter Coverage Is Critical for Robust LLM Scaling Law Extrapolation
122. NeurIPS 2026Trust, but Don’t Verify: Epistemic Blind Spots in LLM Source Evaluation
123. NeurIPS 2026TSFM Meets LLM: Context as Covariate
124. NeurIPS 2026UMAS: System-Level Uncertainty Quantification for Multi-Agent LLM Systems
125. NeurIPS 2026Valid Best-Model Identification for LLM Evaluation via Low-Rank Factorization
126. NeurIPS 2026Values as Style: Disentangling Values from Semantics with One-Way Mixing for Low-Damage LLM Steering
127. NeurIPS 2026VEHBench: A Stage-Local Diagnostic Benchmark for LLM-Assisted Vibration Energy Harvester Design
128. NeurIPS 2026Verifiable LLM-Guided Focal SMT Solving for Quantified Arrays
129. NeurIPS 2026Verified-Source Authority Is Not Generic Sycophancy: Cue-Family Decomposition of LLM Compliance
130. NeurIPS 2026VeriVul: A Verification-Guided Framework for Generating Realistic Vulnerability Benchmarks
131. NeurIPS 2026ViroGym: Realistic Large-Scale Benchmarks for Evaluating Viral Proteins
132. NeurIPS 2026We Need to Improve Benchmarks in AI for Mathematics
133. NeurIPS 2026What Claims Do LLM Benchmark Scores Support?
134. NeurIPS 2026What Should Remain After Forgetting? Rethinking LLM Unlearning as Predictive Posterior Correction
135. NeurIPS 2026When Does Online Imitation Learning Help in LLM Post-Training? The Role of (Non-)Realizability Beyond Horizon
136. NeurIPS 2026When Expert Disagreement Hurts: Auditing Prestige-Sensitive Revision in LLM Decision Pipelines
137. NeurIPS 2026When LLM Routers Overpay: Strong-Model Over-Selection under Loose Budgets
138. NeurIPS 2026XBRIDGE: Entity-Grounded Latent Bridge for Heterogeneous LLM Communication
139. NeurIPS 2025$Q\sharp$: Provably Optimal Distributional RL for LLM Post-Training
140. NeurIPS 2025Accelerating Block Coordinate Descent for LLM Finetuning via Landscape Expansion
141. NeurIPS 2025ACCO: Accumulate While You Communicate for Communication-Overlapped Sharded LLM Training
142. NeurIPS 2025Accurate KV Cache Eviction via Anchor Direction Projection for Efficient LLM Inference
143. NeurIPS 2025Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts
144. NeurIPS 2025AgentRecBench: Benchmarking LLM Agent-based Personalized Recommender Systems
145. NeurIPS 2025Antidistillation Sampling
146. NeurIPS 2025AutoJudge: Judge Decoding Without Manual Annotation
147. NeurIPS 2025BenchmarkCards: Standardized Documentation for Large Language Model Benchmarks
148. NeurIPS 2025Beyond Chemical QA: Evaluating LLM's Chemical Reasoning with Modular Chemical Operations
149. NeurIPS 2025BLEUBERI: BLEU is a surprisingly effective reward for instruction following
150. NeurIPS 2025Bohdi: Heterogeneous LLM Fusion with Automatic Data Exploration

FAQ

How many LLM Benchmarks papers were published at NeurIPS, ICLR, ICML and CVPR from 2023 to 2026?

544 in total: 2 in 2023, 61 in 2024, 174 in 2025, 307 in 2026.

How many LLM Benchmarks papers are at NeurIPS 2026?

NeurIPS 2026 has 138 LLM Benchmarks papers, versus 92 the year before.

How many LLM Benchmarks papers are at ICLR 2026?

ICLR 2026 has 103 LLM Benchmarks papers, versus 50 the year before.

How many LLM Benchmarks papers are at ICML 2026?

ICML 2026 has 60 LLM Benchmarks papers, versus 24 the year before.

How many LLM Benchmarks papers are at CVPR 2026?

CVPR 2026 has 6 LLM Benchmarks papers, versus 8 the year before.

Which conference publishes the most LLM Benchmarks papers?

NeurIPS, with 256 papers from 2023 to 2026.