LLM Benchmarks papers, 2023–2026

Key findings

• LLM Benchmarks went from 2 papers in 2023 to 307 in 2026, roughly 154x in three years.
• NeurIPS publishes the most LLM Benchmarks work: 256 papers from 2023 to 2026, 47% of the four-venue total.
• The fastest-growing overlap is with AI Agents: papers tagged with both rose from 1 in 2023 to 116 in 2026.
• Among the 5 topics under Evaluation and Benchmarks, it ranks #1 by 2026 paper count; the fastest grower in the group, LLM-as-Judge, is up 27x versus 154x here.

Representative papers

Spread across years and venues so every edition shows up

• NeurIPS 2026A Free Lunch in LLM Compression: Revisiting Retraining after Pruning
• ICML 2026Controlled LLM Training on Spectral Sphere
• ICLR 2026Hey, That's My Model! Introducing Chain & Hash, An LLM Fingerprinting Technique
• CVPR 20263DrawAgent Teaching LLM to Draw in 3D with Early Contrastive
• NeurIPS 2025ACCO: Accumulate While You Communicate for Communication-Overlapped Sharded LLM Training
• ICML 2025Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM’s Reasoning Capability
• ICLR 2025Catastrophic Failure of LLM Unlearning via Quantization
• CVPR 2025Period-LLM: Extending the Periodic Capability of Multimodal Large Language Model
• NeurIPS 2024Large Language Models' Expert-level Global History Knowledge Benchmark (HiST-LLM)
• ICML 2024Characterizing ResNet's Universal Approximation Capability
• ICLR 2024Faithful Explanations of Black-box NLP Models Using LLM-generated Counterfactuals
• CVPR 2024Koala: Key Frame-Conditioned Long Video-LLM
• NeurIPS 2023Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing
• NeurIPS 2026Accelerating LLM Pre-Training through Flat-Direction Dynamics Enhancement
• ICML 2026LLM Priors for ERM over Programs
• ICLR 2026How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretraining
• CVPR 20263D-VCD Hallucination Mitigation in 3D-LLM Embodied Agents through Visual Contrastive
• NeurIPS 2025BrainEC-LLM: Brain Effective Connectivity Estimation by Multiscale Mixing LLM
• ICML 2025Position: Medical Large Language Model Benchmarks Should Prioritize Construct Validity
• ICLR 2025Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Rates

All LLM Benchmarks papers

256 papers, page 1 of 6

1. NeurIPS 2026A Free Lunch in LLM Compression: Revisiting Retraining after Pruning
2. NeurIPS 2026A Retained-Signal Interface for LLM Watermark Robustness under Paraphrase
3. NeurIPS 2026Accelerating LLM Pre-Training through Flat-Direction Dynamics Enhancement
4. NeurIPS 2026Accelerating Long-Context LLM Prefill via Layer-wise Progressive Token Pruning in Local Deployment
5. NeurIPS 2026AccelEval: A Whole-Program Benchmark for LLM-Generated CPU-to-GPU Code Acceleration
6. NeurIPS 2026Adaptive Adversaries: A Multi-Turn, Multi-LLM Benchmark for LLM Agent Security
7. NeurIPS 2026ADKV: A Low-Overhead Adaptive Delta Quantization for KV Cache in LLM Inference
8. NeurIPS 2026Agent-ToM: Learning to Monitor Autonomous LLM Agents via Theory-of-Mind Reasoning
9. NeurIPS 2026AgentAbstain: Do LLM Agents Know When Not to Act?
10. NeurIPS 2026Aligning Language Model Benchmarks with Pairwise Preferences
11. NeurIPS 2026Alignment Dynamics in LLM Fine-Tuning
12. NeurIPS 2026Are LLM Safety Judges Policy-Invariant? A Three-Principle Stress-Test
13. NeurIPS 2026Are Multimodal Benchmarks Really Useful? Item-Level Multimodal Benchmark Diagnosis via Structure-Response Co-Calibration
14. NeurIPS 2026Ask in the Crowd: Differentially Private LLM Inference via Dummy-Augmented Shuffling
15. NeurIPS 2026ASPI: Seeking Ambiguity Clarification Amplifies Prompt Injection Vulnerability in LLM Agents
16. NeurIPS 2026AutoDataBench: How Far Are LLM Agents from Autonomously Engineering Post-Training Data Pipelines?
17. NeurIPS 2026BELLS-O: Evaluating the Operational Trade-offs of LLM Supervision Systems
18. NeurIPS 2026Benchmarks Are Not Atomic: Composition-Aware LLM Evaluation using BenchHub
19. NeurIPS 2026Beyond Linear Activation Steering: Invertible Latent Transformations for Controlling LLM Behavior
20. NeurIPS 2026BLADE: Scalable Bi-level Adaptive Data Selection for LLM Training
21. NeurIPS 2026Block-OBS-GS: Exact Per-Block Joint Brain Surgery with Gauss–Seidel Refinement for LLM Pruning
22. NeurIPS 2026BMAttn: Block-Aligned Mixed-Precision Attention Quantization for LLM Inference
23. NeurIPS 2026Boosting LLM Reasoning via Human-Inspired Reward Shaping
24. NeurIPS 2026Brittlebench: Quantifying LLM robustness via prompt sensitivity
25. NeurIPS 2026Calibration Is Not Control: Intervention Advantage for LLM-Agent Oversight
26. NeurIPS 2026Can an LLM Reason Like a Lawyer? Benchmarking the ability of LLMs to map the facts of a case to the elements of the applicable legal rule
27. NeurIPS 2026CapTrack: Multifaceted Evaluation of Forgetting in LLM Post-Training
28. NeurIPS 2026Capturing LLM Capabilities via Evidence-Calibrated Query Clustering
29. NeurIPS 2026Causal Benchmarks for Multimodal AI Should Measure Categorical Difference, Not Capability Gap
30. NeurIPS 2026CLIOPATRA: Extracting Private Information from LLM Insights
31. NeurIPS 2026COMPASS: Composable Policy-Amortized Structured Search for LLM-Based Optimization Modeling
32. NeurIPS 2026Confidence Estimation via Decoupled Smoothing for Dynamic LLM Routing and Aggregation
33. NeurIPS 2026Consolidating Rewarded Perturbations for LLM Post-Training
34. NeurIPS 2026ContractBench: Can LLM Agents Preserve Observation Contracts?
35. NeurIPS 2026Contrastive Representation Shaping for LLM Unlearning
36. NeurIPS 2026Correlation-Aware Contextual Bandits with Surrogate Rewards for LLM Routing
37. NeurIPS 2026CurveRL: Principled Distribution-Aware Context Reweighting for LLM Reasoning
38. NeurIPS 2026DAPS: Dependency-Aware Premise Selection for LLM Theorem Proving
39. NeurIPS 2026Decentralized Aggregation of LLM Predictions via Wagering Mechanisms
40. NeurIPS 2026Demystifying Numerical Errors in LLM Inference: Achieving Reproducible Inference for Mission-Critical Tasks with HEAL
41. NeurIPS 2026DICE: Decoupling Capability from Intervention Necessity in LLM Tutoring
42. NeurIPS 2026Differential Item Functioning as an Item-Level Diagnostic for LLM Benchmarks
43. NeurIPS 2026DistDebug-Bench: Can LLM Agents Diagnose Bugs in Distributed Systems?
44. NeurIPS 2026Don't Always Pick the Highest-Performing Model: An Information Theoretic View of LLM Ensemble Selection
45. NeurIPS 2026Efficient Evaluation of LLM Performance with Statistical Guarantees
46. NeurIPS 2026Efficient Gradient-Aware Asynchronous Reinforcement Learning for LLM Post-Training
47. NeurIPS 2026ENGINE: Endogenous Variational MultiScale Optimization for Zeroth-Order LLM Fine-Tuning
48. NeurIPS 2026Evaluating Physical Reasoning in LLM Agents Requires Construction Benchmarks
49. NeurIPS 2026Every Bit, Everywhere, All At Once: A Binomial Multibit LLM Watermark
50. NeurIPS 2026Fast-dLLM++: Fr\'{e}chet Profile Decoding for Faster Diffusion LLM Inference

FAQ

How many LLM Benchmarks papers were published at NeurIPS, ICLR, ICML and CVPR from 2023 to 2026?

544 in total: 2 in 2023, 61 in 2024, 174 in 2025, 307 in 2026.

How many LLM Benchmarks papers are at NeurIPS 2026?

NeurIPS 2026 has 138 LLM Benchmarks papers, versus 92 the year before.

How many LLM Benchmarks papers are at ICLR 2026?

ICLR 2026 has 103 LLM Benchmarks papers, versus 50 the year before.

How many LLM Benchmarks papers are at ICML 2026?

ICML 2026 has 60 LLM Benchmarks papers, versus 24 the year before.

How many LLM Benchmarks papers are at CVPR 2026?

CVPR 2026 has 6 LLM Benchmarks papers, versus 8 the year before.

Which conference publishes the most LLM Benchmarks papers?

NeurIPS, with 256 papers from 2023 to 2026.