LLM Benchmarks papers, 2023–2026

Key findings

• LLM Benchmarks went from 2 papers in 2023 to 307 in 2026, roughly 154x in three years.
• NeurIPS publishes the most LLM Benchmarks work: 256 papers from 2023 to 2026, 47% of the four-venue total.
• The fastest-growing overlap is with AI Agents: papers tagged with both rose from 1 in 2023 to 116 in 2026.
• Among the 5 topics under Evaluation and Benchmarks, it ranks #1 by 2026 paper count; the fastest grower in the group, LLM-as-Judge, is up 27x versus 154x here.

Representative papers

Spread across years and venues so every edition shows up

• NeurIPS 2026A Free Lunch in LLM Compression: Revisiting Retraining after Pruning
• ICML 2026Controlled LLM Training on Spectral Sphere
• ICLR 2026Hey, That's My Model! Introducing Chain & Hash, An LLM Fingerprinting Technique
• CVPR 20263DrawAgent Teaching LLM to Draw in 3D with Early Contrastive
• NeurIPS 2025ACCO: Accumulate While You Communicate for Communication-Overlapped Sharded LLM Training
• ICML 2025Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM’s Reasoning Capability
• ICLR 2025Catastrophic Failure of LLM Unlearning via Quantization
• CVPR 2025Period-LLM: Extending the Periodic Capability of Multimodal Large Language Model
• NeurIPS 2024Large Language Models' Expert-level Global History Knowledge Benchmark (HiST-LLM)
• ICML 2024Characterizing ResNet's Universal Approximation Capability
• ICLR 2024Faithful Explanations of Black-box NLP Models Using LLM-generated Counterfactuals
• CVPR 2024Koala: Key Frame-Conditioned Long Video-LLM
• NeurIPS 2023Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing
• NeurIPS 2026Accelerating LLM Pre-Training through Flat-Direction Dynamics Enhancement
• ICML 2026LLM Priors for ERM over Programs
• ICLR 2026How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretraining
• CVPR 20263D-VCD Hallucination Mitigation in 3D-LLM Embodied Agents through Visual Contrastive
• NeurIPS 2025BrainEC-LLM: Brain Effective Connectivity Estimation by Multiscale Mixing LLM
• ICML 2025Position: Medical Large Language Model Benchmarks Should Prioritize Construct Validity
• ICLR 2025Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Rates

All LLM Benchmarks papers

60 papers, page 1 of 2

1. ICML 2026$V_0$: A Generalist Value Model for Any Policy at State Zero
2. ICML 2026AdaGC: Enhancing LLM Pretraining Stability via Adaptive Gradient Clipping
3. ICML 2026Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarks
4. ICML 2026Advancing SVD-based LLM Compression via Layer-Wise Error Model Search
5. ICML 2026Beyond Heuristic Tuning: Power-Calibrated LLM Watermarking
6. ICML 2026Beyond Majority Voting: LLM Aggregation by Leveraging Higher-Order Information
7. ICML 2026CapBencher: Give Your LLM Benchmark a Built-in Alarm for Test-Set Overfitting
8. ICML 2026CCLRec: Consensus-driven Contrastive Learning for LLM-enhanced Graph Recommendation
9. ICML 2026Controlled LLM Training on Spectral Sphere
10. ICML 2026CorrSteer: Generation-Time LLM Steering via Correlated Sparse Autoencoder Features
11. ICML 2026CrispEdit: Low-Curvature Projections for Scalable Non-Destructive LLM Editing
12. ICML 2026CriticalKV: Optimizing KV Cache Eviction from an Output Perturbation Perspective
13. ICML 2026Cutting LLM Evaluation Costs with SySRs: A Bandit Algorithm That Provably Exploits Model Similarity
14. ICML 2026Dissecting Quantization Error: A Concentration-Alignment Perspective
15. ICML 2026Distribution-Calibrated Inference Time Compute for Thinking LLM-as-a-Judge
16. ICML 2026DynaSchedBench: Calibrated Dynamic Scheduling Benchmarks and Observability Paradox in LLM-based Scheduling Agents
17. ICML 2026ECHO: Elastic Speculative Decoding with Sparse Gating for High-Concurrency Scenarios
18. ICML 2026EffGen: Enabling Small Language Models as Capable Autonomous Agents
19. ICML 2026Enhancing LLM Training via Spectral Clipping
20. ICML 2026EvoClaw: Evaluating AI Agents on Continuous Software Evolution
21. ICML 2026From Backward Spreading to Forward Replay: Revisiting Target Construction in LLM Parameter Editing
22. ICML 2026GEM: Geometric Entropy Mixing for Optimal LLM Data Curation
23. ICML 2026HexGen-3: A Fully Disaggregated LLM Serving Framework with Fine-Grained Heterogeneous Resource Autoscaling
24. ICML 2026Improving Visual Token Reduction via Rectifying Distortions for Efficient Multimodal LLM Inference
25. ICML 2026Incremental BPE Tokenization
26. ICML 2026LATMiX: Learnable Affine Transformations for Microscaling Quantization of LLMs
27. ICML 2026Learn-to-learn on Arbitrary Textual Conditioning: A Hypernetwork-Driven Meta-gated LLM
28. ICML 2026LLM Priors for ERM over Programs
29. ICML 2026LLM Watermark Evasion via Bias Inversion
30. ICML 2026LLM-based Embeddings: Attention Values Encode Sentence Semantics Better Than Hidden States
31. ICML 2026Minibatch selection for Language Models via Partition Matroid Constrained Gradient Matching
32. ICML 2026Mirror Descent Under Generalized Smoothness
33. ICML 2026OServe: Accelerating LLM Serving via Spatial-Temporal Workload Orchestration
34. ICML 2026OSNIP: Balancing the Privacy-Utility-Efficiency Trilemma in LLM Inference via Obfuscated Semantic Null Space
35. ICML 2026Outrunning LLM Cutoffs: A Live Kernel Crash Resolution Benchmark for All
36. ICML 2026PatternKV: Flattening KV Representation Expands Quantization Headroom
37. ICML 2026PostTrainBench: Can LLM Agents Automate LLM Post-Training?
38. ICML 2026Quantifying Biases in LLM-as-a-Judge Evaluations
39. ICML 2026Quartet II: Accurate LLM Pre-Training in NVFP4 by Improved Unbiased Gradient Estimation
40. ICML 2026QUATRO: Query-Adaptive Trust Region Policy Optimization for LLM Fine-tuning
41. ICML 2026ReSpinQuant: Efficient Layer-Wise LLM Quantization via Subspace Residual Rotation Approximation
42. ICML 2026Scheduling LLM Inference with Uncertainty-Aware Output Length Predictions
43. ICML 2026SetPO: Set-Level Policy Optimization for Diversity-Preserving LLM Reasoning
44. ICML 2026Single-Rollout Hidden-State Dynamics for Training-Free RLVR Data Selection
45. ICML 2026Skill-Pro: Learning Reusable Skills from Experience via Non-Parametric PPO for LLM Agents
46. ICML 2026SPR-RAFT: Parameter-Efficient Regression-Aware Fine-Tuning for Biomedical LLM Regression
47. ICML 2026Spurious Rewards: Rethinking Training Signals in RLVR
48. ICML 2026The Invisible Lottery: How Subtle Cues Steer Algorithm Choice in LLM Code Generation
49. ICML 2026Think Twice Before You Act: Protecting LLM Agents Against Tool Description Poisoning via Isolated Planning
50. ICML 2026Tiny Brains, Giant Impact: Uncovering the Keystone Neurons of LLM with Just a Few Prompts

FAQ

How many LLM Benchmarks papers were published at NeurIPS, ICLR, ICML and CVPR from 2023 to 2026?

544 in total: 2 in 2023, 61 in 2024, 174 in 2025, 307 in 2026.

How many LLM Benchmarks papers are at NeurIPS 2026?

NeurIPS 2026 has 138 LLM Benchmarks papers, versus 92 the year before.

How many LLM Benchmarks papers are at ICLR 2026?

ICLR 2026 has 103 LLM Benchmarks papers, versus 50 the year before.

How many LLM Benchmarks papers are at ICML 2026?

ICML 2026 has 60 LLM Benchmarks papers, versus 24 the year before.

How many LLM Benchmarks papers are at CVPR 2026?

CVPR 2026 has 6 LLM Benchmarks papers, versus 8 the year before.

Which conference publishes the most LLM Benchmarks papers?

NeurIPS, with 256 papers from 2023 to 2026.