LLM Benchmarks papers, 2023–2026

Key findings

• LLM Benchmarks went from 2 papers in 2023 to 307 in 2026, roughly 154x in three years.
• NeurIPS publishes the most LLM Benchmarks work: 256 papers from 2023 to 2026, 47% of the four-venue total.
• The fastest-growing overlap is with AI Agents: papers tagged with both rose from 1 in 2023 to 116 in 2026.
• Among the 5 topics under Evaluation and Benchmarks, it ranks #1 by 2026 paper count; the fastest grower in the group, LLM-as-Judge, is up 27x versus 154x here.

Representative papers

Spread across years and venues so every edition shows up

• NeurIPS 2026A Free Lunch in LLM Compression: Revisiting Retraining after Pruning
• ICML 2026Controlled LLM Training on Spectral Sphere
• ICLR 2026Hey, That's My Model! Introducing Chain & Hash, An LLM Fingerprinting Technique
• CVPR 20263DrawAgent Teaching LLM to Draw in 3D with Early Contrastive
• NeurIPS 2025ACCO: Accumulate While You Communicate for Communication-Overlapped Sharded LLM Training
• ICML 2025Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM’s Reasoning Capability
• ICLR 2025Catastrophic Failure of LLM Unlearning via Quantization
• CVPR 2025Period-LLM: Extending the Periodic Capability of Multimodal Large Language Model
• NeurIPS 2024Large Language Models' Expert-level Global History Knowledge Benchmark (HiST-LLM)
• ICML 2024Characterizing ResNet's Universal Approximation Capability
• ICLR 2024Faithful Explanations of Black-box NLP Models Using LLM-generated Counterfactuals
• CVPR 2024Koala: Key Frame-Conditioned Long Video-LLM
• NeurIPS 2023Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing
• NeurIPS 2026Accelerating LLM Pre-Training through Flat-Direction Dynamics Enhancement
• ICML 2026LLM Priors for ERM over Programs
• ICLR 2026How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretraining
• CVPR 20263D-VCD Hallucination Mitigation in 3D-LLM Embodied Agents through Visual Contrastive
• NeurIPS 2025BrainEC-LLM: Brain Effective Connectivity Estimation by Multiscale Mixing LLM
• ICML 2025Position: Medical Large Language Model Benchmarks Should Prioritize Construct Validity
• ICLR 2025Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Rates

All LLM Benchmarks papers

166 papers, page 1 of 4

1. ICLR 2026<SO$G_k$>: One LLM Token for Explicit Graph Structural Understanding
2. ICLR 2026Activation Steering for LLM Alignment via a Unified ODE-Based Framework
3. ICLR 2026Adaptive Thinking: Large Language Models Know When to Think in Latent Space
4. ICLR 2026ASCIIEval: Benchmarking Models' Visual Perception in Text Strings via ASCII Art
5. ICLR 2026AutoSP: Unlocking Long-Context LLM Training Via Compiler-Based Sequence Parallelism
6. ICLR 2026Beyond Length: Quantifying Long-Range Information for Long-Context LLM Pretraining Data
7. ICLR 2026Calibrating Verbalized Confidence with Self-Generated Distractors
8. ICLR 2026Cancer-Myth: Evaluating Large Language Models on Patient Questions with False Presuppositions
9. ICLR 2026Capability-Based Scaling Trends for LLM-Based Red-Teaming
10. ICLR 2026CMPhysBench: A Benchmark for Evaluating Large Language Models in Condensed Matter Physics
11. ICLR 2026Common Corpus: The Largest Collection of Ethical Data for LLM Pre-Training
12. ICLR 2026Count Counts: Motivating Exploration in LLM Reasoning with Count-based Intrinsic Rewards
13. ICLR 2026Credit-Budgeted ICPC-Style Coding: When LLM Agents Must Pay for Every Decision
14. ICLR 2026DASH: Deterministic Attention Scheduling for High-throughput Reproducible LLM Training
15. ICLR 2026Data Selection for LLM Alignment Using Fine-Grained Preferences
16. ICLR 2026DeepResearch Bench: A Comprehensive Benchmark for Deep Research Agents
17. ICLR 2026Detecting Data Contamination in LLMs via In-Context Learning
18. ICLR 2026Discern Truth from Falsehood: Reducing Over-Refusal via Contrastive Refinement
19. ICLR 2026Discovering Novel LLM Experts via Task-Capability Coevolution
20. ICLR 2026Downgrade to Upgrade: Optimizer Simplification Enhances Robustness in LLM Unlearning
21. ICLR 2026Dr.LLM: Dynamic Layer Routing in LLMs
22. ICLR 2026EditBench: Evaluating LLM Abilities to Perform Real-World Instructed Code Edits
23. ICLR 2026Empowering LLM Tool Invocation with Tool-call Reward Model
24. ICLR 2026Enhancing Instruction Following of LLMs via Activation Steering with Dynamic Rejection
25. ICLR 2026Epistemic Uncertainty Quantification To Improve Decisions From Black-Box Models
26. ICLR 2026Expanding the Capability Frontier of LLM Agents with ZPD-Guided Data Synthesis
27. ICLR 2026ExpGuard: LLM Content Moderation in Specialized Domains
28. ICLR 2026Fast-dLLM v2: Efficient Block-Diffusion LLM
29. ICLR 2026From Static Benchmarks to Dynamic Protocol: Agent-Centric Text Anomaly Detection for Evaluating LLM Reasoning
30. ICLR 2026FutureX: An Advanced Live Benchmark for LLM Agents in Future Prediction
31. ICLR 2026Go-Browse: Training Web Agents with Structured Exploration
32. ICLR 2026Hey, That's My Model! Introducing Chain & Hash, An LLM Fingerprinting Technique
33. ICLR 2026How Do Transformers Learn to Associate Tokens: Gradient Leading Terms Bring Mechanistic Interpretability
34. ICLR 2026How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretraining
35. ICLR 2026How Stable is the Next Token? A Geometric View of LLM Prediction Stability
36. ICLR 2026Improving LLM-based Global Optimization with Search Space Partitioning
37. ICLR 2026In Agents We Trust, but Who Do Agents Trust? Latent Preferences Steer LLM Generations
38. ICLR 2026Jet Expansions: Restructuring LLM Computation for Model Inspection
39. ICLR 2026KL-Regularized Reinforcement Learning is Designed to Mode Collapse
40. ICLR 2026KV-Cache Transform Coding for Compact Storage in LLM Inference
41. ICLR 2026LLM as an Algorithmist: Enhancing Anomaly Detectors via Programmatic Synthesis
42. ICLR 2026LLM Pretraining with Continuous Concepts
43. ICLR 2026LLM Unlearning with LLM Beliefs
44. ICLR 2026LLM-as-a-Prophet: Understanding Predictive Intelligence with Prophet Arena
45. ICLR 2026Log Probability Tracking of LLM APIs
46. ICLR 2026LS-Merge: Merging Language Models in Latent Space
47. ICLR 2026LycheeDecode: Accelerating Long-Context LLM Inference via Hybrid-Head Sparse Decoding
48. ICLR 2026Making Slow Thinking Faster: Compressing LLM Chain-of-Thought via Step Entropy
49. ICLR 2026MCP Security Bench (MSB): Benchmarking Attacks Against Model Context Protocol in LLM Agents
50. ICLR 2026MCP-Bench: Benchmarking Tool-Using LLM Agents with Complex Real-World Tasks via MCP Servers

FAQ

How many LLM Benchmarks papers were published at NeurIPS, ICLR, ICML and CVPR from 2023 to 2026?

544 in total: 2 in 2023, 61 in 2024, 174 in 2025, 307 in 2026.

How many LLM Benchmarks papers are at NeurIPS 2026?

NeurIPS 2026 has 138 LLM Benchmarks papers, versus 92 the year before.

How many LLM Benchmarks papers are at ICLR 2026?

ICLR 2026 has 103 LLM Benchmarks papers, versus 50 the year before.

How many LLM Benchmarks papers are at ICML 2026?

ICML 2026 has 60 LLM Benchmarks papers, versus 24 the year before.

How many LLM Benchmarks papers are at CVPR 2026?

CVPR 2026 has 6 LLM Benchmarks papers, versus 8 the year before.

Which conference publishes the most LLM Benchmarks papers?

NeurIPS, with 256 papers from 2023 to 2026.