LLM Benchmarks papers, 2023–2026

Key findings

• LLM Benchmarks went from 2 papers in 2023 to 307 in 2026, roughly 154x in three years.
• NeurIPS publishes the most LLM Benchmarks work: 256 papers from 2023 to 2026, 47% of the four-venue total.
• The fastest-growing overlap is with AI Agents: papers tagged with both rose from 1 in 2023 to 116 in 2026.
• Among the 5 topics under Evaluation and Benchmarks, it ranks #1 by 2026 paper count; the fastest grower in the group, LLM-as-Judge, is up 27x versus 154x here.

Representative papers

Spread across years and venues so every edition shows up

• NeurIPS 2026A Free Lunch in LLM Compression: Revisiting Retraining after Pruning
• ICML 2026Controlled LLM Training on Spectral Sphere
• ICLR 2026Hey, That's My Model! Introducing Chain & Hash, An LLM Fingerprinting Technique
• CVPR 20263DrawAgent Teaching LLM to Draw in 3D with Early Contrastive
• NeurIPS 2025ACCO: Accumulate While You Communicate for Communication-Overlapped Sharded LLM Training
• ICML 2025Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM’s Reasoning Capability
• ICLR 2025Catastrophic Failure of LLM Unlearning via Quantization
• CVPR 2025Period-LLM: Extending the Periodic Capability of Multimodal Large Language Model
• NeurIPS 2024Large Language Models' Expert-level Global History Knowledge Benchmark (HiST-LLM)
• ICML 2024Characterizing ResNet's Universal Approximation Capability
• ICLR 2024Faithful Explanations of Black-box NLP Models Using LLM-generated Counterfactuals
• CVPR 2024Koala: Key Frame-Conditioned Long Video-LLM
• NeurIPS 2023Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing
• NeurIPS 2026Accelerating LLM Pre-Training through Flat-Direction Dynamics Enhancement
• ICML 2026LLM Priors for ERM over Programs
• ICLR 2026How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretraining
• CVPR 20263D-VCD Hallucination Mitigation in 3D-LLM Embodied Agents through Visual Contrastive
• NeurIPS 2025BrainEC-LLM: Brain Effective Connectivity Estimation by Multiscale Mixing LLM
• ICML 2025Position: Medical Large Language Model Benchmarks Should Prioritize Construct Validity
• ICLR 2025Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Rates

All LLM Benchmarks papers

544 papers, page 2 of 11

51. ICLR 2026Log Probability Tracking of LLM APIs
52. ICLR 2026LS-Merge: Merging Language Models in Latent Space
53. ICLR 2026LycheeDecode: Accelerating Long-Context LLM Inference via Hybrid-Head Sparse Decoding
54. ICLR 2026Making Slow Thinking Faster: Compressing LLM Chain-of-Thought via Step Entropy
55. ICLR 2026MCP Security Bench (MSB): Benchmarking Attacks Against Model Context Protocol in LLM Agents
56. ICLR 2026MCP-Bench: Benchmarking Tool-Using LLM Agents with Complex Real-World Tasks via MCP Servers
57. ICLR 2026Metis: Training LLMs with FP4 Quantization
58. ICLR 2026Mix-Ecom: Towards Mixed-Type E-Commerce Dialogues with Complex Domain Rules
59. ICLR 2026NetArena: Dynamically Generated LLM Benchmarks for Network Applications
60. ICLR 2026NewtonBench: Benchmarking Generalizable Scientific Law Discovery in LLM Agents
61. ICLR 2026No Prompt Left Behind: Exploiting Zero-Variance Prompts in LLM Reinforcement Learning via Entropy-Guided Advantage Shaping
62. ICLR 2026On Entropy Control in LLM-RL Algorithms
63. ICLR 2026Operationalizing Data Minimization for Privacy-Preserving LLM Prompting
64. ICLR 2026PersonaX: Multimodal Datasets with LLM-Inferred Behavior Traits
65. ICLR 2026PILOT-Bench: Probabilistic Interaction for LLM Operations in Tool-driven Scenarios
66. ICLR 2026Preference Leakage: A Contamination Problem in LLM-as-a-judge
67. ICLR 2026Prompt Curriculum Learning for Efficient LLM Post-Training
68. ICLR 2026Randomization Boosts KV Caching, Learning Balances Query Load: A Joint Perspective
69. ICLR 2026Randomized Antipodal Search Done Right for Data Pareto Improvement of LLM Unlearning
70. ICLR 2026Rating Quality of Diverse Time Series Data by Meta-learning from LLM Judgment
71. ICLR 2026RCPU: Rotation-Constrained Error Compensation for Structured Pruning of a Large Language Model
72. ICLR 2026Reasoning Scaffolding: Distilling the Flow of Thought from LLMs
73. ICLR 2026Rectifying LLM Thought from Lens of Optimization
74. ICLR 2026RedSage: A Cybersecurity Generalist LLM
75. ICLR 2026RefineBench: Evaluating Refinement Capability in Language Models
76. ICLR 2026Rethinking Residual Errors in Compensation-based LLM Quantization
77. ICLR 2026Revisiting Parameter Server in LLM Post-Training
78. ICLR 2026Revisiting the Scaling Properties of Downstream Metrics in Large Language Model Training
79. ICLR 2026RL makes MLLMs see better than SFT
80. ICLR 2026RouterArena: An Open Platform for Comprehensive Comparison of LLM Routers
81. ICLR 2026Scheduling Your LLM Reinforcement Learning with Reasoning Trees
82. ICLR 2026SeedPrints: Fingerprints Can Even Tell Which Seed Your Large Language Model Was Trained From
83. ICLR 2026SelfReflect: Can LLMs Communicate Their Internal Answer Distribution?
84. ICLR 2026Semantic Regexes: Auto-Interpreting LLM Features with a Structured Language
85. ICLR 2026Semantic-Aware Diffusion LLM Inference With Adaptive Block Size
86. ICLR 2026SigLIP-HD by Fine-to-Coarse Supervision
87. ICLR 2026SK2Decompile: LLM-based Two-Phase Binary Decompilation from Skeleton to Skin
88. ICLR 2026Solving the Granularity Mismatch: Hierarchical Preference Learning for Long-Horizon LLM Agents
89. ICLR 2026SpareTrain: Fault-Tolerant LLM Training via Low-Cost Dual Modular Redundancy
90. ICLR 2026Stabilizing Policy Gradients for Sample-Efficient Reinforcement Learning in LLM Reasoning
91. ICLR 2026Taming the Fragility of KV Cache Eviction in LLM Inference
92. ICLR 2026Temperature as a Meta-Policy: Adaptive Temperature in LLM Reinforcement Learning
93. ICLR 2026Textual Bayes: Quantifying Uncertainty in LLM-Based Systems
94. ICLR 2026The Geometry of LLM Quantization: GPTQ as Babai's Nearest Plane Algorithm
95. ICLR 2026The Ideation-Execution Gap: Execution Outcomes of LLM-Generated versus Human Research Ideas
96. ICLR 2026The Mind's Transformer: Computational Neuroanatomy of LLM-Brain Alignment
97. ICLR 2026The Unseen Frontier: Pushing the Limits of LLM Sparsity with Surrogate-Free ADMM
98. ICLR 2026To Compress or Not? Pushing the Frontier of Lossless GenAI Model Weights Compression with Exponent Concentration
99. ICLR 2026Towards Multimodal Data-Driven Scientific Discovery Powered by LLM Agents
100. ICLR 2026TRAJECT-Bench:A Trajectory-Aware Benchmark for Evaluating Agentic Tool Use

FAQ

How many LLM Benchmarks papers were published at NeurIPS, ICLR, ICML and CVPR from 2023 to 2026?

544 in total: 2 in 2023, 61 in 2024, 174 in 2025, 307 in 2026.

How many LLM Benchmarks papers are at NeurIPS 2026?

NeurIPS 2026 has 138 LLM Benchmarks papers, versus 92 the year before.

How many LLM Benchmarks papers are at ICLR 2026?

ICLR 2026 has 103 LLM Benchmarks papers, versus 50 the year before.

How many LLM Benchmarks papers are at ICML 2026?

ICML 2026 has 60 LLM Benchmarks papers, versus 24 the year before.

How many LLM Benchmarks papers are at CVPR 2026?

CVPR 2026 has 6 LLM Benchmarks papers, versus 8 the year before.

Which conference publishes the most LLM Benchmarks papers?

NeurIPS, with 256 papers from 2023 to 2026.