LLM Benchmarks papers, 2023–2026

Key findings

• LLM Benchmarks went from 2 papers in 2023 to 307 in 2026, roughly 154x in three years.
• NeurIPS publishes the most LLM Benchmarks work: 256 papers from 2023 to 2026, 47% of the four-venue total.
• The fastest-growing overlap is with AI Agents: papers tagged with both rose from 1 in 2023 to 116 in 2026.
• Among the 5 topics under Evaluation and Benchmarks, it ranks #1 by 2026 paper count; the fastest grower in the group, LLM-as-Judge, is up 27x versus 154x here.

Representative papers

Spread across years and venues so every edition shows up

• NeurIPS 2026A Free Lunch in LLM Compression: Revisiting Retraining after Pruning
• ICML 2026Controlled LLM Training on Spectral Sphere
• ICLR 2026Hey, That's My Model! Introducing Chain & Hash, An LLM Fingerprinting Technique
• CVPR 20263DrawAgent Teaching LLM to Draw in 3D with Early Contrastive
• NeurIPS 2025ACCO: Accumulate While You Communicate for Communication-Overlapped Sharded LLM Training
• ICML 2025Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM’s Reasoning Capability
• ICLR 2025Catastrophic Failure of LLM Unlearning via Quantization
• CVPR 2025Period-LLM: Extending the Periodic Capability of Multimodal Large Language Model
• NeurIPS 2024Large Language Models' Expert-level Global History Knowledge Benchmark (HiST-LLM)
• ICML 2024Characterizing ResNet's Universal Approximation Capability
• ICLR 2024Faithful Explanations of Black-box NLP Models Using LLM-generated Counterfactuals
• CVPR 2024Koala: Key Frame-Conditioned Long Video-LLM
• NeurIPS 2023Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing
• NeurIPS 2026Accelerating LLM Pre-Training through Flat-Direction Dynamics Enhancement
• ICML 2026LLM Priors for ERM over Programs
• ICLR 2026How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretraining
• CVPR 20263D-VCD Hallucination Mitigation in 3D-LLM Embodied Agents through Visual Contrastive
• NeurIPS 2025BrainEC-LLM: Brain Effective Connectivity Estimation by Multiscale Mixing LLM
• ICML 2025Position: Medical Large Language Model Benchmarks Should Prioritize Construct Validity
• ICLR 2025Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Rates

All LLM Benchmarks papers

544 papers, page 8 of 11

351. ICLR 2025Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Models
352. ICLR 2025PhyloLM: Inferring the Phylogeny of Large Language Models and Predicting their Performances in Benchmarks
353. ICLR 2025RevisEval: Improving LLM-as-a-Judge via Response-Adapted References
354. ICLR 2025Robotouille: An Asynchronous Planning Benchmark for LLM Agents
355. ICLR 2025SeedLM: Compressing LLM Weights into Seeds of Pseudo-Random Generators
356. ICLR 2025Selective Attention Improves Transformer
357. ICLR 2025SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal
358. ICLR 2025SpinQuant: LLM Quantization with Learned Rotations
359. ICLR 2025SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget
360. ICLR 2025StringLLM: Understanding the String Processing Capability of Large Language Models
361. ICLR 2025Style Outweighs Substance: Failure Modes of LLM Judges in Alignment Benchmarking
362. ICLR 2025TODO: Enhancing LLM Alignment with Ternary Preferences
363. ICLR 2025ToolACE: Winning the Points of LLM Function Calling
364. ICLR 2025Towards a Theoretical Understanding of Synthetic Data in LLM Post-Training: A Reverse-Bottleneck Perspective
365. ICLR 2025Trajectory-LLM: A Language-based Data Generator for Trajectory Prediction in Autonomous Driving
366. ICML 2025CateKV: On Sequential Consistency for Long-Context LLM Inference Acceleration
367. ICML 2025Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM’s Reasoning Capability
368. ICML 2025Demystifying Singular Defects in Large Language Models
369. ICML 2025Exploring Criteria of Loss Reweighting to Enhance LLM Unlearning
370. ICML 2025FlatQuant: Flatness Matters for LLM Quantization
371. ICML 2025From Crowdsourced Data to High-quality Benchmarks: Arena-Hard and Benchbuilder Pipeline
372. ICML 2025From Jack of All Trades to Master of One: Specializing LLM-based Autoraters to a Test Set
373. ICML 2025La RoSA: Enhancing LLM Efficiency via Layerwise Rotated Sparse Activation
374. ICML 2025Large Language-Geometry Model: When LLM meets Equivariance
375. ICML 2025LongRoPE2: Near-Lossless LLM Context Window Scaling
376. ICML 2025Memorization Sinks: Isolating Memorization during LLM Training
377. ICML 2025Minerva: A Programmable Memory Test Benchmark for Language Models
378. ICML 2025Overestimation in LLM Evaluation: A Controlled Large-Scale Study on Data Contamination’s Impact on Machine Translation
379. ICML 2025Position: Don't Use the CLT in LLM Evals With Fewer Than a Few Hundred Datapoints
380. ICML 2025Position: Medical Large Language Model Benchmarks Should Prioritize Construct Validity
381. ICML 2025Position: Principles of Animal Cognition to Improve LLM Evaluations
382. ICML 2025Position: Scaling LLM Agents Requires Asymptotic Analysis with LLM Primitives
383. ICML 2025Position: The Most Expensive Part of an LLM *should* be its Training Data
384. ICML 2025RBench: Graduate-level Multi-disciplinary Benchmarks for LLM & MLLM Complex Reasoning Evaluation
385. ICML 2025RE-Bench: Evaluating Frontier AI R&D Capabilities of Language Model Agents against Human Experts
386. ICML 2025Reward-Augmented Data Enhances Direct Preference Alignment of LLMs
387. ICML 2025RZ-NAS: Enhancing LLM-guided Neural Architecture Search via Reflective Zero-Cost Strategy
388. ICML 2025Sketch to Adapt: Fine-Tunable Sketches for Efficient LLM Adaptation
389. ICML 2025TruthFlow: Truthful LLM Generation via Representation Flow Correction
390. NeurIPS 2025$Q\sharp$: Provably Optimal Distributional RL for LLM Post-Training
391. NeurIPS 2025Accelerating Block Coordinate Descent for LLM Finetuning via Landscape Expansion
392. NeurIPS 2025ACCO: Accumulate While You Communicate for Communication-Overlapped Sharded LLM Training
393. NeurIPS 2025Accurate KV Cache Eviction via Anchor Direction Projection for Efficient LLM Inference
394. NeurIPS 2025Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts
395. NeurIPS 2025AgentRecBench: Benchmarking LLM Agent-based Personalized Recommender Systems
396. NeurIPS 2025Antidistillation Sampling
397. NeurIPS 2025AutoJudge: Judge Decoding Without Manual Annotation
398. NeurIPS 2025BenchmarkCards: Standardized Documentation for Large Language Model Benchmarks
399. NeurIPS 2025Beyond Chemical QA: Evaluating LLM's Chemical Reasoning with Modular Chemical Operations
400. NeurIPS 2025BLEUBERI: BLEU is a surprisingly effective reward for instruction following

FAQ

How many LLM Benchmarks papers were published at NeurIPS, ICLR, ICML and CVPR from 2023 to 2026?

544 in total: 2 in 2023, 61 in 2024, 174 in 2025, 307 in 2026.

How many LLM Benchmarks papers are at NeurIPS 2026?

NeurIPS 2026 has 138 LLM Benchmarks papers, versus 92 the year before.

How many LLM Benchmarks papers are at ICLR 2026?

ICLR 2026 has 103 LLM Benchmarks papers, versus 50 the year before.

How many LLM Benchmarks papers are at ICML 2026?

ICML 2026 has 60 LLM Benchmarks papers, versus 24 the year before.

How many LLM Benchmarks papers are at CVPR 2026?

CVPR 2026 has 6 LLM Benchmarks papers, versus 8 the year before.

Which conference publishes the most LLM Benchmarks papers?

NeurIPS, with 256 papers from 2023 to 2026.