LLM Benchmarks papers, 2023–2026

Key findings

• LLM Benchmarks went from 2 papers in 2023 to 307 in 2026, roughly 154x in three years.
• NeurIPS publishes the most LLM Benchmarks work: 256 papers from 2023 to 2026, 47% of the four-venue total.
• The fastest-growing overlap is with AI Agents: papers tagged with both rose from 1 in 2023 to 116 in 2026.
• Among the 5 topics under Evaluation and Benchmarks, it ranks #1 by 2026 paper count; the fastest grower in the group, LLM-as-Judge, is up 27x versus 154x here.

Representative papers

Spread across years and venues so every edition shows up

• NeurIPS 2026A Free Lunch in LLM Compression: Revisiting Retraining after Pruning
• ICML 2026Controlled LLM Training on Spectral Sphere
• ICLR 2026Hey, That's My Model! Introducing Chain & Hash, An LLM Fingerprinting Technique
• CVPR 20263DrawAgent Teaching LLM to Draw in 3D with Early Contrastive
• NeurIPS 2025ACCO: Accumulate While You Communicate for Communication-Overlapped Sharded LLM Training
• ICML 2025Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM’s Reasoning Capability
• ICLR 2025Catastrophic Failure of LLM Unlearning via Quantization
• CVPR 2025Period-LLM: Extending the Periodic Capability of Multimodal Large Language Model
• NeurIPS 2024Large Language Models' Expert-level Global History Knowledge Benchmark (HiST-LLM)
• ICML 2024Characterizing ResNet's Universal Approximation Capability
• ICLR 2024Faithful Explanations of Black-box NLP Models Using LLM-generated Counterfactuals
• CVPR 2024Koala: Key Frame-Conditioned Long Video-LLM
• NeurIPS 2023Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing
• NeurIPS 2026Accelerating LLM Pre-Training through Flat-Direction Dynamics Enhancement
• ICML 2026LLM Priors for ERM over Programs
• ICLR 2026How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretraining
• CVPR 20263D-VCD Hallucination Mitigation in 3D-LLM Embodied Agents through Visual Contrastive
• NeurIPS 2025BrainEC-LLM: Brain Effective Connectivity Estimation by Multiscale Mixing LLM
• ICML 2025Position: Medical Large Language Model Benchmarks Should Prioritize Construct Validity
• ICLR 2025Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Rates

All LLM Benchmarks papers

544 papers, page 9 of 11

401. NeurIPS 2025Bohdi: Heterogeneous LLM Fusion with Automatic Data Exploration
402. NeurIPS 2025BrainEC-LLM: Brain Effective Connectivity Estimation by Multiscale Mixing LLM
403. NeurIPS 2025C-SEO Bench: Does Conversational SEO Work?
404. NeurIPS 2025Can Dependencies Induced by LLM-Agent Workflows Be Trusted?
405. NeurIPS 2025Causal LLM Routing: End-to-End Regret Minimization from Observational Data
406. NeurIPS 2025ComPO: Preference Alignment via Comparison Oracles
407. NeurIPS 2025ConnectomeBench: Can LLMs proofread the connectome?
408. NeurIPS 2025Consistent Paths Lead to Truth: Self-Rewarding Reinforcement Learning for LLM Reasoning
409. NeurIPS 2025DartQuant: Efficient Rotational Distribution Calibration for LLM Quantization
410. NeurIPS 2025Dependency Matters: Enhancing LLM Reasoning with Explicit Knowledge Grounding
411. NeurIPS 2025DISC: Dynamic Decomposition Improves LLM Inference Scaling
412. NeurIPS 2025Distribution-Aligned Decoding for Efficient LLM Task Adaptation
413. NeurIPS 2025Do Language Models Use Their Depth Efficiently?
414. NeurIPS 2025DP-LLM: Runtime Model Adaptation with Dynamic Layer-wise Precision Assignment
415. NeurIPS 2025EffiBench-X: A Multi-Language Benchmark for Measuring Efficiency of LLM-Generated Code
416. NeurIPS 2025Efficient Large Language Model Inference with Neural Block Linearization
417. NeurIPS 2025Enhancing Multilingual LLM Pretraining with Model-Based Data Selection
418. NeurIPS 2025Evaluating LLM-contaminated Crowdsourcing Data Without Ground Truth
419. NeurIPS 2025Extrapolation by Association: Length Generalization Transfer In Transformers
420. NeurIPS 2025Factorio Learning Environment
421. NeurIPS 2025FFN Fusion: Rethinking Sequential Computation in Large Language Models
422. NeurIPS 2025FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on Technical Documents
423. NeurIPS 2025GPAS: Accelerating Convergence of LLM Pretraining via Gradient-Preserving Activation Scaling
424. NeurIPS 2025GPO: Learning from Critical Steps to Improve LLM Reasoning
425. NeurIPS 2025ICPC-Eval: Probing the Frontiers of LLM Reasoning with Competitive Programming Contests
426. NeurIPS 2025IR-OptSet: An Optimization-Sensitive Dataset for Advancing LLM-Based IR Optimizer
427. NeurIPS 2025Learning Grouped Lattice Vector Quantizers for Low-Bit LLM Compression
428. NeurIPS 2025Less is More: Improving LLM Alignment via Preference Data Selection
429. NeurIPS 2025Let the LLM Stick to Its Strengths: Learning to Route Economical LLM
430. NeurIPS 2025LLM Layers Immediately Correct Each Other
431. NeurIPS 2025LLM Meeting Decision Trees on Tabular Data
432. NeurIPS 2025LLM-DAMVC: A Large Language Model Assisted Dynamic Agent for Multi-View Clustering
433. NeurIPS 2025LLM-Explorer: A Plug-in Reinforcement Learning Policy Exploration Enhancement Driven by Large Language Models
434. NeurIPS 2025Lookahead Routing for Large Language Models
435. NeurIPS 2025LTD-Bench: Evaluating Large Language Models by Letting Them Draw
436. NeurIPS 2025Lua-LLM: Learning Unstructured-Sparsity Allocation for Large Language Models
437. NeurIPS 2025Masked Gated Linear Unit
438. NeurIPS 2025MESS+: Dynamically Learned Inference-Time LLM Routing in Model Zoos with Service Level Guarantees
439. NeurIPS 2025metaTextGrad: Automatically optimizing language model optimizers
440. NeurIPS 2025Mitigating Forgetting in LLM Fine-Tuning via Low-Perplexity Token Learning
441. NeurIPS 2025MODEL SHAPLEY: Find Your Ideal Parameter Player via One Gradient Backpropagation
442. NeurIPS 2025OmniBench: Towards The Future of Universal Omni-Language Models
443. NeurIPS 2025OpenUnlearning: Accelerating LLM Unlearning via Unified Benchmarking of Methods and Metrics
444. NeurIPS 2025PARALLELPROMPT: Extracting Parallelism from Large Language Model Queries
445. NeurIPS 2025Polar Sparsity: High Throughput Batched LLM Inferencing with Scalable Contextual Sparsity
446. NeurIPS 2025Predicting Empirical AI Research Outcomes with Language Models
447. NeurIPS 2025Preserving LLM Capabilities through Calibration Data Curation: From Analysis to Optimization
448. NeurIPS 2025Q-Palette: Fractional-Bit Quantizers Toward Optimal Bit Allocation for Efficient LLM Deployment
449. NeurIPS 2025QuadEnhancer: Leveraging Quadratic Transformations to Enhance Deep Neural Networks
450. NeurIPS 2025REAL: Benchmarking Autonomous Agents on Deterministic Simulations of Real Websites

FAQ

How many LLM Benchmarks papers were published at NeurIPS, ICLR, ICML and CVPR from 2023 to 2026?

544 in total: 2 in 2023, 61 in 2024, 174 in 2025, 307 in 2026.

How many LLM Benchmarks papers are at NeurIPS 2026?

NeurIPS 2026 has 138 LLM Benchmarks papers, versus 92 the year before.

How many LLM Benchmarks papers are at ICLR 2026?

ICLR 2026 has 103 LLM Benchmarks papers, versus 50 the year before.

How many LLM Benchmarks papers are at ICML 2026?

ICML 2026 has 60 LLM Benchmarks papers, versus 24 the year before.

How many LLM Benchmarks papers are at CVPR 2026?

CVPR 2026 has 6 LLM Benchmarks papers, versus 8 the year before.

Which conference publishes the most LLM Benchmarks papers?

NeurIPS, with 256 papers from 2023 to 2026.