LLM Benchmarks papers, 2023–2026

Key findings

• LLM Benchmarks went from 2 papers in 2023 to 307 in 2026, roughly 154x in three years.
• NeurIPS publishes the most LLM Benchmarks work: 256 papers from 2023 to 2026, 47% of the four-venue total.
• The fastest-growing overlap is with AI Agents: papers tagged with both rose from 1 in 2023 to 116 in 2026.
• Among the 5 topics under Evaluation and Benchmarks, it ranks #1 by 2026 paper count; the fastest grower in the group, LLM-as-Judge, is up 27x versus 154x here.

Representative papers

Spread across years and venues so every edition shows up

• NeurIPS 2026A Free Lunch in LLM Compression: Revisiting Retraining after Pruning
• ICML 2026Controlled LLM Training on Spectral Sphere
• ICLR 2026Hey, That's My Model! Introducing Chain & Hash, An LLM Fingerprinting Technique
• CVPR 20263DrawAgent Teaching LLM to Draw in 3D with Early Contrastive
• NeurIPS 2025ACCO: Accumulate While You Communicate for Communication-Overlapped Sharded LLM Training
• ICML 2025Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM’s Reasoning Capability
• ICLR 2025Catastrophic Failure of LLM Unlearning via Quantization
• CVPR 2025Period-LLM: Extending the Periodic Capability of Multimodal Large Language Model
• NeurIPS 2024Large Language Models' Expert-level Global History Knowledge Benchmark (HiST-LLM)
• ICML 2024Characterizing ResNet's Universal Approximation Capability
• ICLR 2024Faithful Explanations of Black-box NLP Models Using LLM-generated Counterfactuals
• CVPR 2024Koala: Key Frame-Conditioned Long Video-LLM
• NeurIPS 2023Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing
• NeurIPS 2026Accelerating LLM Pre-Training through Flat-Direction Dynamics Enhancement
• ICML 2026LLM Priors for ERM over Programs
• ICLR 2026How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretraining
• CVPR 20263D-VCD Hallucination Mitigation in 3D-LLM Embodied Agents through Visual Contrastive
• NeurIPS 2025BrainEC-LLM: Brain Effective Connectivity Estimation by Multiscale Mixing LLM
• ICML 2025Position: Medical Large Language Model Benchmarks Should Prioritize Construct Validity
• ICLR 2025Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Rates

All LLM Benchmarks papers

256 papers, page 2 of 6

51. NeurIPS 2026FEP-Agent: Grounding LLM Agent Self-Evolution in Active Inference with Semantic Memory
52. NeurIPS 2026Fisher-R1: Training LLM Agents for Reliable Hypothesis Testing
53. NeurIPS 2026Forced Deferral: Manipulating Routing Decisions in Multimodal LLM Cascades
54. NeurIPS 2026Forced Orders: What LLM Leaderboards Hide About Model Comparisons
55. NeurIPS 2026From Chats to Markets: AgenticPay for LLM-Powered Negotiation in Multi-Agent Commerce
56. NeurIPS 2026From History to State: Constant-Context Skill Learning for LLM Agents
57. NeurIPS 2026From Infrastructure to Interface, the AI Value Chain Drives LLM Homogenization
58. NeurIPS 2026From Local Skills to Long-Horizon Tasks: Progressive Skill Exploration for LLM Web Agents
59. NeurIPS 2026From Reasoning Chains to Verifiable Subproblems: Curriculum Reinforcement Learning Enables Credit Assignment for LLM Reasoning
60. NeurIPS 2026From Talking Words to Sharing Thoughts: Scalable Multi-LLM Aggregation via Structured Message Passing
61. NeurIPS 2026From Walls to Synergy: A Joint LLM-Evolution Framework for MILP Solvers
62. NeurIPS 2026GraphInstruct: A Progressive Benchmark for Diagnosing Capability Gaps in LLM Graph Generation
63. NeurIPS 2026GroupMemBench: Benchmarking LLM Agent Memory in Multi-Party Conversations
64. NeurIPS 2026HARP: Training-Free Dual-Profile Agentic Communication for LLM-Based Recommendation
65. NeurIPS 2026Hidden Positives: Why Code Retrieval Benchmarks Underestimate Model Quality
66. NeurIPS 2026HireWatch: Evaluating LLM Compliance with U.S. Employment Law Under Contextual Pressure
67. NeurIPS 2026Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering
68. NeurIPS 2026Improving LLM Final Representations with Inter-Layer Geometry
69. NeurIPS 2026Instrumental Choices: Measuring the Propensity of LLM Agents to Pursue Instrumental Behaviors
70. NeurIPS 2026Jacobian Scopes: A Unified Geometric Framework for Token-Level LLM Attributions
71. NeurIPS 2026Joint Sequence--Vocabulary Selection for Efficient LLM Distillation
72. NeurIPS 2026JuICE: A Benchmark for Evaluating LLM-Judge in Identifying Cultural Errors
73. NeurIPS 2026Knowing When to Ask: Segment-Level Credit Assignment for LLM Tool Use
74. NeurIPS 2026Learning Cost-Efficient Autoscaling for Latency-Constrained Disaggregated LLM Serving
75. NeurIPS 2026Learning What to Forget: Improving LLM Unlearning via Learned Token-Level Importance
76. NeurIPS 2026LLM Agents Already Know When to Call Tools - Even Without Reasoning
77. NeurIPS 2026LLM-AutoSciLab: Closed-Loop Scientific Law Discovery via Active Experimentation with LLMs
78. NeurIPS 2026LLM-Enhanced Random Forests in Orthogonal Hyperbolic Subspaces for Tabular Learning
79. NeurIPS 2026MAGE: All-[MASK] Block Already Knows Where to Look in Block Diffusion LLM
80. NeurIPS 2026Measuring Collapse and Correction in Homogeneous-Panel LLM Debate
81. NeurIPS 2026MemeEconomy : Do LLM Agents Trade Ethics for Survival?
82. NeurIPS 2026Meta-Learning Preferences for Multilingual LLM Alignment
83. NeurIPS 2026MinPath: Learning Efficient LLM Reasoning via Minimal Dependency Paths
84. NeurIPS 2026More Value per Key: Asymmetric Sparse Attention for Faster LLM Decoding
85. NeurIPS 2026Neglected Free Lunch from Post-training: Progress Advantage for LLM Agents
86. NeurIPS 2026Not All Gap Correction Helps: A Geometric View of External Information in LLM Inference
87. NeurIPS 2026Not All Proofs Are Equal: Evaluating LLM Proof Quality Beyond Correctness
88. NeurIPS 2026NuMuon: Nuclear-Norm-Constrained Muon for Compressible LLM Training
89. NeurIPS 2026NyoomFloat12: Accelerating LLM Inference via Lossless 12-bit Weight Compression
90. NeurIPS 2026On-Policy Consistency Training Improves LLM Safety with Minimal Capability Degradation
91. NeurIPS 2026OrchestraRL: Learning to Orchestrate LLM Agent Swarms with Entropy-Aware Communication Control
92. NeurIPS 2026Pacing Branch Parallelism in LLM Serving
93. NeurIPS 2026PCBInnoBench: Benchmarking LLM Agents on Real-World PCB Design
94. NeurIPS 2026Preserving Exploration for LLM Reasoning via Mean Order-Statistic Alignment
95. NeurIPS 2026PROBE: Learning to Audit Policy Compliance in Tool-Using LLM Agents
96. NeurIPS 2026Process-Aware LNS for Large-Scale MILP via Context-Enhanced Fine-Tuned LLM-driven Selector
97. NeurIPS 2026RA$^2$: Retain-Anchored Attraction Preserves Forget-Adjacent Utility in LLM Unlearning
98. NeurIPS 2026Rethinking Latency Denial-of-Service: Attack the LLM Serving Framework, Not the Model
99. NeurIPS 2026RIGOR: Risk-Gated Topology Adaptation for Robust LLM Multi-Agent Reasoning
100. NeurIPS 2026Salvation Lies Within: Proactive Prefix Re-forming for LLM-based Tagging

FAQ

How many LLM Benchmarks papers were published at NeurIPS, ICLR, ICML and CVPR from 2023 to 2026?

544 in total: 2 in 2023, 61 in 2024, 174 in 2025, 307 in 2026.

How many LLM Benchmarks papers are at NeurIPS 2026?

NeurIPS 2026 has 138 LLM Benchmarks papers, versus 92 the year before.

How many LLM Benchmarks papers are at ICLR 2026?

ICLR 2026 has 103 LLM Benchmarks papers, versus 50 the year before.

How many LLM Benchmarks papers are at ICML 2026?

ICML 2026 has 60 LLM Benchmarks papers, versus 24 the year before.

How many LLM Benchmarks papers are at CVPR 2026?

CVPR 2026 has 6 LLM Benchmarks papers, versus 8 the year before.

Which conference publishes the most LLM Benchmarks papers?

NeurIPS, with 256 papers from 2023 to 2026.