LLM Benchmarks papers, 2023–2026

Key findings

• LLM Benchmarks went from 2 papers in 2023 to 307 in 2026, roughly 154x in three years.
• NeurIPS publishes the most LLM Benchmarks work: 256 papers from 2023 to 2026, 47% of the four-venue total.
• The fastest-growing overlap is with AI Agents: papers tagged with both rose from 1 in 2023 to 116 in 2026.
• Among the 5 topics under Evaluation and Benchmarks, it ranks #1 by 2026 paper count; the fastest grower in the group, LLM-as-Judge, is up 27x versus 154x here.

Representative papers

Spread across years and venues so every edition shows up

• NeurIPS 2026A Free Lunch in LLM Compression: Revisiting Retraining after Pruning
• ICML 2026Controlled LLM Training on Spectral Sphere
• ICLR 2026Hey, That's My Model! Introducing Chain & Hash, An LLM Fingerprinting Technique
• CVPR 20263DrawAgent Teaching LLM to Draw in 3D with Early Contrastive
• NeurIPS 2025ACCO: Accumulate While You Communicate for Communication-Overlapped Sharded LLM Training
• ICML 2025Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM’s Reasoning Capability
• ICLR 2025Catastrophic Failure of LLM Unlearning via Quantization
• CVPR 2025Period-LLM: Extending the Periodic Capability of Multimodal Large Language Model
• NeurIPS 2024Large Language Models' Expert-level Global History Knowledge Benchmark (HiST-LLM)
• ICML 2024Characterizing ResNet's Universal Approximation Capability
• ICLR 2024Faithful Explanations of Black-box NLP Models Using LLM-generated Counterfactuals
• CVPR 2024Koala: Key Frame-Conditioned Long Video-LLM
• NeurIPS 2023Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing
• NeurIPS 2026Accelerating LLM Pre-Training through Flat-Direction Dynamics Enhancement
• ICML 2026LLM Priors for ERM over Programs
• ICLR 2026How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretraining
• CVPR 20263D-VCD Hallucination Mitigation in 3D-LLM Embodied Agents through Visual Contrastive
• NeurIPS 2025BrainEC-LLM: Brain Effective Connectivity Estimation by Multiscale Mixing LLM
• ICML 2025Position: Medical Large Language Model Benchmarks Should Prioritize Construct Validity
• ICLR 2025Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Rates

All LLM Benchmarks papers

256 papers, page 5 of 6

201. NeurIPS 2025Reparameterized LLM Training via Orthogonal Equivalence Transformation
202. NeurIPS 2025Right Question is Already Half the Answer: Fully Unsupervised LLM Reasoning Incentivization
203. NeurIPS 2025Scale-invariant attention
204. NeurIPS 2025Self-Generated In-Context Examples Improve LLM Agents for Sequential Decision-Making Tasks
205. NeurIPS 2025Semantic-KG: Using Knowledge Graphs to Construct Benchmarks for Measuring Semantic Similarity
206. NeurIPS 2025Set-LLM: A Permutation-Invariant LLM
207. NeurIPS 2025Shape it Up! Restoring LLM Safety during Finetuning
208. NeurIPS 2025Sheetpedia: A 300K-Spreadsheet Corpus for Spreadsheet Intelligence and LLM Fine-Tuning
209. NeurIPS 2025Silencer: From Discovery to Mitigation of Self-Bias in LLM-as-Benchmark-Generator
210. NeurIPS 2025SmallKV: Small Model Assisted Compensation of KV Cache Compression for Efficient LLM Inference
211. NeurIPS 2025SmartCache: Context-aware Semantic Cache for Efficient Multi-turn LLM Inference
212. NeurIPS 2025Spatially-aware Weights Tokenization for NeRF-Language Models
213. NeurIPS 2025Stackelberg Self-Annotation: A Robust Approach to Data-Efficient LLM Alignment
214. NeurIPS 2025SuperGPQA: Scaling LLM Evaluation across 285 Graduate Disciplines
215. NeurIPS 2025Superposition Yields Robust Neural Scaling
216. NeurIPS 2025SWE-bench Goes Live!
217. NeurIPS 2025Tail-Optimized Caching for LLM Inference
218. NeurIPS 2025The Automated LLM Speedrunning Benchmark: Reproducing NanoGPT Improvements
219. NeurIPS 2025The Leaderboard Illusion
220. NeurIPS 2025The Lighthouse of Language: Enhancing LLM Agents via Critique-Guided Improvement
221. NeurIPS 2025TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks
222. NeurIPS 2025Thinkless: LLM Learns When to Think
223. NeurIPS 2025Toward Engineering AGI: Benchmarking the Engineering Design Capabilities of LLMs
224. NeurIPS 2025Towards Fully FP8 GEMM LLM Training at Scale
225. NeurIPS 2025Towards Thinking-Optimal Scaling of Test-Time Compute for LLM Reasoning
226. NeurIPS 2025Trajectory Bellman Residual Minimization: A Simple Value-Based Method for LLM Reasoning
227. NeurIPS 2025Transformer Copilot: Learning from The Mistake Log in LLM Fine-tuning
228. NeurIPS 2025WALL-E: World Alignment by NeuroSymbolic Learning improves World Model-based LLM Agents
229. NeurIPS 2025WritingBench: A Comprehensive Benchmark for Generative Writing
230. NeurIPS 2025Your Pre-trained LLM is Secretly an Unsupervised Confidence Calibrator
231. NeurIPS 2024Cascade Speculative Drafting for Even Faster LLM Inference
232. NeurIPS 2024DISP-LLM: Dimension-Independent Structural Pruning for Large Language Models
233. NeurIPS 2024Distributional Preference Alignment of LLMs via Optimal Transport
234. NeurIPS 2024EHRNoteQA: An LLM Benchmark for Real-World Clinical Practice Using Discharge Summaries
235. NeurIPS 2024Enhancing LLM’s Cognition via Structurization
236. NeurIPS 2024Evaluating Copyright Takedown Methods for Language Models
237. NeurIPS 2024Exploiting LLM Quantization
238. NeurIPS 2024Gorilla: Large Language Model Connected with Massive APIs
239. NeurIPS 2024HYDRA: Model Factorization Framework for Black-Box LLM Personalization
240. NeurIPS 2024IaC-Eval: A Code Generation Benchmark for Cloud Infrastructure-as-Code Programs
241. NeurIPS 2024Large Language Models' Expert-level Global History Knowledge Benchmark (HiST-LLM)
242. NeurIPS 2024LLM Processes: Numerical Predictive Distributions Conditioned on Natural Language
243. NeurIPS 2024LLM-AutoDA: Large Language Model-Driven Automatic Data Augmentation for Long-tailed Problems
244. NeurIPS 2024NoMAD-Attention: Efficient LLM Inference on CPUs Through Multiply-add-free Attention
245. NeurIPS 2024ReMoDetect: Reward Models Recognize Aligned LLM's Generations
246. NeurIPS 2024Reversing the Forget-Retain Objectives: An Efficient LLM Unlearning Framework from Logit Difference
247. NeurIPS 2024RouterDC: Query-Based Router by Dual Contrastive Learning for Assembling Large Language Models
248. NeurIPS 2024SDP4Bit: Toward 4-bit Communication Quantization in Sharded Data Parallelism for LLM Training
249. NeurIPS 2024SpecExec: Massively Parallel Speculative Decoding For Interactive LLM Inference on Consumer Devices
250. NeurIPS 2024Stacking Your Transformers: A Closer Look at Model Growth for Efficient LLM Pre-Training

FAQ

How many LLM Benchmarks papers were published at NeurIPS, ICLR, ICML and CVPR from 2023 to 2026?

544 in total: 2 in 2023, 61 in 2024, 174 in 2025, 307 in 2026.

How many LLM Benchmarks papers are at NeurIPS 2026?

NeurIPS 2026 has 138 LLM Benchmarks papers, versus 92 the year before.

How many LLM Benchmarks papers are at ICLR 2026?

ICLR 2026 has 103 LLM Benchmarks papers, versus 50 the year before.

How many LLM Benchmarks papers are at ICML 2026?

ICML 2026 has 60 LLM Benchmarks papers, versus 24 the year before.

How many LLM Benchmarks papers are at CVPR 2026?

CVPR 2026 has 6 LLM Benchmarks papers, versus 8 the year before.

Which conference publishes the most LLM Benchmarks papers?

NeurIPS, with 256 papers from 2023 to 2026.