LLM Benchmarks papers, 2023–2026

Key findings

• LLM Benchmarks went from 2 papers in 2023 to 307 in 2026, roughly 154x in three years.
• NeurIPS publishes the most LLM Benchmarks work: 256 papers from 2023 to 2026, 47% of the four-venue total.
• The fastest-growing overlap is with AI Agents: papers tagged with both rose from 1 in 2023 to 116 in 2026.
• Among the 5 topics under Evaluation and Benchmarks, it ranks #1 by 2026 paper count; the fastest grower in the group, LLM-as-Judge, is up 27x versus 154x here.

Representative papers

Spread across years and venues so every edition shows up

• NeurIPS 2026A Free Lunch in LLM Compression: Revisiting Retraining after Pruning
• ICML 2026Controlled LLM Training on Spectral Sphere
• ICLR 2026Hey, That's My Model! Introducing Chain & Hash, An LLM Fingerprinting Technique
• CVPR 20263DrawAgent Teaching LLM to Draw in 3D with Early Contrastive
• NeurIPS 2025ACCO: Accumulate While You Communicate for Communication-Overlapped Sharded LLM Training
• ICML 2025Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM’s Reasoning Capability
• ICLR 2025Catastrophic Failure of LLM Unlearning via Quantization
• CVPR 2025Period-LLM: Extending the Periodic Capability of Multimodal Large Language Model
• NeurIPS 2024Large Language Models' Expert-level Global History Knowledge Benchmark (HiST-LLM)
• ICML 2024Characterizing ResNet's Universal Approximation Capability
• ICLR 2024Faithful Explanations of Black-box NLP Models Using LLM-generated Counterfactuals
• CVPR 2024Koala: Key Frame-Conditioned Long Video-LLM
• NeurIPS 2023Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing
• NeurIPS 2026Accelerating LLM Pre-Training through Flat-Direction Dynamics Enhancement
• ICML 2026LLM Priors for ERM over Programs
• ICLR 2026How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretraining
• CVPR 20263D-VCD Hallucination Mitigation in 3D-LLM Embodied Agents through Visual Contrastive
• NeurIPS 2025BrainEC-LLM: Brain Effective Connectivity Estimation by Multiscale Mixing LLM
• ICML 2025Position: Medical Large Language Model Benchmarks Should Prioritize Construct Validity
• ICLR 2025Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Rates

All LLM Benchmarks papers

544 papers, page 6 of 11

251. NeurIPS 2026Meta-Learning Preferences for Multilingual LLM Alignment
252. NeurIPS 2026MinPath: Learning Efficient LLM Reasoning via Minimal Dependency Paths
253. NeurIPS 2026More Value per Key: Asymmetric Sparse Attention for Faster LLM Decoding
254. NeurIPS 2026Neglected Free Lunch from Post-training: Progress Advantage for LLM Agents
255. NeurIPS 2026Not All Gap Correction Helps: A Geometric View of External Information in LLM Inference
256. NeurIPS 2026Not All Proofs Are Equal: Evaluating LLM Proof Quality Beyond Correctness
257. NeurIPS 2026NuMuon: Nuclear-Norm-Constrained Muon for Compressible LLM Training
258. NeurIPS 2026NyoomFloat12: Accelerating LLM Inference via Lossless 12-bit Weight Compression
259. NeurIPS 2026On-Policy Consistency Training Improves LLM Safety with Minimal Capability Degradation
260. NeurIPS 2026OrchestraRL: Learning to Orchestrate LLM Agent Swarms with Entropy-Aware Communication Control
261. NeurIPS 2026Pacing Branch Parallelism in LLM Serving
262. NeurIPS 2026PCBInnoBench: Benchmarking LLM Agents on Real-World PCB Design
263. NeurIPS 2026Preserving Exploration for LLM Reasoning via Mean Order-Statistic Alignment
264. NeurIPS 2026PROBE: Learning to Audit Policy Compliance in Tool-Using LLM Agents
265. NeurIPS 2026Process-Aware LNS for Large-Scale MILP via Context-Enhanced Fine-Tuned LLM-driven Selector
266. NeurIPS 2026RA$^2$: Retain-Anchored Attraction Preserves Forget-Adjacent Utility in LLM Unlearning
267. NeurIPS 2026Rethinking Latency Denial-of-Service: Attack the LLM Serving Framework, Not the Model
268. NeurIPS 2026RIGOR: Risk-Gated Topology Adaptation for Robust LLM Multi-Agent Reasoning
269. NeurIPS 2026Salvation Lies Within: Proactive Prefix Re-forming for LLM-based Tagging
270. NeurIPS 2026SchemeArena: Factorized Stress Testing of Scheming in LLM Agents
271. NeurIPS 2026SCTI: Self-Calibrated Trident Identification of Black-Box LLM Watermarks
272. NeurIPS 2026Sequential Behavioral Watermarking for LLM Agents
273. NeurIPS 2026SFT-then-RL Outperforms Mixed-Policy Methods for LLM Reasoning
274. NeurIPS 2026SGEvolve: Semantic Gradient Guidance for LLM-Driven Evolutionary Search
275. NeurIPS 2026SkillMaster: Toward Autonomous Skill Mastery in LLM Agents
276. NeurIPS 2026Soteria: Formally Verified Planning with Runtime Enforcement for Safe LLM Agents
277. NeurIPS 2026Spatial-IQ: Deconstructing Spatial Intelligence via Hierarchical Capability Tests
278. NeurIPS 2026Specialists Hold, Generalists Discount: Asymmetric Equilibrium in LLM Routing Auctions
279. NeurIPS 2026Strategic Evaluation: Incentivizing AI Capability Coverage with Private Benchmarks
280. NeurIPS 2026STRIDE: Learnable Stepwise Language Feedback for LLM Reasoning
281. NeurIPS 2026Strong Teacher Not Needed? On Distillation in LLM Pretraining
282. NeurIPS 2026SudoBench: A Contextual Authorization Benchmark for LLM Agents
283. NeurIPS 2026SVDP: Training-Free Contextual Sparsity Predictors for Fast LLM Inference
284. NeurIPS 2026TextSeal: A Localized LLM Watermark for Provenance & Distillation Protection
285. NeurIPS 2026The Geometry of Forgetting: Temporal Knowledge Drift as an Independent Axis in LLM Representations
286. NeurIPS 2026The Illusion of Diversity: Aligning LLM Exploration via Effective Entropy
287. NeurIPS 2026The Labeling Problem in Hallucination Detection Benchmarks: An Empirical Evaluation
288. NeurIPS 2026The Scaling Paradox of Tool-Calling LLM Agents Under Realistic MCP Faults
289. NeurIPS 2026The Text Uncanny Valley: Non-Monotonic Performance Degradation in LLM Information Retrieval
290. NeurIPS 2026Tokens-per-Parameter Coverage Is Critical for Robust LLM Scaling Law Extrapolation
291. NeurIPS 2026Trust, but Don’t Verify: Epistemic Blind Spots in LLM Source Evaluation
292. NeurIPS 2026TSFM Meets LLM: Context as Covariate
293. NeurIPS 2026UMAS: System-Level Uncertainty Quantification for Multi-Agent LLM Systems
294. NeurIPS 2026Valid Best-Model Identification for LLM Evaluation via Low-Rank Factorization
295. NeurIPS 2026Values as Style: Disentangling Values from Semantics with One-Way Mixing for Low-Damage LLM Steering
296. NeurIPS 2026VEHBench: A Stage-Local Diagnostic Benchmark for LLM-Assisted Vibration Energy Harvester Design
297. NeurIPS 2026Verifiable LLM-Guided Focal SMT Solving for Quantified Arrays
298. NeurIPS 2026Verified-Source Authority Is Not Generic Sycophancy: Cue-Family Decomposition of LLM Compliance
299. NeurIPS 2026VeriVul: A Verification-Guided Framework for Generating Realistic Vulnerability Benchmarks
300. NeurIPS 2026ViroGym: Realistic Large-Scale Benchmarks for Evaluating Viral Proteins

FAQ

How many LLM Benchmarks papers were published at NeurIPS, ICLR, ICML and CVPR from 2023 to 2026?

544 in total: 2 in 2023, 61 in 2024, 174 in 2025, 307 in 2026.

How many LLM Benchmarks papers are at NeurIPS 2026?

NeurIPS 2026 has 138 LLM Benchmarks papers, versus 92 the year before.

How many LLM Benchmarks papers are at ICLR 2026?

ICLR 2026 has 103 LLM Benchmarks papers, versus 50 the year before.

How many LLM Benchmarks papers are at ICML 2026?

ICML 2026 has 60 LLM Benchmarks papers, versus 24 the year before.

How many LLM Benchmarks papers are at CVPR 2026?

CVPR 2026 has 6 LLM Benchmarks papers, versus 8 the year before.

Which conference publishes the most LLM Benchmarks papers?

NeurIPS, with 256 papers from 2023 to 2026.