LLM-as-Judge papers, 2023–2026

Key findings

• LLM-as-Judge went from 8 papers in 2023 to 219 in 2026, roughly 27x in three years.
• NeurIPS publishes the most LLM-as-Judge work: 173 papers from 2023 to 2026, 43% of the four-venue total.
• The fastest-growing overlap is with AI Agents: papers tagged with both rose from 1 in 2023 to 78 in 2026.
• Among the 5 topics under Evaluation and Benchmarks, it ranks #2 by 2026 paper count; the fastest grower in the group, LLM Benchmarks, is up 154x versus 27x here.

Representative papers

Spread across years and venues so every edition shows up

• NeurIPS 2026A Free Lunch in LLM Compression: Revisiting Retraining after Pruning
• ICML 2026A Judge-Aware Ranking Framework for Evaluating Large Language Models without Ground Truth
• ICLR 2026Addressing Pitfalls in the Evaluation of Uncertainty Estimation Methods for Natural Language Generation
• CVPR 2026Pixels Dont Lie But Your Detector Might Bootstrapping MLLM-as-a-Judge for
• NeurIPS 2025$\texttt{BetaConform}$: Efficient MAP Estimation of LLM Ensemble Judgment Performance with Prior Transfer
• ICML 2025Active Evaluation Acquisition for Efficient LLM Benchmarking
• ICLR 2025An Empirical Analysis of Uncertainty in Large Language Model Evaluations
• CVPR 2025ASHiTA: Automatic Scene-grounded HIerarchical Task Analysis
• NeurIPS 2024BertaQA: How Much Do Language Models Know About Local Culture?
• ICML 2024CHAI: Clustered Head Attention for Efficient LLM Inference
• ICLR 2024PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization
• NeurIPS 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• ICML 2023SAM operates far from home: eigenvalue regularization as a dynamical phenomenon
• ICLR 2023On the complexity of nonsmooth automatic differentiation
• NeurIPS 2026A Stratified Multi-Rater Evaluation of LLM-Based Virtual Standardized Patients with a Deployed Data-Generation Platform
• ICML 2026Are LLM Evaluators Really Narcissists? Sanity Checking Self-Preference Evaluations
• ICLR 2026AutoMetrics: Approximate Human Judgments with Automatically Generated Evaluators
• CVPR 20263DrawAgent Teaching LLM to Draw in 3D with Early Contrastive
• NeurIPS 2025Adaptive Prediction-Powered AutoEval with Reliability and Efficiency Guarantees
• ICML 2025Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluators

All LLM-as-Judge papers

401 papers, page 8 of 9

351. ICLR 2024Generative Judge for Evaluating Alignment
352. ICLR 2024On the Humanity of Conversational AI: Evaluating the Psychological Portrayal of LLMs
353. ICLR 2024PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization
354. ICLR 2024PB-LLM: Partially Binarized Large Language Models
355. ICLR 2024Prometheus: Inducing Fine-Grained Evaluation Capability in Language Models
356. ICLR 2024The LLM Surgeon
357. ICLR 2024To the Cutoff... and Beyond? A Longitudinal Perspective on LLM Data Contamination
358. ICML 2024CHAI: Clustered Head Attention for Efficient LLM Inference
359. ICML 2024ExCP: Extreme LLM Checkpoint Compression via Weight-Momentum Joint Shrinking
360. ICML 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark
361. ICML 2024MuxServe: Flexible Spatial-Temporal Multiplexing for Multiple LLM Serving
362. ICML 2024NExT-GPT: Any-to-Any Multimodal LLM
363. ICML 2024Position: Fundamental Limitations of LLM Censorship Necessitate New Approaches
364. ICML 2024Position: Key Claims in LLM Research Have a Long Tail of Footnotes
365. ICML 2024Self-Rewarding Language Models
366. ICML 2024SparQ Attention: Bandwidth-Efficient LLM Inference
367. NeurIPS 2024Accuracy is Not All You Need
368. NeurIPS 2024AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents
369. NeurIPS 2024Apathetic or Empathetic? Evaluating LLMs' Emotional Alignments with Humans
370. NeurIPS 2024Automatic Outlier Rectification via Optimal Transport
371. NeurIPS 2024AvaTaR: Optimizing LLM Agents for Tool Usage via Contrastive Reasoning
372. NeurIPS 2024Benchmarking Complex Instruction-Following with Multiple Constraints Composition
373. NeurIPS 2024BertaQA: How Much Do Language Models Know About Local Culture?
374. NeurIPS 2024CLAVE: An Adaptive Framework for Evaluating Values of LLM Generated Responses
375. NeurIPS 2024CriticEval: Evaluating Large-scale Language Model as Critic
376. NeurIPS 2024Efficient multi-prompt evaluation of LLMs
377. NeurIPS 2024Enhancing LLM’s Cognition via Structurization
378. NeurIPS 2024Evaluating Copyright Takedown Methods for Language Models
379. NeurIPS 2024Exploiting LLM Quantization
380. NeurIPS 2024IDGen: Item Discrimination Induced Prompt Generation for LLM Evaluation
381. NeurIPS 2024IQA-EVAL: Automatic Evaluation of Human-Model Interactive Question Answering
382. NeurIPS 2024Large Language Models' Expert-level Global History Knowledge Benchmark (HiST-LLM)
383. NeurIPS 2024Learn To be Efficient: Build Structured Sparsity in Large Language Models
384. NeurIPS 2024LLM-AutoDA: Large Language Model-Driven Automatic Data Augmentation for Long-tailed Problems
385. NeurIPS 2024MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixtures
386. NeurIPS 2024On scalable oversight with weak LLMs judging strong LLMs
387. NeurIPS 2024Prediction-Powered Ranking of Large Language Models
388. NeurIPS 2024ReMoDetect: Reward Models Recognize Aligned LLM's Generations
389. NeurIPS 2024Reversing the Forget-Retain Objectives: An Efficient LLM Unlearning Framework from Logit Difference
390. NeurIPS 2024Stacking Your Transformers: A Closer Look at Model Growth for Efficient LLM Pre-Training
391. NeurIPS 2024Star-Agents: Automatic Data Optimization with LLM Agents for Instruction Tuning
392. NeurIPS 2024The Art of Saying No: Contextual Noncompliance in Language Models
393. NeurIPS 2024WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Models
394. ICLR 2023On the complexity of nonsmooth automatic differentiation
395. ICML 2023AutoCoreset: An Automatic Practical Coreset Construction Framework
396. ICML 2023SAM operates far from home: eigenvalue regularization as a dynamical phenomenon
397. NeurIPS 2023ClusterFomer: Clustering As A Universal Visual Learner
398. NeurIPS 2023FELM: Benchmarking Factuality Evaluation of Large Language Models
399. NeurIPS 2023Jailbroken: How Does LLM Safety Training Fail?
400. NeurIPS 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

FAQ

How many LLM-as-Judge papers were published at NeurIPS, ICLR, ICML and CVPR from 2023 to 2026?

401 in total: 8 in 2023, 48 in 2024, 126 in 2025, 219 in 2026.

How many LLM-as-Judge papers are at NeurIPS 2026?

NeurIPS 2026 has 88 LLM-as-Judge papers, versus 53 the year before.

How many LLM-as-Judge papers are at ICLR 2026?

ICLR 2026 has 62 LLM-as-Judge papers, versus 37 the year before.

How many LLM-as-Judge papers are at ICML 2026?

ICML 2026 has 64 LLM-as-Judge papers, versus 33 the year before.

How many LLM-as-Judge papers are at CVPR 2026?

CVPR 2026 has 5 LLM-as-Judge papers, versus 3 the year before.

Which conference publishes the most LLM-as-Judge papers?

NeurIPS, with 173 papers from 2023 to 2026.