LLM-as-Judge papers, 2023–2026

Key findings

• LLM-as-Judge went from 8 papers in 2023 to 219 in 2026, roughly 27x in three years.
• NeurIPS publishes the most LLM-as-Judge work: 173 papers from 2023 to 2026, 43% of the four-venue total.
• The fastest-growing overlap is with AI Agents: papers tagged with both rose from 1 in 2023 to 78 in 2026.
• Among the 5 topics under Evaluation and Benchmarks, it ranks #2 by 2026 paper count; the fastest grower in the group, LLM Benchmarks, is up 154x versus 27x here.

Representative papers

Spread across years and venues so every edition shows up

• NeurIPS 2026A Free Lunch in LLM Compression: Revisiting Retraining after Pruning
• ICML 2026A Judge-Aware Ranking Framework for Evaluating Large Language Models without Ground Truth
• ICLR 2026Addressing Pitfalls in the Evaluation of Uncertainty Estimation Methods for Natural Language Generation
• CVPR 2026Pixels Dont Lie But Your Detector Might Bootstrapping MLLM-as-a-Judge for
• NeurIPS 2025$\texttt{BetaConform}$: Efficient MAP Estimation of LLM Ensemble Judgment Performance with Prior Transfer
• ICML 2025Active Evaluation Acquisition for Efficient LLM Benchmarking
• ICLR 2025An Empirical Analysis of Uncertainty in Large Language Model Evaluations
• CVPR 2025ASHiTA: Automatic Scene-grounded HIerarchical Task Analysis
• NeurIPS 2024BertaQA: How Much Do Language Models Know About Local Culture?
• ICML 2024CHAI: Clustered Head Attention for Efficient LLM Inference
• ICLR 2024PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization
• NeurIPS 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• ICML 2023SAM operates far from home: eigenvalue regularization as a dynamical phenomenon
• ICLR 2023On the complexity of nonsmooth automatic differentiation
• NeurIPS 2026A Stratified Multi-Rater Evaluation of LLM-Based Virtual Standardized Patients with a Deployed Data-Generation Platform
• ICML 2026Are LLM Evaluators Really Narcissists? Sanity Checking Self-Preference Evaluations
• ICLR 2026AutoMetrics: Approximate Human Judgments with Automatically Generated Evaluators
• CVPR 20263DrawAgent Teaching LLM to Draw in 3D with Early Contrastive
• NeurIPS 2025Adaptive Prediction-Powered AutoEval with Reliability and Efficiency Guarantees
• ICML 2025Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluators

All LLM-as-Judge papers

37 papers, page 1 of 1

1. ICLR 2025A Statistical Framework for Ranking LLM-based Chatbots
2. ICLR 2025AgentSquare: Automatic LLM Agent Search in Modular Design Space
3. ICLR 2025An Empirical Analysis of Uncertainty in Large Language Model Evaluations
4. ICLR 2025BadJudge: Backdoor Vulnerabilities of LLM-As-A-Judge
5. ICLR 2025Better than Your Teacher: LLM Agents that learn from Privileged AI Feedback
6. ICLR 2025Beyond correlation: The impact of human uncertainty in measuring the effectiveness of automatic evaluation and LLM-as-a-judge
7. ICLR 2025Catastrophic Failure of LLM Unlearning via Quantization
8. ICLR 2025Cheating Automatic LLM Benchmarks: Null Models Achieve High Win Rates
9. ICLR 2025Combatting Dimensional Collapse in LLM Pre-Training Data via Submodular File Selection
10. ICLR 2025Controlled LLM Decoding via Discrete Auto-regressive Biasing
11. ICLR 2025Diversity Empowers Intelligence: Integrating Expertise of Software Engineering Agents
12. ICLR 2025Do LLMs have Consistent Values?
13. ICLR 2025GraphEval: A Lightweight Graph-Based LLM Framework for Idea Evaluation
14. ICLR 2025HShare: Fast LLM Decoding by Hierarchical Key-Value Sharing
15. ICLR 2025Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignment
16. ICLR 2025JudgeBench: A Benchmark for Evaluating LLM-Based Judges
17. ICLR 2025Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge
18. ICLR 2025Language Model Alignment in Multilingual Trolley Problems
19. ICLR 2025Language Models Are Implicitly Continuous
20. ICLR 2025Latent Space Chain-of-Embedding Enables Output-free LLM Self-Evaluation
21. ICLR 2025Learning LLM-as-a-Judge for Preference Alignment
22. ICLR 2025Limits to scalable evaluation at the frontier: LLM as judge won’t beat twice the data
23. ICLR 2025LiveBench: A Challenging, Contamination-Limited LLM Benchmark
24. ICLR 2025On Calibration of LLM-based Guard Models for Reliable Content Moderation
25. ICLR 2025Pangea: A Fully Open Multilingual Multimodal LLM for 39 Languages
26. ICLR 2025Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Models
27. ICLR 2025Progressive Mixed-Precision Decoding for Efficient LLM Inference
28. ICLR 2025RevisEval: Improving LLM-as-a-Judge via Response-Adapted References
29. ICLR 2025RMB: Comprehensively benchmarking reward models in LLM alignment
30. ICLR 2025RocketEval: Efficient automated LLM evaluation via grading checklist
31. ICLR 2025SeedLM: Compressing LLM Weights into Seeds of Pseudo-Random Generators
32. ICLR 2025Selective Attention Improves Transformer
33. ICLR 2025SpinQuant: LLM Quantization with Learned Rotations
34. ICLR 2025SqueezeAttention: 2D Management of KV-Cache in LLM Inference via Layer-wise Optimal Budget
35. ICLR 2025Style Outweighs Substance: Failure Modes of LLM Judges in Alignment Benchmarking
36. ICLR 2025ToolACE: Winning the Points of LLM Function Calling
37. ICLR 2025Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreement

FAQ

How many LLM-as-Judge papers were published at NeurIPS, ICLR, ICML and CVPR from 2023 to 2026?

401 in total: 8 in 2023, 48 in 2024, 126 in 2025, 219 in 2026.

How many LLM-as-Judge papers are at NeurIPS 2026?

NeurIPS 2026 has 88 LLM-as-Judge papers, versus 53 the year before.

How many LLM-as-Judge papers are at ICLR 2026?

ICLR 2026 has 62 LLM-as-Judge papers, versus 37 the year before.

How many LLM-as-Judge papers are at ICML 2026?

ICML 2026 has 64 LLM-as-Judge papers, versus 33 the year before.

How many LLM-as-Judge papers are at CVPR 2026?

CVPR 2026 has 5 LLM-as-Judge papers, versus 3 the year before.

Which conference publishes the most LLM-as-Judge papers?

NeurIPS, with 173 papers from 2023 to 2026.