LLM-as-Judge papers, 2023–2026

Key findings

• LLM-as-Judge went from 8 papers in 2023 to 219 in 2026, roughly 27x in three years.
• NeurIPS publishes the most LLM-as-Judge work: 173 papers from 2023 to 2026, 43% of the four-venue total.
• The fastest-growing overlap is with AI Agents: papers tagged with both rose from 1 in 2023 to 78 in 2026.
• Among the 5 topics under Evaluation and Benchmarks, it ranks #2 by 2026 paper count; the fastest grower in the group, LLM Benchmarks, is up 154x versus 27x here.

Representative papers

Spread across years and venues so every edition shows up

• NeurIPS 2026A Free Lunch in LLM Compression: Revisiting Retraining after Pruning
• ICML 2026A Judge-Aware Ranking Framework for Evaluating Large Language Models without Ground Truth
• ICLR 2026Addressing Pitfalls in the Evaluation of Uncertainty Estimation Methods for Natural Language Generation
• CVPR 2026Pixels Dont Lie But Your Detector Might Bootstrapping MLLM-as-a-Judge for
• NeurIPS 2025$\texttt{BetaConform}$: Efficient MAP Estimation of LLM Ensemble Judgment Performance with Prior Transfer
• ICML 2025Active Evaluation Acquisition for Efficient LLM Benchmarking
• ICLR 2025An Empirical Analysis of Uncertainty in Large Language Model Evaluations
• CVPR 2025ASHiTA: Automatic Scene-grounded HIerarchical Task Analysis
• NeurIPS 2024BertaQA: How Much Do Language Models Know About Local Culture?
• ICML 2024CHAI: Clustered Head Attention for Efficient LLM Inference
• ICLR 2024PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization
• NeurIPS 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• ICML 2023SAM operates far from home: eigenvalue regularization as a dynamical phenomenon
• ICLR 2023On the complexity of nonsmooth automatic differentiation
• NeurIPS 2026A Stratified Multi-Rater Evaluation of LLM-Based Virtual Standardized Patients with a Deployed Data-Generation Platform
• ICML 2026Are LLM Evaluators Really Narcissists? Sanity Checking Self-Preference Evaluations
• ICLR 2026AutoMetrics: Approximate Human Judgments with Automatically Generated Evaluators
• CVPR 20263DrawAgent Teaching LLM to Draw in 3D with Early Contrastive
• NeurIPS 2025Adaptive Prediction-Powered AutoEval with Reliability and Efficiency Guarantees
• ICML 2025Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluators

All LLM-as-Judge papers

53 papers, page 1 of 2

1. NeurIPS 2025$\texttt{BetaConform}$: Efficient MAP Estimation of LLM Ensemble Judgment Performance with Prior Transfer
2. NeurIPS 2025ACT as Human: Multimodal Large Language Model Data Annotation with Critical Thinking
3. NeurIPS 2025Adaptive Prediction-Powered AutoEval with Reliability and Efficiency Guarantees
4. NeurIPS 2025Any Large Language Model Can Be a Reliable Judge: Debiasing with a Reasoning-based Bias Detector
5. NeurIPS 2025Ask a Strong LLM Judge when Your Reward Model is Uncertain
6. NeurIPS 2025AutoJudge: Judge Decoding Without Manual Annotation
7. NeurIPS 2025Beyond the Surface: Enhancing LLM-as-a-Judge Alignment with Human via Internal Representations
8. NeurIPS 2025Bohdi: Heterogeneous LLM Fusion with Automatic Data Exploration
9. NeurIPS 2025BrainEC-LLM: Brain Effective Connectivity Estimation by Multiscale Mixing LLM
10. NeurIPS 2025Bridging Human and LLM Judgments: Understanding and Narrowing the Gap
11. NeurIPS 2025Comparison requires valid measurement: Rethinking attack success rate comparisons in AI red teaming
12. NeurIPS 2025Correlation Dimension of Autoregressive Large Language Models
13. NeurIPS 2025Cost-aware LLM-based Online Dataset Annotation
14. NeurIPS 2025Distributional LLM-as-a-Judge
15. NeurIPS 2025Do Automatic Factuality Metrics Measure Factuality? A Critical Evaluation
16. NeurIPS 2025Do Language Models Use Their Depth Efficiently?
17. NeurIPS 2025Enhancing Multilingual LLM Pretraining with Model-Based Data Selection
18. NeurIPS 2025Evaluating LLM-contaminated Crowdsourcing Data Without Ground Truth
19. NeurIPS 2025Flex-Judge: Text-Only Reasoning Unleashes Zero-Shot Multimodal Evaluators
20. NeurIPS 2025From Replication to Redesign: Exploring Pairwise Comparisons for LLM-Based Peer Review
21. NeurIPS 2025GPAS: Accelerating Convergence of LLM Pretraining via Gradient-Preserving Activation Scaling
22. NeurIPS 2025HyGen: Efficient LLM Serving via Elastic Online-Offline Request Co-location
23. NeurIPS 2025KORGym: A Dynamic Game Platform for LLM Reasoning Evaluation
24. NeurIPS 2025Less is More: Improving LLM Alignment via Preference Data Selection
25. NeurIPS 2025Let the LLM Stick to Its Strengths: Learning to Route Economical LLM
26. NeurIPS 2025LiveCodeBench Pro: How Do Olympiad Medalists Judge LLMs in Competitive Programming?
27. NeurIPS 2025Lua-LLM: Learning Unstructured-Sparsity Allocation for Large Language Models
28. NeurIPS 2025Martingale Score: An Unsupervised Metric for Bayesian Rationality in LLM Reasoning
29. NeurIPS 2025Masked Gated Linear Unit
30. NeurIPS 2025MemSim: A Bayesian Simulator for Evaluating Memory of LLM-based Personal Assistants
31. NeurIPS 2025metaTextGrad: Automatically optimizing language model optimizers
32. NeurIPS 2025MODEL SHAPLEY: Find Your Ideal Parameter Player via One Gradient Backpropagation
33. NeurIPS 2025MOOSE-Chem2: Exploring LLM Limits in Fine-Grained Scientific Hypothesis Discovery via Hierarchical Search
34. NeurIPS 2025Multi-Agent Debate for LLM Judges with Adaptive Stability Detection
35. NeurIPS 2025On Evaluating LLM Alignment by Evaluating LLMs as Judges
36. NeurIPS 2025PARALLELPROMPT: Extracting Parallelism from Large Language Model Queries
37. NeurIPS 2025Permissioned LLMs: Enforcing Access Control in Large Language Models
38. NeurIPS 2025Position: Benchmarking is Broken - Don't Let AI be Its Own Judge
39. NeurIPS 2025Predicting Empirical AI Research Outcomes with Language Models
40. NeurIPS 2025QuadEnhancer: Leveraging Quadratic Transformations to Enhance Deep Neural Networks
41. NeurIPS 2025Reparameterized LLM Training via Orthogonal Equivalence Transformation
42. NeurIPS 2025Reverse Engineering Human Preferences with Reinforcement Learning
43. NeurIPS 2025SEC-bench: Automated Benchmarking of LLM Agents on Real-World Software Security Tasks
44. NeurIPS 2025Set-LLM: A Permutation-Invariant LLM
45. NeurIPS 2025SuperGPQA: Scaling LLM Evaluation across 285 Graduate Disciplines
46. NeurIPS 2025The Fragile Truth of Saliency: Improving LLM Input Attribution via Attention Bias Optimization
47. NeurIPS 2025The Leaderboard Illusion
48. NeurIPS 2025The Lighthouse of Language: Enhancing LLM Agents via Critique-Guided Improvement
49. NeurIPS 2025Towards Fully FP8 GEMM LLM Training at Scale
50. NeurIPS 2025Towards Reliable LLM-based Robots Planning via Combined Uncertainty Estimation

FAQ

How many LLM-as-Judge papers were published at NeurIPS, ICLR, ICML and CVPR from 2023 to 2026?

401 in total: 8 in 2023, 48 in 2024, 126 in 2025, 219 in 2026.

How many LLM-as-Judge papers are at NeurIPS 2026?

NeurIPS 2026 has 88 LLM-as-Judge papers, versus 53 the year before.

How many LLM-as-Judge papers are at ICLR 2026?

ICLR 2026 has 62 LLM-as-Judge papers, versus 37 the year before.

How many LLM-as-Judge papers are at ICML 2026?

ICML 2026 has 64 LLM-as-Judge papers, versus 33 the year before.

How many LLM-as-Judge papers are at CVPR 2026?

CVPR 2026 has 5 LLM-as-Judge papers, versus 3 the year before.

Which conference publishes the most LLM-as-Judge papers?

NeurIPS, with 173 papers from 2023 to 2026.