LLM-as-Judge papers, 2023–2026

Key findings

• LLM-as-Judge went from 8 papers in 2023 to 219 in 2026, roughly 27x in three years.
• NeurIPS publishes the most LLM-as-Judge work: 173 papers from 2023 to 2026, 43% of the four-venue total.
• The fastest-growing overlap is with AI Agents: papers tagged with both rose from 1 in 2023 to 78 in 2026.
• Among the 5 topics under Evaluation and Benchmarks, it ranks #2 by 2026 paper count; the fastest grower in the group, LLM Benchmarks, is up 154x versus 27x here.

Representative papers

Spread across years and venues so every edition shows up

• NeurIPS 2026A Free Lunch in LLM Compression: Revisiting Retraining after Pruning
• ICML 2026A Judge-Aware Ranking Framework for Evaluating Large Language Models without Ground Truth
• ICLR 2026Addressing Pitfalls in the Evaluation of Uncertainty Estimation Methods for Natural Language Generation
• CVPR 2026Pixels Dont Lie But Your Detector Might Bootstrapping MLLM-as-a-Judge for
• NeurIPS 2025$\texttt{BetaConform}$: Efficient MAP Estimation of LLM Ensemble Judgment Performance with Prior Transfer
• ICML 2025Active Evaluation Acquisition for Efficient LLM Benchmarking
• ICLR 2025An Empirical Analysis of Uncertainty in Large Language Model Evaluations
• CVPR 2025ASHiTA: Automatic Scene-grounded HIerarchical Task Analysis
• NeurIPS 2024BertaQA: How Much Do Language Models Know About Local Culture?
• ICML 2024CHAI: Clustered Head Attention for Efficient LLM Inference
• ICLR 2024PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization
• NeurIPS 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• ICML 2023SAM operates far from home: eigenvalue regularization as a dynamical phenomenon
• ICLR 2023On the complexity of nonsmooth automatic differentiation
• NeurIPS 2026A Stratified Multi-Rater Evaluation of LLM-Based Virtual Standardized Patients with a Deployed Data-Generation Platform
• ICML 2026Are LLM Evaluators Really Narcissists? Sanity Checking Self-Preference Evaluations
• ICLR 2026AutoMetrics: Approximate Human Judgments with Automatically Generated Evaluators
• CVPR 20263DrawAgent Teaching LLM to Draw in 3D with Early Contrastive
• NeurIPS 2025Adaptive Prediction-Powered AutoEval with Reliability and Efficiency Guarantees
• ICML 2025Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluators

All LLM-as-Judge papers

108 papers, page 2 of 3

51. ICML 2026REAL: Regression-Aware Reinforcement Learning for LLM-as-a-Judge
52. ICML 2026Reasoning Is Not Free: Robust Adaptive Cost-Efficient Routing for LLM-as-a-Judge
53. ICML 2026Reward Modeling from Natural Language Human Feedback
54. ICML 2026RLCracker: Evaluating the Worst-Case Vulnerability of LLM Watermarks with Adaptive RL Attacks
55. ICML 2026SCOPE: Selective Conformal Optimized Pairwise LLM Judging
56. ICML 2026Security--Fidelity Tradeoffs: The Hidden Cost of Prompt Injection Defense
57. ICML 2026SelfJudge: Faster Speculative Decoding via Self-Supervised Judge Verification
58. ICML 2026SpreadsheetArena: Decomposing Preference in LLM Generation of Spreadsheet Workbooks
59. ICML 2026Talk, Judge, Cooperate: Gossip-Driven Indirect Reciprocity in Self-Interested LLM Agents
60. ICML 2026Training-Free Hashing-Based Attention via Binary Principal Components
61. ICML 2026Turning Bias into Bugs: Bandit-Guided Style Manipulation Attacks on LLM Judges
62. ICML 2026UniRRM: Unified Reasoning Reward Models Across Languages and Evaluation Paradigms
63. ICML 2026Who can we trust? LLM-as-a-jury for Comparative Assessment
64. ICML 2026Zero-source LLM Hallucination Detection with Human-like Criteria Probing
65. ICML 2025Accelerating Unbiased LLM Evaluation via Synthetic Feedback
66. ICML 2025Active Evaluation Acquisition for Efficient LLM Benchmarking
67. ICML 2025Agent-as-a-Judge: Evaluate Agents with Agents
68. ICML 2025am-ELO: A Stable Framework for Arena-based LLM Evaluation
69. ICML 2025Are LLMs Prescient? A Continuous Evaluation using Daily News as the Oracle
70. ICML 2025BaWA: Automatic Optimizing Pruning Metric for Large Language Models with Balanced Weight and Activation
71. ICML 2025Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compression
72. ICML 2025Copilot Arena: A Platform for Code LLM Evaluation in the Wild
73. ICML 2025CoSER: Coordinating LLM-Based Persona Simulation of Established Roles
74. ICML 2025Demystifying Singular Defects in Large Language Models
75. ICML 2025Emoji Attack: Enhancing Jailbreak Attacks Against Judge LLM Detection
76. ICML 2025Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluators
77. ICML 2025Exploring Criteria of Loss Reweighting to Enhance LLM Unlearning
78. ICML 2025From Jack of All Trades to Master of One: Specializing LLM-based Autoraters to a Test Set
79. ICML 2025Great Models Think Alike and this Undermines AI Oversight
80. ICML 2025Investigating Non-Transitivity in LLM-as-a-Judge
81. ICML 2025La RoSA: Enhancing LLM Efficiency via Layerwise Rotated Sparse Activation
82. ICML 2025Large Language-Geometry Model: When LLM meets Equivariance
83. ICML 2025Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judge
84. ICML 2025MAGELLAN: Metacognitive predictions of learning progress guide autotelic LLM agents in large goal spaces
85. ICML 2025Memorization Sinks: Isolating Memorization during LLM Training
86. ICML 2025Monte Carlo Tree Search for Comprehensive Exploration in LLM-Based Automatic Heuristic Design
87. ICML 2025Overestimation in LLM Evaluation: A Controlled Large-Scale Study on Data Contamination’s Impact on Machine Translation
88. ICML 2025Position: Don't Use the CLT in LLM Evals With Fewer Than a Few Hundred Datapoints
89. ICML 2025Position: LLM Social Simulations Are a Promising Research Method
90. ICML 2025Position: Principles of Animal Cognition to Improve LLM Evaluations
91. ICML 2025Position: The Most Expensive Part of an LLM *should* be its Training Data
92. ICML 2025Prompt-to-Leaderboard: Prompt-Adaptive LLM Evaluations
93. ICML 2025RZ-NAS: Enhancing LLM-guided Neural Architecture Search via Reflective Zero-Cost Strategy
94. ICML 2025The Emperor's New Clothes in Benchmarking? A Rigorous Examination of Mitigation Strategies for LLM Benchmark Data Contamination
95. ICML 2025Tuning LLM Judge Design Decisions for 1/1000 of the Cost
96. ICML 2025TypyBench: Evaluating LLM Type Inference for Untyped Python Repositories
97. ICML 2025Uncertainty Quantification for LLM-Based Survey Simulations
98. ICML 2024CHAI: Clustered Head Attention for Efficient LLM Inference
99. ICML 2024ExCP: Extreme LLM Checkpoint Compression via Weight-Momentum Joint Shrinking
100. ICML 2024MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark

FAQ

How many LLM-as-Judge papers were published at NeurIPS, ICLR, ICML and CVPR from 2023 to 2026?

401 in total: 8 in 2023, 48 in 2024, 126 in 2025, 219 in 2026.

How many LLM-as-Judge papers are at NeurIPS 2026?

NeurIPS 2026 has 88 LLM-as-Judge papers, versus 53 the year before.

How many LLM-as-Judge papers are at ICLR 2026?

ICLR 2026 has 62 LLM-as-Judge papers, versus 37 the year before.

How many LLM-as-Judge papers are at ICML 2026?

ICML 2026 has 64 LLM-as-Judge papers, versus 33 the year before.

How many LLM-as-Judge papers are at CVPR 2026?

CVPR 2026 has 5 LLM-as-Judge papers, versus 3 the year before.

Which conference publishes the most LLM-as-Judge papers?

NeurIPS, with 173 papers from 2023 to 2026.