LLM-as-Judge papers, 2023–2026

Key findings

• LLM-as-Judge went from 8 papers in 2023 to 219 in 2026, roughly 27x in three years.
• NeurIPS publishes the most LLM-as-Judge work: 173 papers from 2023 to 2026, 43% of the four-venue total.
• The fastest-growing overlap is with AI Agents: papers tagged with both rose from 1 in 2023 to 78 in 2026.
• Among the 5 topics under Evaluation and Benchmarks, it ranks #2 by 2026 paper count; the fastest grower in the group, LLM Benchmarks, is up 154x versus 27x here.

Representative papers

Spread across years and venues so every edition shows up

• NeurIPS 2026A Free Lunch in LLM Compression: Revisiting Retraining after Pruning
• ICML 2026A Judge-Aware Ranking Framework for Evaluating Large Language Models without Ground Truth
• ICLR 2026Addressing Pitfalls in the Evaluation of Uncertainty Estimation Methods for Natural Language Generation
• CVPR 2026Pixels Dont Lie But Your Detector Might Bootstrapping MLLM-as-a-Judge for
• NeurIPS 2025$\texttt{BetaConform}$: Efficient MAP Estimation of LLM Ensemble Judgment Performance with Prior Transfer
• ICML 2025Active Evaluation Acquisition for Efficient LLM Benchmarking
• ICLR 2025An Empirical Analysis of Uncertainty in Large Language Model Evaluations
• CVPR 2025ASHiTA: Automatic Scene-grounded HIerarchical Task Analysis
• NeurIPS 2024BertaQA: How Much Do Language Models Know About Local Culture?
• ICML 2024CHAI: Clustered Head Attention for Efficient LLM Inference
• ICLR 2024PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization
• NeurIPS 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• ICML 2023SAM operates far from home: eigenvalue regularization as a dynamical phenomenon
• ICLR 2023On the complexity of nonsmooth automatic differentiation
• NeurIPS 2026A Stratified Multi-Rater Evaluation of LLM-Based Virtual Standardized Patients with a Deployed Data-Generation Platform
• ICML 2026Are LLM Evaluators Really Narcissists? Sanity Checking Self-Preference Evaluations
• ICLR 2026AutoMetrics: Approximate Human Judgments with Automatically Generated Evaluators
• CVPR 20263DrawAgent Teaching LLM to Draw in 3D with Early Contrastive
• NeurIPS 2025Adaptive Prediction-Powered AutoEval with Reliability and Efficiency Guarantees
• ICML 2025Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluators

All LLM-as-Judge papers

401 papers, page 2 of 9

51. ICLR 2026Rethinking LLM-as-a-Judge: Representation-as-a-Judge with Small Language Models via Semantic Capacity Asymmetry
52. ICLR 2026RouterArena: An Open Platform for Comprehensive Comparison of LLM Routers
53. ICLR 2026SCI-Verifier: Scientific Verifier with Thinking
54. ICLR 2026Semantic Voting: A Self-Evaluation-Free Approach for Efficient LLM Self-Improvement on Unverifiable Open-ended Tasks
55. ICLR 2026SK2Decompile: LLM-based Two-Phase Binary Decompilation from Skeleton to Skin
56. ICLR 2026SocialHarmBench: Revealing LLM Vulnerabilities to Socially Harmful Requests
57. ICLR 2026SparseEval: Efficient Evaluation of Large Language Models by Sparse Optimization
58. ICLR 2026STAR: Strategy-driven Automatic Jailbreak Red-teaming For Large Language Model
59. ICLR 2026Talk, Evaluate, Diagnose: User-aware Agent Evaluation with Automated Error Analysis
60. ICLR 2026The Ideation-Execution Gap: Execution Outcomes of LLM-Generated versus Human Research Ideas
61. ICLR 2026The Unseen Frontier: Pushing the Limits of LLM Sparsity with Surrogate-Free ADMM
62. ICLR 2026Towards Reliable Benchmarking: A Contamination Free, Controllable Evaluation Framework for Multi-step LLM Function Calling
63. ICLR 2026TrustJudge: Inconsistencies of LLM-as-a-Judge and How to Alleviate Them
64. ICLR 2026Unlocking Full Efficiency of Token Filtering in Large Language Model Training
65. ICLR 2026Watermarking Diffusion Language Models
66. ICLR 2026WebDevJudge: Evaluating (M)LLMs as Critiques for Web Development Quality
67. ICLR 2026Your Agent May Misevolve: Emergent Risks in Self-evolving LLM Agents
68. ICML 2026A Coin Flip for Safety: LLM Judges Fail to Reliably Measure Adversarial Robustness
69. ICML 2026A Judge-Aware Ranking Framework for Evaluating Large Language Models without Ground Truth
70. ICML 2026Adaptive Contracts for Cost-Effective AI Delegation
71. ICML 2026Are LLM Evaluators Really Narcissists? Sanity Checking Self-Preference Evaluations
72. ICML 2026Automatic Construction of Clinical Scoring Systems with LLM Agents
73. ICML 2026Beyond Drift: Stabilizing Subjective LLM Evaluation with Information-Theoretic Rubrics
74. ICML 2026Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for Open-Ended LLM Reasoning
75. ICML 2026CARE: Confounder-Aware Aggregation for Reliable LLM Evaluation
76. ICML 2026Controlled LLM Training on Spectral Sphere
77. ICML 2026Cutting LLM Evaluation Costs with SySRs: A Bandit Algorithm That Provably Exploits Model Similarity
78. ICML 2026D-Judge: Disrupting Multi-Turn Jailbreaks using Semantics-Preserving Output Rewriting
79. ICML 2026Demystifying LLM-as-a-Judge: Analytically Tractable Model for Inference-Time Scaling
80. ICML 2026Dependence-Aware Label Aggregation for LLM-as-a-Judge via Ising Models
81. ICML 2026Diagnosing the Reliability of LLM-as-a-Judge via Item Response Theory
82. ICML 2026Distribution-Calibrated Inference Time Compute for Thinking LLM-as-a-Judge
83. ICML 2026Distributional Open-Ended Evaluation of LLM Cultural Value Alignment Based on Value Codebook
84. ICML 2026Does Reinforcement Fine-Tuning Improve Generalization of LLM Agents? An Empirical Study
85. ICML 2026Easier to Judge than to Find: Predicting In-Context Learning Success for Demonstration Selection
86. ICML 2026Efficient Distributed MLLM Training with Cornstarch
87. ICML 2026Efficient Inference for Noisy LLM-as-a-Judge Evaluation
88. ICML 2026Enhancing LLM Training via Spectral Clipping
89. ICML 2026Evaluating Language Models in Realistic Conversational Contexts
90. ICML 2026ExCyTIn-Bench: Evaluating LLM agents on Cyber Threat Investigation
91. ICML 2026FairJudge : An Adaptive, Debiased, and Consistent LLM-as-a-Judge
92. ICML 2026FiRE: Fine-grained Ranking Evaluation for Machine Translation
93. ICML 2026Game-Theoretic Co-Evolution for LLM-Based Heuristic Discovery
94. ICML 2026How to Correctly Report LLM-as-a-Judge Evaluations
95. ICML 2026InnoEval: On Research Idea Evaluation as a Knowledge-Grounded, Multi-Perspective Reasoning Problem
96. ICML 2026JADE: Expert-Grounded Dynamic Evaluation for Open-Ended Professional Tasks
97. ICML 2026Judging What We Cannot Solve: A Consequence-Based Approach for Oracle-Free Evaluation of Research-Level Math
98. ICML 2026Learn-to-learn on Arbitrary Textual Conditioning: A Hypernetwork-Driven Meta-gated LLM
99. ICML 2026LitReview Arena: Evaluating Literature Review Agents with Battle-style Peer Review Platform
100. ICML 2026LLM Priors for ERM over Programs

FAQ

How many LLM-as-Judge papers were published at NeurIPS, ICLR, ICML and CVPR from 2023 to 2026?

401 in total: 8 in 2023, 48 in 2024, 126 in 2025, 219 in 2026.

How many LLM-as-Judge papers are at NeurIPS 2026?

NeurIPS 2026 has 88 LLM-as-Judge papers, versus 53 the year before.

How many LLM-as-Judge papers are at ICLR 2026?

ICLR 2026 has 62 LLM-as-Judge papers, versus 37 the year before.

How many LLM-as-Judge papers are at ICML 2026?

ICML 2026 has 64 LLM-as-Judge papers, versus 33 the year before.

How many LLM-as-Judge papers are at CVPR 2026?

CVPR 2026 has 5 LLM-as-Judge papers, versus 3 the year before.

Which conference publishes the most LLM-as-Judge papers?

NeurIPS, with 173 papers from 2023 to 2026.