LLM-as-Judge papers, 2023–2026

Key findings

• LLM-as-Judge went from 8 papers in 2023 to 219 in 2026, roughly 27x in three years.
• NeurIPS publishes the most LLM-as-Judge work: 173 papers from 2023 to 2026, 43% of the four-venue total.
• The fastest-growing overlap is with AI Agents: papers tagged with both rose from 1 in 2023 to 78 in 2026.
• Among the 5 topics under Evaluation and Benchmarks, it ranks #2 by 2026 paper count; the fastest grower in the group, LLM Benchmarks, is up 154x versus 27x here.

Representative papers

Spread across years and venues so every edition shows up

• NeurIPS 2026A Free Lunch in LLM Compression: Revisiting Retraining after Pruning
• ICML 2026A Judge-Aware Ranking Framework for Evaluating Large Language Models without Ground Truth
• ICLR 2026Addressing Pitfalls in the Evaluation of Uncertainty Estimation Methods for Natural Language Generation
• CVPR 2026Pixels Dont Lie But Your Detector Might Bootstrapping MLLM-as-a-Judge for
• NeurIPS 2025$\texttt{BetaConform}$: Efficient MAP Estimation of LLM Ensemble Judgment Performance with Prior Transfer
• ICML 2025Active Evaluation Acquisition for Efficient LLM Benchmarking
• ICLR 2025An Empirical Analysis of Uncertainty in Large Language Model Evaluations
• CVPR 2025ASHiTA: Automatic Scene-grounded HIerarchical Task Analysis
• NeurIPS 2024BertaQA: How Much Do Language Models Know About Local Culture?
• ICML 2024CHAI: Clustered Head Attention for Efficient LLM Inference
• ICLR 2024PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization
• NeurIPS 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• ICML 2023SAM operates far from home: eigenvalue regularization as a dynamical phenomenon
• ICLR 2023On the complexity of nonsmooth automatic differentiation
• NeurIPS 2026A Stratified Multi-Rater Evaluation of LLM-Based Virtual Standardized Patients with a Deployed Data-Generation Platform
• ICML 2026Are LLM Evaluators Really Narcissists? Sanity Checking Self-Preference Evaluations
• ICLR 2026AutoMetrics: Approximate Human Judgments with Automatically Generated Evaluators
• CVPR 20263DrawAgent Teaching LLM to Draw in 3D with Early Contrastive
• NeurIPS 2025Adaptive Prediction-Powered AutoEval with Reliability and Efficiency Guarantees
• ICML 2025Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluators

All LLM-as-Judge papers

108 papers, page 1 of 3

1. ICML 2026A Coin Flip for Safety: LLM Judges Fail to Reliably Measure Adversarial Robustness
2. ICML 2026A Judge-Aware Ranking Framework for Evaluating Large Language Models without Ground Truth
3. ICML 2026Adaptive Contracts for Cost-Effective AI Delegation
4. ICML 2026Are LLM Evaluators Really Narcissists? Sanity Checking Self-Preference Evaluations
5. ICML 2026Automatic Construction of Clinical Scoring Systems with LLM Agents
6. ICML 2026Beyond Drift: Stabilizing Subjective LLM Evaluation with Information-Theoretic Rubrics
7. ICML 2026Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for Open-Ended LLM Reasoning
8. ICML 2026CARE: Confounder-Aware Aggregation for Reliable LLM Evaluation
9. ICML 2026Controlled LLM Training on Spectral Sphere
10. ICML 2026Cutting LLM Evaluation Costs with SySRs: A Bandit Algorithm That Provably Exploits Model Similarity
11. ICML 2026D-Judge: Disrupting Multi-Turn Jailbreaks using Semantics-Preserving Output Rewriting
12. ICML 2026Demystifying LLM-as-a-Judge: Analytically Tractable Model for Inference-Time Scaling
13. ICML 2026Dependence-Aware Label Aggregation for LLM-as-a-Judge via Ising Models
14. ICML 2026Diagnosing the Reliability of LLM-as-a-Judge via Item Response Theory
15. ICML 2026Distribution-Calibrated Inference Time Compute for Thinking LLM-as-a-Judge
16. ICML 2026Distributional Open-Ended Evaluation of LLM Cultural Value Alignment Based on Value Codebook
17. ICML 2026Does Reinforcement Fine-Tuning Improve Generalization of LLM Agents? An Empirical Study
18. ICML 2026Easier to Judge than to Find: Predicting In-Context Learning Success for Demonstration Selection
19. ICML 2026Efficient Distributed MLLM Training with Cornstarch
20. ICML 2026Efficient Inference for Noisy LLM-as-a-Judge Evaluation
21. ICML 2026Enhancing LLM Training via Spectral Clipping
22. ICML 2026Evaluating Language Models in Realistic Conversational Contexts
23. ICML 2026ExCyTIn-Bench: Evaluating LLM agents on Cyber Threat Investigation
24. ICML 2026FairJudge : An Adaptive, Debiased, and Consistent LLM-as-a-Judge
25. ICML 2026FiRE: Fine-grained Ranking Evaluation for Machine Translation
26. ICML 2026Game-Theoretic Co-Evolution for LLM-Based Heuristic Discovery
27. ICML 2026How to Correctly Report LLM-as-a-Judge Evaluations
28. ICML 2026InnoEval: On Research Idea Evaluation as a Knowledge-Grounded, Multi-Perspective Reasoning Problem
29. ICML 2026JADE: Expert-Grounded Dynamic Evaluation for Open-Ended Professional Tasks
30. ICML 2026Judging What We Cannot Solve: A Consequence-Based Approach for Oracle-Free Evaluation of Research-Level Math
31. ICML 2026Learn-to-learn on Arbitrary Textual Conditioning: A Hypernetwork-Driven Meta-gated LLM
32. ICML 2026LitReview Arena: Evaluating Literature Review Agents with Battle-style Peer Review Platform
33. ICML 2026LLM Priors for ERM over Programs
34. ICML 2026LLM Watermark Evasion via Bias Inversion
35. ICML 2026LLM-based Embeddings: Attention Values Encode Sentence Semantics Better Than Hidden States
36. ICML 2026Margin-Adaptive Confidence Ranking for Reliable LLM Judgement
37. ICML 2026Mitigating Perceptual Judgment Bias in Multimodal LLM-as-a-Judge via Perceptual Perturbation and Reward Modeling
38. ICML 2026Mitigating Translationese Bias in Multilingual LLM-as-a-Judge via Disentangled Information Bottleneck
39. ICML 2026No Free Lunch: Non-Asymptotic Analysis of Prediction-Powered Inference
40. ICML 2026Nonparametric LLM Evaluation from Preference Data
41. ICML 2026On the Limits of LLM Adaptability: Impact of Model-Internalized Priors on Annotation Task Performance
42. ICML 2026Order Matters: Unveiling the Hidden Impact of Macro Placement Sequences via Proxy-Guided LLM Evolution
43. ICML 2026PPDL: LLM-Based Flows as Probabilistic Programs
44. ICML 2026PPT-Eval: A Benchmark for Computer-Use Agents on PowerPoint Tasks
45. ICML 2026Prediction-Powered Risk Monitoring of Deployed Models for Detecting Harmful Distribution Shifts
46. ICML 2026Pressure Reveals Character: Behavioural Alignment Evaluation at Depth
47. ICML 2026ProbeLLM: Automating Principled Diagnosis of LLM Failures
48. ICML 2026ProtocolBench: Which LLM MultiAgent Protocol to Choose?
49. ICML 2026Quantifying Biases in LLM-as-a-Judge Evaluations
50. ICML 2026Quartet II: Accurate LLM Pre-Training in NVFP4 by Improved Unbiased Gradient Estimation

FAQ

How many LLM-as-Judge papers were published at NeurIPS, ICLR, ICML and CVPR from 2023 to 2026?

401 in total: 8 in 2023, 48 in 2024, 126 in 2025, 219 in 2026.

How many LLM-as-Judge papers are at NeurIPS 2026?

NeurIPS 2026 has 88 LLM-as-Judge papers, versus 53 the year before.

How many LLM-as-Judge papers are at ICLR 2026?

ICLR 2026 has 62 LLM-as-Judge papers, versus 37 the year before.

How many LLM-as-Judge papers are at ICML 2026?

ICML 2026 has 64 LLM-as-Judge papers, versus 33 the year before.

How many LLM-as-Judge papers are at CVPR 2026?

CVPR 2026 has 5 LLM-as-Judge papers, versus 3 the year before.

Which conference publishes the most LLM-as-Judge papers?

NeurIPS, with 173 papers from 2023 to 2026.