LLM-as-Judge papers, 2023–2026

Key findings

• LLM-as-Judge went from 8 papers in 2023 to 219 in 2026, roughly 27x in three years.
• NeurIPS publishes the most LLM-as-Judge work: 173 papers from 2023 to 2026, 43% of the four-venue total.
• The fastest-growing overlap is with AI Agents: papers tagged with both rose from 1 in 2023 to 78 in 2026.
• Among the 5 topics under Evaluation and Benchmarks, it ranks #2 by 2026 paper count; the fastest grower in the group, LLM Benchmarks, is up 154x versus 27x here.

Representative papers

Spread across years and venues so every edition shows up

• NeurIPS 2026A Free Lunch in LLM Compression: Revisiting Retraining after Pruning
• ICML 2026A Judge-Aware Ranking Framework for Evaluating Large Language Models without Ground Truth
• ICLR 2026Addressing Pitfalls in the Evaluation of Uncertainty Estimation Methods for Natural Language Generation
• CVPR 2026Pixels Dont Lie But Your Detector Might Bootstrapping MLLM-as-a-Judge for
• NeurIPS 2025$\texttt{BetaConform}$: Efficient MAP Estimation of LLM Ensemble Judgment Performance with Prior Transfer
• ICML 2025Active Evaluation Acquisition for Efficient LLM Benchmarking
• ICLR 2025An Empirical Analysis of Uncertainty in Large Language Model Evaluations
• CVPR 2025ASHiTA: Automatic Scene-grounded HIerarchical Task Analysis
• NeurIPS 2024BertaQA: How Much Do Language Models Know About Local Culture?
• ICML 2024CHAI: Clustered Head Attention for Efficient LLM Inference
• ICLR 2024PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization
• NeurIPS 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• ICML 2023SAM operates far from home: eigenvalue regularization as a dynamical phenomenon
• ICLR 2023On the complexity of nonsmooth automatic differentiation
• NeurIPS 2026A Stratified Multi-Rater Evaluation of LLM-Based Virtual Standardized Patients with a Deployed Data-Generation Platform
• ICML 2026Are LLM Evaluators Really Narcissists? Sanity Checking Self-Preference Evaluations
• ICLR 2026AutoMetrics: Approximate Human Judgments with Automatically Generated Evaluators
• CVPR 20263DrawAgent Teaching LLM to Draw in 3D with Early Contrastive
• NeurIPS 2025Adaptive Prediction-Powered AutoEval with Reliability and Efficiency Guarantees
• ICML 2025Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluators

All LLM-as-Judge papers

173 papers, page 2 of 4

51. NeurIPS 2026Models Recall What They Violate: Constraint Adherence in Multi-Turn LLM Ideation
52. NeurIPS 2026NeuroFaith: Evaluating Mechanistic Faithfulness of LLM Free Text Self-Explanation at the Concept Level
53. NeurIPS 2026Not All Proofs Are Equal: Evaluating LLM Proof Quality Beyond Correctness
54. NeurIPS 2026NuMuon: Nuclear-Norm-Constrained Muon for Compressible LLM Training
55. NeurIPS 2026NyoomFloat12: Accelerating LLM Inference via Lossless 12-bit Weight Compression
56. NeurIPS 2026Pacing Branch Parallelism in LLM Serving
57. NeurIPS 2026RA$^2$: Retain-Anchored Attraction Preserves Forget-Adjacent Utility in LLM Unlearning
58. NeurIPS 2026RAOP: Step-Level Resource Orchestration for LLM Agents across Edge and Cloud
59. NeurIPS 2026RelAgent: LLM Agents as Data Scientists for Relational Learning
60. NeurIPS 2026Rethinking Latency Denial-of-Service: Attack the LLM Serving Framework, Not the Model
61. NeurIPS 2026Rethinking Rubric Generation for Improving LLM Judge and Reward Modeling for Open-ended Tasks
62. NeurIPS 2026Salvation Lies Within: Proactive Prefix Re-forming for LLM-based Tagging
63. NeurIPS 2026SCDBench: A Benchmark for LLM-Based Smart Contract Decompilers
64. NeurIPS 2026SchemeArena: Factorized Stress Testing of Scheming in LLM Agents
65. NeurIPS 2026SCTI: Self-Calibrated Trident Identification of Black-Box LLM Watermarks
66. NeurIPS 2026Self-EvolveRec: Self-Evolving Recommender Systems with LLM-based Directional Feedback
67. NeurIPS 2026SkillMaster: Toward Autonomous Skill Mastery in LLM Agents
68. NeurIPS 2026Social Interaction Breaks Replicate Independence in Controlled-Clone LLM Agents
69. NeurIPS 2026Step-wise Rubric Rewards for LLM Reasoning
70. NeurIPS 2026Strong Teacher Not Needed? On Distillation in LLM Pretraining
71. NeurIPS 2026TextSeal: A Localized LLM Watermark for Provenance & Distillation Protection
72. NeurIPS 2026The Geometry of Forgetting: Temporal Knowledge Drift as an Independent Axis in LLM Representations
73. NeurIPS 2026The Missing Corpus: Infinite Ground Truth for File-Grounded LLM Evaluation
74. NeurIPS 2026The Ruler and the Judge: Benchmark-Conditional Evaluation of LLM-as-a-Judge
75. NeurIPS 2026Too Smart to Teach: The Articulability Ceiling in Language Models
76. NeurIPS 2026Towards Reliable LLM Evaluation: Correcting the Winner’s Curse in Adaptive Benchmarking
77. NeurIPS 2026Treat Domain-Specific Languages as Design Variables in LLM Agents
78. NeurIPS 2026Trust, but Don’t Verify: Epistemic Blind Spots in LLM Source Evaluation
79. NeurIPS 2026TwinRouterBench: Fast Static and Live Dynamic Evaluation for Realistic Agentic LLM Routing
80. NeurIPS 2026Valid Best-Model Identification for LLM Evaluation via Low-Rank Factorization
81. NeurIPS 2026Verifiable LLM-Guided Focal SMT Solving for Quantified Arrays
82. NeurIPS 2026Verified-Source Authority Is Not Generic Sycophancy: Cue-Family Decomposition of LLM Compliance
83. NeurIPS 2026VEX-Bench: Benchmarking Verification Complexity of LLM-Generated Misinformation
84. NeurIPS 2026What Claims Do LLM Benchmark Scores Support?
85. NeurIPS 2026What Should Remain After Forgetting? Rethinking LLM Unlearning as Predictive Posterior Correction
86. NeurIPS 2026When Empathy Misses the Goal: A Benchmark for Goal Displacement in LLM Advice
87. NeurIPS 2026When Expert Disagreement Hurts: Auditing Prestige-Sensitive Revision in LLM Decision Pipelines
88. NeurIPS 2026When LLM Routers Overpay: Strong-Model Over-Selection under Loose Budgets
89. NeurIPS 2025$\texttt{BetaConform}$: Efficient MAP Estimation of LLM Ensemble Judgment Performance with Prior Transfer
90. NeurIPS 2025ACT as Human: Multimodal Large Language Model Data Annotation with Critical Thinking
91. NeurIPS 2025Adaptive Prediction-Powered AutoEval with Reliability and Efficiency Guarantees
92. NeurIPS 2025Any Large Language Model Can Be a Reliable Judge: Debiasing with a Reasoning-based Bias Detector
93. NeurIPS 2025Ask a Strong LLM Judge when Your Reward Model is Uncertain
94. NeurIPS 2025AutoJudge: Judge Decoding Without Manual Annotation
95. NeurIPS 2025Beyond the Surface: Enhancing LLM-as-a-Judge Alignment with Human via Internal Representations
96. NeurIPS 2025Bohdi: Heterogeneous LLM Fusion with Automatic Data Exploration
97. NeurIPS 2025BrainEC-LLM: Brain Effective Connectivity Estimation by Multiscale Mixing LLM
98. NeurIPS 2025Bridging Human and LLM Judgments: Understanding and Narrowing the Gap
99. NeurIPS 2025Comparison requires valid measurement: Rethinking attack success rate comparisons in AI red teaming
100. NeurIPS 2025Correlation Dimension of Autoregressive Large Language Models

FAQ

How many LLM-as-Judge papers were published at NeurIPS, ICLR, ICML and CVPR from 2023 to 2026?

401 in total: 8 in 2023, 48 in 2024, 126 in 2025, 219 in 2026.

How many LLM-as-Judge papers are at NeurIPS 2026?

NeurIPS 2026 has 88 LLM-as-Judge papers, versus 53 the year before.

How many LLM-as-Judge papers are at ICLR 2026?

ICLR 2026 has 62 LLM-as-Judge papers, versus 37 the year before.

How many LLM-as-Judge papers are at ICML 2026?

ICML 2026 has 64 LLM-as-Judge papers, versus 33 the year before.

How many LLM-as-Judge papers are at CVPR 2026?

CVPR 2026 has 5 LLM-as-Judge papers, versus 3 the year before.

Which conference publishes the most LLM-as-Judge papers?

NeurIPS, with 173 papers from 2023 to 2026.