LLM-as-Judge papers, 2023–2026

Key findings

• LLM-as-Judge went from 8 papers in 2023 to 219 in 2026, roughly 27x in three years.
• NeurIPS publishes the most LLM-as-Judge work: 173 papers from 2023 to 2026, 43% of the four-venue total.
• The fastest-growing overlap is with AI Agents: papers tagged with both rose from 1 in 2023 to 78 in 2026.
• Among the 5 topics under Evaluation and Benchmarks, it ranks #2 by 2026 paper count; the fastest grower in the group, LLM Benchmarks, is up 154x versus 27x here.

Representative papers

Spread across years and venues so every edition shows up

• NeurIPS 2026A Free Lunch in LLM Compression: Revisiting Retraining after Pruning
• ICML 2026A Judge-Aware Ranking Framework for Evaluating Large Language Models without Ground Truth
• ICLR 2026Addressing Pitfalls in the Evaluation of Uncertainty Estimation Methods for Natural Language Generation
• CVPR 2026Pixels Dont Lie But Your Detector Might Bootstrapping MLLM-as-a-Judge for
• NeurIPS 2025$\texttt{BetaConform}$: Efficient MAP Estimation of LLM Ensemble Judgment Performance with Prior Transfer
• ICML 2025Active Evaluation Acquisition for Efficient LLM Benchmarking
• ICLR 2025An Empirical Analysis of Uncertainty in Large Language Model Evaluations
• CVPR 2025ASHiTA: Automatic Scene-grounded HIerarchical Task Analysis
• NeurIPS 2024BertaQA: How Much Do Language Models Know About Local Culture?
• ICML 2024CHAI: Clustered Head Attention for Efficient LLM Inference
• ICLR 2024PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization
• NeurIPS 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• ICML 2023SAM operates far from home: eigenvalue regularization as a dynamical phenomenon
• ICLR 2023On the complexity of nonsmooth automatic differentiation
• NeurIPS 2026A Stratified Multi-Rater Evaluation of LLM-Based Virtual Standardized Patients with a Deployed Data-Generation Platform
• ICML 2026Are LLM Evaluators Really Narcissists? Sanity Checking Self-Preference Evaluations
• ICLR 2026AutoMetrics: Approximate Human Judgments with Automatically Generated Evaluators
• CVPR 20263DrawAgent Teaching LLM to Draw in 3D with Early Contrastive
• NeurIPS 2025Adaptive Prediction-Powered AutoEval with Reliability and Efficiency Guarantees
• ICML 2025Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluators

All LLM-as-Judge papers

401 papers, page 4 of 9

151. NeurIPS 2026Capturing LLM Capabilities via Evidence-Calibrated Query Clustering
152. NeurIPS 2026CogBench: Evaluating Cognitive-Level Control in LLM Question Generation
153. NeurIPS 2026CompJudge: Fine-Grained Comparative Evaluation using Multimodal LLM for Subject-Driven Generation
154. NeurIPS 2026Consolidating Rewarded Perturbations for LLM Post-Training
155. NeurIPS 2026ContractBench: Can LLM Agents Preserve Observation Contracts?
156. NeurIPS 2026CUDABeaver: Benchmarking LLM-Based Automated CUDA Debugging
157. NeurIPS 2026CyBiasBench: Benchmarking Bias in LLM Agents for Cyber-Attack Scenarios
158. NeurIPS 2026Decentralized Aggregation of LLM Predictions via Wagering Mechanisms
159. NeurIPS 2026Demystifying Numerical Errors in LLM Inference: Achieving Reproducible Inference for Mission-Critical Tasks with HEAL
160. NeurIPS 2026DICE: Decoupling Capability from Intervention Necessity in LLM Tutoring
161. NeurIPS 2026Efficient Evaluation of LLM Performance with Statistical Guarantees
162. NeurIPS 2026Every Bit, Everywhere, All At Once: A Binomial Multibit LLM Watermark
163. NeurIPS 2026Forced Deferral: Manipulating Routing Decisions in Multimodal LLM Cascades
164. NeurIPS 2026Forced Orders: What LLM Leaderboards Hide About Model Comparisons
165. NeurIPS 2026From Infrastructure to Interface, the AI Value Chain Drives LLM Homogenization
166. NeurIPS 2026From Uncertain Judgments to Calibrated Rankings: Conformal Elo Estimation for LLM Evaluation
167. NeurIPS 2026Hidden Measurement Error in LLM Pipelines Distorts Annotation, Evaluation, and Benchmarking
168. NeurIPS 2026HireWatch: Evaluating LLM Compliance with U.S. Employment Law Under Contextual Pressure
169. NeurIPS 2026How Does Personalized Memory Shape LLM Behavior? Benchmarking Rational Preference Utilization in Personalized Assistants
170. NeurIPS 2026Imagine3D-LLM: Teaching MLLMs to Imagine 3D Scenes Before Answering
171. NeurIPS 2026Improving LLM Final Representations with Inter-Layer Geometry
172. NeurIPS 2026Instrumental Choices: Measuring the Propensity of LLM Agents to Pursue Instrumental Behaviors
173. NeurIPS 2026Jacobian Scopes: A Unified Geometric Framework for Token-Level LLM Attributions
174. NeurIPS 2026JuICE: A Benchmark for Evaluating LLM-Judge in Identifying Cultural Errors
175. NeurIPS 2026L2-Bench: An Evaluation Benchmark for Measuring LLM Capabilities in Second Language Education
176. NeurIPS 2026Learning What to Forget: Improving LLM Unlearning via Learned Token-Level Importance
177. NeurIPS 2026LensCT: Fine-Grained AI-Involved Text Detection via Temporal-Hierarchical Tomograms of LLM Internals
178. NeurIPS 2026LLM Judge Validation Under Sparse Overlap: From Inference to Design
179. NeurIPS 2026LLM-Auction: Generative Auction towards LLM-Native Advertising
180. NeurIPS 2026MEMAUDIT: An Exact Package-Oracle Evaluation Protocol for Budgeted Long-Term LLM Memory Writing
181. NeurIPS 2026Model-Adaptive Tool Necessity Reveals the Knowing-Doing Gap in LLM Tool Use
182. NeurIPS 2026Models Recall What They Violate: Constraint Adherence in Multi-Turn LLM Ideation
183. NeurIPS 2026NeuroFaith: Evaluating Mechanistic Faithfulness of LLM Free Text Self-Explanation at the Concept Level
184. NeurIPS 2026Not All Proofs Are Equal: Evaluating LLM Proof Quality Beyond Correctness
185. NeurIPS 2026NuMuon: Nuclear-Norm-Constrained Muon for Compressible LLM Training
186. NeurIPS 2026NyoomFloat12: Accelerating LLM Inference via Lossless 12-bit Weight Compression
187. NeurIPS 2026Pacing Branch Parallelism in LLM Serving
188. NeurIPS 2026RA$^2$: Retain-Anchored Attraction Preserves Forget-Adjacent Utility in LLM Unlearning
189. NeurIPS 2026RAOP: Step-Level Resource Orchestration for LLM Agents across Edge and Cloud
190. NeurIPS 2026RelAgent: LLM Agents as Data Scientists for Relational Learning
191. NeurIPS 2026Rethinking Latency Denial-of-Service: Attack the LLM Serving Framework, Not the Model
192. NeurIPS 2026Rethinking Rubric Generation for Improving LLM Judge and Reward Modeling for Open-ended Tasks
193. NeurIPS 2026Salvation Lies Within: Proactive Prefix Re-forming for LLM-based Tagging
194. NeurIPS 2026SCDBench: A Benchmark for LLM-Based Smart Contract Decompilers
195. NeurIPS 2026SchemeArena: Factorized Stress Testing of Scheming in LLM Agents
196. NeurIPS 2026SCTI: Self-Calibrated Trident Identification of Black-Box LLM Watermarks
197. NeurIPS 2026Self-EvolveRec: Self-Evolving Recommender Systems with LLM-based Directional Feedback
198. NeurIPS 2026SkillMaster: Toward Autonomous Skill Mastery in LLM Agents
199. NeurIPS 2026Social Interaction Breaks Replicate Independence in Controlled-Clone LLM Agents
200. NeurIPS 2026Step-wise Rubric Rewards for LLM Reasoning

FAQ

How many LLM-as-Judge papers were published at NeurIPS, ICLR, ICML and CVPR from 2023 to 2026?

401 in total: 8 in 2023, 48 in 2024, 126 in 2025, 219 in 2026.

How many LLM-as-Judge papers are at NeurIPS 2026?

NeurIPS 2026 has 88 LLM-as-Judge papers, versus 53 the year before.

How many LLM-as-Judge papers are at ICLR 2026?

ICLR 2026 has 62 LLM-as-Judge papers, versus 37 the year before.

How many LLM-as-Judge papers are at ICML 2026?

ICML 2026 has 64 LLM-as-Judge papers, versus 33 the year before.

How many LLM-as-Judge papers are at CVPR 2026?

CVPR 2026 has 5 LLM-as-Judge papers, versus 3 the year before.

Which conference publishes the most LLM-as-Judge papers?

NeurIPS, with 173 papers from 2023 to 2026.