LLM-as-Judge papers, 2023–2026
Automatic evaluation
Scope: NeurIPS, ICLR, ICML and CVPR, 2023–2026. 401 papers are tagged with this topic.
Key findings
- LLM-as-Judge went from 8 papers in 2023 to 219 in 2026, roughly 27x in three years.
- NeurIPS publishes the most LLM-as-Judge work: 173 papers from 2023 to 2026, 43% of the four-venue total.
- The fastest-growing overlap is with AI Agents: papers tagged with both rose from 1 in 2023 to 78 in 2026.
- Among the 5 topics under Evaluation and Benchmarks, it ranks #2 by 2026 paper count; the fastest grower in the group, LLM Benchmarks, is up 154x versus 27x here.
Papers per venue
Counts by year and venue
| Year | NeurIPS | ICLR | ICML | CVPR | Total |
|---|---|---|---|---|---|
| 2023 | 5 | 1 | 2 | 0 | 8 |
| 2024 | 27 | 12 | 9 | 0 | 48 |
| 2025 | 53 | 37 | 33 | 3 | 126 |
| 2026 | 88 | 62 | 64 | 5 | 219 |
Representative papers
Spread across years and venues so every edition shows up
- NeurIPS 2026A Free Lunch in LLM Compression: Revisiting Retraining after Pruning
- ICML 2026A Judge-Aware Ranking Framework for Evaluating Large Language Models without Ground Truth
- ICLR 2026Addressing Pitfalls in the Evaluation of Uncertainty Estimation Methods for Natural Language Generation
- CVPR 2026Pixels Dont Lie But Your Detector Might Bootstrapping MLLM-as-a-Judge for
- NeurIPS 2025$\texttt{BetaConform}$: Efficient MAP Estimation of LLM Ensemble Judgment Performance with Prior Transfer
- ICML 2025Active Evaluation Acquisition for Efficient LLM Benchmarking
- ICLR 2025An Empirical Analysis of Uncertainty in Large Language Model Evaluations
- CVPR 2025ASHiTA: Automatic Scene-grounded HIerarchical Task Analysis
- NeurIPS 2024BertaQA: How Much Do Language Models Know About Local Culture?
- ICML 2024CHAI: Clustered Head Attention for Efficient LLM Inference
- ICLR 2024PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization
- NeurIPS 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
- ICML 2023SAM operates far from home: eigenvalue regularization as a dynamical phenomenon
- ICLR 2023On the complexity of nonsmooth automatic differentiation
- NeurIPS 2026A Stratified Multi-Rater Evaluation of LLM-Based Virtual Standardized Patients with a Deployed Data-Generation Platform
- ICML 2026Are LLM Evaluators Really Narcissists? Sanity Checking Self-Preference Evaluations
- ICLR 2026AutoMetrics: Approximate Human Judgments with Automatically Generated Evaluators
- CVPR 20263DrawAgent Teaching LLM to Draw in 3D with Early Contrastive
- NeurIPS 2025Adaptive Prediction-Powered AutoEval with Reliability and Efficiency Guarantees
- ICML 2025Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluators
All LLM-as-Judge papers
219 papers, page 5 of 5
- NeurIPS 2026Strong Teacher Not Needed? On Distillation in LLM Pretraining
- NeurIPS 2026TextSeal: A Localized LLM Watermark for Provenance & Distillation Protection
- NeurIPS 2026The Geometry of Forgetting: Temporal Knowledge Drift as an Independent Axis in LLM Representations
- NeurIPS 2026The Missing Corpus: Infinite Ground Truth for File-Grounded LLM Evaluation
- NeurIPS 2026The Ruler and the Judge: Benchmark-Conditional Evaluation of LLM-as-a-Judge
- NeurIPS 2026Too Smart to Teach: The Articulability Ceiling in Language Models
- NeurIPS 2026Towards Reliable LLM Evaluation: Correcting the Winner’s Curse in Adaptive Benchmarking
- NeurIPS 2026Treat Domain-Specific Languages as Design Variables in LLM Agents
- NeurIPS 2026Trust, but Don’t Verify: Epistemic Blind Spots in LLM Source Evaluation
- NeurIPS 2026TwinRouterBench: Fast Static and Live Dynamic Evaluation for Realistic Agentic LLM Routing
- NeurIPS 2026Valid Best-Model Identification for LLM Evaluation via Low-Rank Factorization
- NeurIPS 2026Verifiable LLM-Guided Focal SMT Solving for Quantified Arrays
- NeurIPS 2026Verified-Source Authority Is Not Generic Sycophancy: Cue-Family Decomposition of LLM Compliance
- NeurIPS 2026VEX-Bench: Benchmarking Verification Complexity of LLM-Generated Misinformation
- NeurIPS 2026What Claims Do LLM Benchmark Scores Support?
- NeurIPS 2026What Should Remain After Forgetting? Rethinking LLM Unlearning as Predictive Posterior Correction
- NeurIPS 2026When Empathy Misses the Goal: A Benchmark for Goal Displacement in LLM Advice
- NeurIPS 2026When Expert Disagreement Hurts: Auditing Prestige-Sensitive Revision in LLM Decision Pipelines
- NeurIPS 2026When LLM Routers Overpay: Strong-Model Over-Selection under Loose Budgets
Ask about this trend from Claude or Cursor
Connect the Paperlist MCP and your assistant can pull the numbers on this page while you draft a survey, pick a direction or plan a submission: papers per year and venue, growth, the fastest-rising sub-themes and representative papers, shaped around your question.
Copy this prompt into your assistant
Use paperlist to show how LLM-as-Judge papers grew at NeurIPS from 2023 to 2026, and which year jumped the most.
Tool the assistant calls: get_topic_stats {"topic": "llm-as-judge"}
Set up Paperlist MCP in two minutes (Claude Code, Cursor, Codex, ChatGPT)
FAQ
- How many LLM-as-Judge papers were published at NeurIPS, ICLR, ICML and CVPR from 2023 to 2026?
- 401 in total: 8 in 2023, 48 in 2024, 126 in 2025, 219 in 2026.
- How many LLM-as-Judge papers are at NeurIPS 2026?
- NeurIPS 2026 has 88 LLM-as-Judge papers, versus 53 the year before.
- How many LLM-as-Judge papers are at ICLR 2026?
- ICLR 2026 has 62 LLM-as-Judge papers, versus 37 the year before.
- How many LLM-as-Judge papers are at ICML 2026?
- ICML 2026 has 64 LLM-as-Judge papers, versus 33 the year before.
- How many LLM-as-Judge papers are at CVPR 2026?
- CVPR 2026 has 5 LLM-as-Judge papers, versus 3 the year before.
- Which conference publishes the most LLM-as-Judge papers?
- NeurIPS, with 173 papers from 2023 to 2026.