Rethinking Rubric Generation for Improving LLM Judge and Reward Modeling for Open-ended Tasks

Abstract

Recently, rubrics have been used to guide LLM judges in capturing nuanced, multi-dimensional human preferences, and have further been extended as reward signals for reinforcement fine-tuning (RFT). However, open-ended generation typically lacks a unique canonical target, and its evaluation rubrics are latent rather than directly observed. This makes rubric generation under-determined and difficult to control: rubrics often lack coverage, conflate dimensions, misalign preference direction, and contain highly correlated criteria, degrading judge accuracy and producing suboptimal rewards during RFT. We propose RRD, a practical framework for rubric refinement built on a recursive decompose–filter cycle. RRD decomposes coarse rubrics into fine-grained, discriminative criteria, expanding coverage while sharpening separation between responses. A complementary filtering mechanism removes misaligned and redundant rubrics, and a correlation-aware weighting scheme to prevent over-representing highly correlated criteria, yielding rubric sets that are informative, comprehensive, and non-redundant. Empirically, RRD delivers large, consistent gains across all evaluation and training: it improves preference-judgment accuracy on JudgeBench and PPE for both GPT-4o and Llama3.1-405B judges, achieving top performance in all settings with up to +17.7 points on JudgeBench. When used as the reward source for RFT on WildChat, it yields substantially stronger and more stable learning signals, boosting reward by up to 160% (Qwen3-4B) and 60% (Llama3.1-8B) versus ∼10--20% for prior rubric baselines, with gains that transfer to HealthBench-Hard and BiGGen Bench. Overall, RRD establish recursive rubric refinement as a scalable and interpretable foundation for LLM judging and reward modeling in open-ended domains.