Omni-RRM: Advancing Omni Reward Modeling via Automatic Rubric-Grounded Preference Synthesis

Abstract

Multimodal large language models (MLLMs) struggle withalignment due to the limitations of existing reward models (RMs), whichare predominantly vision-centric, dependent on costly human labels, andprovide opaque scalar scores that fail to capture nuanced reasoning,leading to brittle alignment. We present Omni-RRM, an Omni-modalRubric-grounded Reward Model that generates multi-dimensional re-ward signals across text, image, video, and audio. To overcome the highcost and inherent inconsistency of human-centric evaluation in multi-dimensional reasoning, we introduce Omni-Preference, a high-qualitydataset constructed via automatic rubric-grounded preference synthe-sis. In this pipeline, teacher models reconcile raw preferences into ex-plicit justifications, ensuring that the synthesized supervision is bothhigh-fidelity and interpretable. Omni-RRM is trained using a progres-sive SFT + GRPO regimen, specifically optimized to sharpen rewarddiscrimination on low-margin, hard preference pairs. It achieves state-of-the-art accuracy on video (80.2% on ShareGPT-Video) and audiobenchmarks (66.8% on Audio-HH-RLHF and 65.0% on TA2T), yield-ing a five-benchmark Overall accuracy of 70.4% and a +17.0% relativegain over its backbone. Furthermore, Omni-RRM effectively guides Best-of-N selection and exhibits robust transfer to text-only alignment. Allresources, including the dataset, training and inference code, and modelcheckpoints are available at https://tmfk418.github.io/Omni-RRM.