M-Prometheus: A Suite of Open Multilingual LLM Judges

摘要

Employing language models as evaluators of long-form output (LLM-as-a-Judge) has become the \textit{de facto} standard for automatic evaluation. However, most LLM judges have been optimized exclusively for English outputs, with strategies for enhancing judges' multilingual evaluation capabilities remaining largely unexplored in the current literature. This has created a disparity in the quality of automatic evaluation methods for other languages, ultimately hindering the development of models with better multilingual capabilities. To bridge this gap, we introduce M-Prometheus, a suite of open-weight LLM judges ranging from 3B to 14B parameters that can provide both direct assessment and pairwise comparison feedback on multilingual outputs. M-Prometheus models outperform state-of-the-art open LLM judges on multilingual reward benchmarks spanning more than 20 languages, as well as on literary machine translation evaluation covering 4 language pairs. Furthermore, we find M-Prometheus models can be used with quality-aware decoding methods to significantly improve generated outputs, showcasing their utility for the development of better multilingual models. Crucially, through extensive ablations, we identify key strategies for training an effective multilingual judge. Our findings highlight the significance of model size and base model selection, and the advantages of using natively multilingual data rather than translated data. We release our models, training dataset, and code to reproduce our experiments.