Large Language Model is not a (Multilingual) Compositional Relation Reasoner

摘要

We present a comprehensive evaluation of large language models' capability to reason compositional relations through a benchmark encompassing 1,800 test cases in both English and Chinese, covering six distinct categories of composition relations: Positional, Comparative, Personal, Mathematical, Identity, and Other. We expand our assessment to the multilingual realm by including translations of the benchmark suite into Japanese, French, and Korean. Our Multilingual Composition Relation (MCR) benchmark aims at investigating the robustness and adaptability of LLMs in handling compositional relation reasoning across diverse linguistic contexts.