Robobench: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models as Embodied Brain

Abstract

Building robots that can perceive, reason, and act in dy-namic, unstructured environments remains a core challenge. Recent em-bodied systems often adopt a dual-system paradigm, where System 2handles high-level reasoning while System 1 executes low-level control.In this work, we refer to System 2 as the embodied brain, emphasizingits role as the cognitive core for reasoning and decision-making in ma-nipulation tasks. Given this role, systematic evaluation of the embodiedbrain is essential for advancing robotic intelligence. Yet existing bench-marks emphasize execution success, or, when targeting high-level reason-ing, su_x001B_er from incomplete dimensions and limited task realism, o_x001B_eringonly a partial picture of cognitive capability. To bridge this gap, we intro-duce RoboBench, a benchmark that systematically evaluates multimodallarge language models (MLLMs) as embodied brains. Motivated by thedistinct cognitive roles required across the full manipulation pipeline,RoboBench de_x001C_nes _x001C_ve dimensions_x0016_Instruction Comprehension, Percep-tion Reasoning, Generalized Planning, A_x001B_ordance Prediction, and Fail-ure Analysis_x0016_spanning 14 capabilities, 25 tasks, and 6092 QA pairs. Toensure realism, we curate datasets across diverse embodiments, attribute-rich objects, multi-view scenes, and memory-driven navigation, drawingfrom large-scale real robotic data and in-house collection. For planning,RoboBench introduces an evaluation framework that uses an MLLM as aworld simulator. It moves beyond symbolic matching to evaluate embod-ied feasibility by simulating whether predicted plans can achieve criticalobject-state changes under physical and visual constraints, enabling faith-ful assessment of long-horizon reasoning. Experiments on 18 state-of-the-art MLLMs reveal fundamental limitations: di_x001E_culties with implicit in-struction comprehension, spatiotemporal reasoning, cross-scenario plan-ning, _x001C_ne-grained a_x001B_ordance understanding, and execution failure diagno-Dimension Subdimension Performance DownstreamInstuctionComprehension Explicit Object Goal Implicit Demand GoalPerception Robotic- Object- Scene- Task-Reasoning Centric Centric Centric CentricGeneralization EmbodimentCross- Cross- Cross- Cross-Planning Object View TaskAffordance Static Dynamic NavigationPrediction Affordance Affordance AffordanceFailure Low-Level High-LevelAnanlysis Execution Error Planning Error. Instuction Comprehension - . Long-Horizon Planning - . Next-Step Planning - . Static AffordanceImplicit Demand Goal: Question: Question: Question:Please help me tidy up the apple How to complete the goal? What's the next subgoal? What's the contact point?Put the apple into the drawer open(drawer) pick_up(apple) open(drawer) [x , y ]place(apple, drawer) close(drawer)decomposelong-horizen goal adapt subgoalinto substeps in closed-loopinterpret gather diagnose, monitor state refine subgoal intohuman task-relevant analyze, in closed-loop Yes spatial aligned cuesintention. Perception Reasoning adjust error - . Dynamic Affordanceinformation . Failure Analysis - . Task State EstimationTask-Related:robotic type? single arm Question: Question: Question:object attribute? apple: rigid Is there any error? What type of error? Has current subgoal been completed? What's the movement trajectory?operation note? drawer: closed Yes, high-level error---missing steps Yes No [x , y ], [x ,y ], [x , y ],[x , y ]spatial relation? apple lower than drawermonitor state refine subgoal intoin closed-loop spatial aligned cuesNoFig. 1: Overview of RoboBench We evaluate MLLMs as embodied brains with25 tasks color-coded by 5 dimensions (top left). These dimensions follow the em-bodied execution pipeline (bottom)_x0016_from intent understanding to failure diagnos-ing_x0016_capturing the core cognitive roles of System 2. Performance comparison (top mid-dle) reveals signi_x001C_cant gaps among state-of-the-art MLLMs (top right) RoboBenchscores strongly correlate with downstream VLA performance on CALVIN.sis. We further analyze how embodied cognitive abilities relate to down-stream robotic control. RoboBench provides a comprehensive sca_x001B_oldto quantify high-level cognition, clarify the role of the embodied brain,and guide the development of next-generation MLLMs for more robustrobotic intelligence. Project page: https://robo-bench.github.io.