Evaluating Reasoning Coherence in Video Generative Models with Text and Visual Hints

摘要

Video generative models show emerging reasoning behaviors.It is essential to ensure that generated events remain causally consistentacross frames for reliable deployment, a property we define as reasoningcoherence. To bridge the gap in literature for missing reasoning coher-ence evaluation, we propose MME-CoF-Pro, a comprehensive videoreasoning benchmark to assess reasoning coherence in video models.Specifically, MME-CoF-Pro contains 303 samples across 16 categories,ranging from visual logical to scientific reasoning. It introduces Rea-soning Score as evaluation metric for assessing process-level necessaryintermediate reasoning steps, and includes three evaluation settings, (a)no hint, (b) text hint, and (c) visual hint, enabling a controlled investi-gation into the underlying mechanisms of reasoning hint guidance. Eval-uation results in 7 open and closed-source video models reveals insightsincluding: (1) Video generative models exhibit weak reasoning coherence,decoupled from generation quality. (2) Text hints boost apparent correct-ness but often cause inconsistency and hallucinated reasoning (3) Visualhints benefit structured perceptual tasks but struggle with fine-grainedperception. We provide our project page along with data and code at:https://video-reasoning-coherence.github.io/.