OmniCoT: A Benchmark for Global and Multi-Step Panoramic Reasoning

摘要

Multimodal Large Language Models (MLLMs) have demon-strated promising spatial reasoning capabilities, while these abilities re-main underexplored in the emerging visual modality of panoramic im-agery. The full 360◦ ×180◦ field of view of panoramas essentially supportscomplex global multi-step reasoning, which is also the fundamental ad-vantage of panoramas in applications such as embodied intelligence. How-ever, existing panoramic benchmarks largely focus on simplistic queriesthat rely on local cues or single-/few-step reasoning, thereby ignoring thefundamental advantage of panoramas and failing to fully exploit theirpotential. To address this gap, we introduce OmniCoT , a panoramicspatial reasoning suite designed to enable MLLMs to use global evi-dence and perform multi-step inference across viewpoints. It includesOmniCoT-B (6.7K data) for evaluation, which measures both answeraccuracy and reasoning quality, OmniCoT-Real (1K data) as a man-ually annotated real-world subset to quantify the Sim-to-Real gap. Fortraining, OmniCoT-T (14.3K data) is purpose-built with structuredstepwise Chain-of-Thought annotations that explicitly link intermediatereasoning steps to panoramic evidence. Based on OmniCoT-T, we intro-duce OmniCoT-R1 and adopt a two-stage training strategy tailoredto the geometrically complex panoramic space, where Supervised Fine-Tuning (SFT) anchors reasoning to panoramic evidence (e.g., bearings,proximity) and GRPO penalizes geometrically incoherent paths to con-solidate global 360◦ spatial consistency. Through OmniCoT, we aim torecalibrate the difficulty of panoramic spatial reasoning to bet-ter align with the intrinsic capabilities of panoramic imagery ,thereby fostering meaningful progress in this research area.