What if? Emulative Simulation with World Models for Situated Reasoning

Abstract

Situated reasoning often relies on active exploration, yet inmany real-world scenarios such exploration is infeasible due to phys-ical constraints of robots or safety concerns of visually impaired users.Given only a limited observation, can an agent mentally simulate a futuretrajectory toward a target situation and answer spatial “what-if” ques-tions? We introduce WanderDream, the first large-scale dataset designedfor the emulative simulation of mental exploration, enabling models toreason without active exploration. WanderDream-Gen comprises 15.8Kpanoramic videos across 1,088 real scenes from HM3D, ScanNet++,and real-world captures, depicting imagined trajectories from currentviewpoints to target situations. WanderDream-QA contains 158K ques-tion–answer pairs, covering starting states, paths, and end states alongeach trajectory to comprehensively evaluate exploration-based reason-ing. Extensive experiments with world models and MLLMs demonstrate(1) that mental exploration is essential for situated reasoning, (2) thatworld models achieve compelling performance on WanderDream-Gen, (3)that imagination substantially facilitates reasoning on WanderDream-QA, and (4) that WanderDream data exhibit remarkable transferabilityto real-world scenarios. The source code and all data are released athttps://github.com/RuipingL/WanderDream.