RoMan-4D: Learning Robot Arm Manipulation from 4D World Models

Abstract

Video world models can generate realistic futures from a sin-gle instruction, but they often fail to track the same physical pointsconsistently across time. As a result, the generated videos appear plausi-ble, yet lack the physical grounding required for reliable action execution,such as robot manipulation. We present GEM-4D, a geometry-groundedvideo world model that resolves this limitation by injecting dense 4Dcorrespondence supervision distilled from a pretrained geometry foun-dation model into the video generative backbone during training. Thissupervision enables the video world model to jointly capture appearanceand geometric structure while retaining a single-stream architecture withno additional inference cost. We further introduce an inverse dynam-ics module that converts correspondence-consistent video rollouts intoexecutable robot trajectories, enabling direct deployment in both real-world and simulated manipulation. GEM-4D achieves state-of-the-artperformance on both video prediction and geometric consistency acrossboth simulation and realistic scenarios and improves real-world manipu-lation success from 61% to 81%. Additional results are available at thehttps://gem-4d.github.io/.