MemoBench: Benchmarking World Modeling in Dynamically Changing Environments
作者: Haoyu Chen, Kaichen Zhou, Hang Hua, Kaile Zhang, Jingwen Qian, Wufei Ma, Haonan Chen, Chunjiang Liu, Yizhou Zhao, Xiaoyuan Wang, Weiyue Li, Alan Yuille, Paul Pu Liang, Yilun Du
摘要
Video generation models aspire to simulate dynamic environ-ments, and several benchmarks now evaluate memory consistency acrossframes. However, most assess consistency only while the target remainsin view, and the few that force objects out of view evaluate static sceneswhere nothing changes during occlusion. To bridge this gap, we introduceMemoBench, a diagnostic benchmark built around the disappear-and-reappear paradigm in dynamically changing environments: a target ob-ject undergoes a physical process, disappears from view, and must be cor-rectly recovered in its updated state upon reappearance. We curate 360ground-truth clips spanning synthetic and real-world scenes, and designan evaluation suite combining automated metrics with VQA-based as-sessment across four diagnostic pillars. Evaluation of ten state-of-the-artmodels reveals key insights and open challenges regarding memory con-sistency under the disappear-and-reappear paradigm. Our dataset, code,and leaderboard are available at https://github.com/MemoBench-Team.