MemLearner: Learning to Query Context Memory for Video World Models

Abstract

Video World Models are interactive video generation modelsthat predict future world states based on user actions and history videoframes. A critical challenge in video world models is the lack of memory,causing inconsistent generated scenes over extended durations. Previousmethods explored rule-based context frame retrieval as memory, but theyfail to generalize in scenarios with scene occlusions and dynamic objects.We propose MemLearner, a learning-based adaptive context querymethod using query tokens to bridge context and predicted tokens. Byleveraging the video generation model itself for context querying, Mem-Learner exploits pre-trained visual priors without training additionalmodules from scratch, and incorporates e_x001E_cient strategies for trainingand inference. We collect a dataset of long videos with scene occlusionsand dynamic objects, paired with camera pose annotations, and proposea multi-dataset training strategy leveraging both annotated renderedand unannotated real-world videos. Extensive experiments demonstratethat MemLearner signi_x001C_cantly outperforms prior video world modelsin terms of scene consistency and memory, particularly under challengingocclusion and dynamic scenarios.