PhysRAG: Enhancing Physics-Awareness in Video Generation via Retrieval-Augmented Generation

摘要

Developing physically aware video generation models remainsa significant challenge due to the difficulty in capturing diverse physicalphenomena, such as thermal dynamics, mechanics, and optics. In thiswork, we introduce PhysRAG, a novel pipeline that enhances physicalawareness in video generation through Retrieval-Augmented Generation(RAG). To address the issue of limited high-quality data, we design atwo-stage data filtering pipeline based on the WISA-80K dataset, result-ing in a curated set of 7K high-quality videos for training. Furthermore,we construct a physical video database and develop a mechanism to injectphysical knowledge into a video diffusion model using learnable queries.Our method achieves state-of-the-art performance in both visual qualityand physical rule compliance, surpassing existing models in benchmarkssuch as PhyGenBench and VBench. We conduct extensive ablation stud-ies to validate the effectiveness of our key components, including the datafiltering pipeline, RAG mechanism, and method for physical informationextraction. To facilitate future research, our code, data, and models areprepared for release at https://github.com/sediment1024/PhysRAG.