Joint Alignment and Distillation for Video Generation via Sample-Guided Distribution Matching

Abstract

Aligning video generative models to human preferences heav-ily relies on Reinforcement Learning (RL), which suffers from extensivecomputational overhead. Existing workflows typically treat RL and dis-tillation as disconnected stages: applying RL before distillation incursprohibitive computational costs, whereas applying RL after distillationfrequently leads to model collapse. To overcome these limitations, wepropose a unified, single-stage optimization framework grounded in Dis-tribution Matching (DM). In the standard DM framework, distillationupdates the model via a gradient direction that minimizes the gap be-tween the real and fake models, guiding generations toward clarity andhigh fidelity. Building upon this, we introduce DM-Align, which derivesa complementary gradient direction to guide the model toward human-preferred samples. Inspired by DPO and GRPO, our method leveragesthe distributional gap—formulated from either preference pairs or intra-group exploration—to directly construct this preference-guided gradient.By synergizing these two gradient directions, our approach eliminatesthe need for multi-step reward evaluation and complex ODE-SDE con-versions inherent in traditional RL. Comprehensive experiments acrossmultiple foundational video models demonstrate that this sample-guidedframework robustly enhances both distillation quality and preferencealignment, consistently outperforming both standalone variants and se-quential two-stage pipelines.