Layer-Aware Video Composition via Split-then-Merge

Abstract

We present Split-then-Merge (StM), a controllable genera-tive video composition framework that minimizes reliance on annotateddatasets and handcrafted rules. Instead of requiring manual supervision,StM decomposes unlabeled videos into dynamic foreground and back-ground layers. By self-composing these elements, the model learns tosynthesize complex interactions between moving subjects and diversescenes, capturing the intricate dynamics necessary for high-_x001C_delity videogeneration. Speci_x001C_cally, StM introduces a transformation-aware train-ing pipeline utilizing multi-layer fusion and augmentation to address af-fordance challenges in video composition. An identity-preservation lossfurther maintains foreground _x001C_delity during blending. Extensive exper-iments show that StM outperforms state-of-the-art methods on bothquantitative benchmarks and qualitative evaluations, including humanstudies and Vision-Language Model assessments. Finally, we release StM-50K, the _x001C_rst multi-layer video dataset, to facilitate future research ingenerative video composition. Data, code, and more details are availableat our project page.Fig. 2: Video Composition. Given input foreground and background videos, image-based methods (a)_x0015_(b) use only the _x001C_rst frame, while (c)_x0015_(e) take full video inputs.(a) Object insertion [59] followed by Image-to-Video (I2V) and (b) end-to-end I2Vcomposition SkyReels [13] fails to retain motion due to lack of video access. (c) Manualcopy-paste preserves motion but violates a_x001B_ordance (swan placed on ground). (d) Naivegenerative composition yields appearance and motion drift (e.g ., black swan turnswhite). (e) Our method preserves identity and motion, and achieves a_x001B_ordance-awareplacement with realistic blending (swan placed in water with wave and shadows).