Mixture of States: Routing Token-Level Dynamics for Multimodal Generation
作者: Haozhe Liu, Ding Liu, Mingchen Zhuge, Zijian Zhou, Tian Xie, Sen He, Yukang Yang, Shuming Liu, Yuren Cong, Jiadong Guo, Hongyu Xu, Ke Xu, Kam Woh Ng, Juan C. Perez, Juan-Manuel Perez-Rua, Tao Xiang, Wei Liu, Shikun Liu, Jurgen Schmidhuber
摘要
We introduce MoS (Mixture of States), a novel fusion paradigm for multimodal diffusion models that merges modalities using flexible, state-based interactions. The core of MoS is a learnable, token-wise router that creates denoising timestep- and input-dependent interactions between modalities' hidden states, precisely aligning token-level features with the diffusion trajectory. This router sparsely selects the top-k hidden states and is trained with an epsilon-greedy strategy, efficiently selecting contextual features with minimal learnable parameters and negligible computational overhead. We validate our design with text-to-image generation (MoS-Image) and editing (MoS-Editing), which achieve state-of-the-art results. With only 3B to 5B parameters, our models match or surpass counterparts up to 4xlarger. These findings establish MoS as a flexible and compute-efficient paradigm for scaling multimodal diffusion models. The project homepage is available at https://haozheliu-st.github.io/mos-homepage/