UniMotion: A Unified Framework for Motion-Text-Vision Understanding and Generation

Abstract

We present UniMotion, to our knowledge the first unifiedframework for simultaneous understanding and generation of humanmotion, natural language, and RGB images within a single architecture.Existing unified models handle only restricted modality subsets (e.g.,Motion–Text or static Pose–Image) and predominantly rely on discretetokenization, which introduces quantization errors and disrupts temporalcontinuity. UniMotion overcomes both limitations through a core princi-ple: treating motion as a first-class continuous modality on equal footingwith RGB. A novel Cross-Modal Aligned Motion VAE (CMA-VAE) andsymmetric dual-path embedders construct parallel continuous pathwaysfor Motion and RGB within a shared LLM backbone. To inject visual-semantic priors into motion representations without requiring images atinference, we propose Dual-Posterior KL Alignment (DPA), which distillsa vision-fused encoder’s richer posterior into the motion-only encoder. Toaddress the cold-start problem—where text supervision alone is too sparseto calibrate the newly introduced motion pathway—we further proposeLatent Reconstruction Alignment (LRA), a self-supervised pre-trainingstrategy that uses dense motion latents as unambiguous conditions toco-calibrate the embedder, backbone, and flow head, establishing a stablemotion-aware foundation for all downstream tasks. UniMotion achievesstate-of-the-art performance across seven tasks spanning any-to-any un-derstanding, generation, and editing among the three modalities, withespecially strong advantages on cross-modal compositional tasks.