Kirin: Animal Motion Generation from In-the-Wild Video

Abstract

Understanding animal motion is fundamental to modelinganimal behavior and biomechanics, yet progress in this area lags far be-hind human motion research due to the scarcity of high-quality motiondata. While human motion can be captured in controlled environments, itis impractical for most animal species, resulting in small, domain-limiteddatasets that restrict downstream applications such as animation. To ad-dress this challenge, we introduce Kirin, a framework that reconstructsmotion from video, learns motion priors at scale, and generates realisticmotion that can be directly applied to animated assets. Using large col-lections of in-the-wild animal videos, we reconstruct 3D motion sequencesand pair them with captions to create AiM3D, the first large-scale datasetoffering aligned video-text-motion tuples for quadruped animals. Build-ing on this dataset, we develop a visual-guided motion generation modelthat conditions on both text and image to guide the generation of re-alistic motion across diverse animal species. Finally, by leveraging anoff-the-shelf image-to-3D model, we automatically rig and animate 3Dmeshes using generated motion, producing ready-to-render animated an-imals. Together, our dataset and framework establish a new foundationfor large-scale, text and image conditioned animal motion generation andanimation. Project page: https://kirin-ani.github.io/.