RoboGesture: Real-Time Semantic-aligned Co-Speech Gestures Generation for Humanoid Interaction

摘要

Enabling humanoid robots to respond to human speech withsynchronized and semantically meaningful gestures is fundamental tonatural human-robot interaction. However, this task faces three criticalbarriers: the scarcity of semantically rich datasets, the "modality eclipse"where models ignore audio cues in favor of kinematic inertia, and the sim-to-real gap regarding physical safety. We propose RoboGesture, a robot-centric framework that co-designs data, modeling, and control to powera complete interactive human–humanoid system in which the robot lis-tens, responds, and gestures in real time. We first establish the Robo-Gesture dataset featuring over 300 gesture categories and develop anautomated pipeline to synthesize large-scale collision-free, robot-specificaudio-motion pairs. Our architecture features a Hierarchical Semantic-Acoustic Aligner that extracts multi-granular prosodic and semantic cuesdirectly from raw audio tokens. These cues drive a Streaming ConditionalMotion Generator based on a diffusion transformer with conditional flowmatching. To ensure high responsiveness, we introduce Anti-Inertia CFGMasking, which prevents the model from collapsing into repetitive his-torical patterns by compelling it to proactively mine control signals fromthe audio modality. Finally, an MPC-based safety filter ensures real-time, collision-free execution on physical hardware. Experiments on aUnitree G1 humanoid demonstrate that RoboGesture generates safer,more rhythmic, and more semantically appropriate responses comparedto state-of-the-art baselines.