Sound-based Multi-Person 3D Pose Estimation

摘要

Can we recover the 3D poses of multiple people using onlysound? This paper presents the first attempt to estimate multi-person3D poses solely from acoustic signals. Estimating the poses of multipleindividuals using acoustic signals is inherently challenging due to the su-perposition of motion-dependent signal variations. Unlike single-personscenarios, the presence of multiple subjects leads to overlapping acousticsignatures, making it difficult to attribute specific signal changes to anindividual’s pose. Furthermore, the complexity is compounded by inter-person reflections, which introduce intricate propagation delays that ob-scure the temporal motion-acoustic relationship. To address these issues,we propose SoundMHPE (Sound-based Multi-person Human Pose Es-timator), a novel encoder-decoder framework consisting of two key com-ponents. First, the Acoustic Multi-scale Encoder captures diverse tem-poral and fine-grained frequency features to isolate subtle acoustic sig-natures from complex, overlapping signals. Second, the Temporal PoseDecoder employs an attention mechanism to disentangle multi-personinformation across successive frames. By jointly accounting for tempo-ral dynamics and inter-person dependencies, this component preciselyreconstructs frame-wise individual poses. To validate our approach, weconstructed the 6-hour Acoustic Multi-person Pose (AMP) dataset con-sisting of 432K synchronized frames of multi-person pose and acous-tic data, and demonstrated that our SoundMHPE outperforms baselinemodels. Project page: https://oumi03.github.io/sound-mhpe/