FingerCap: Fine-grained Finger-level Hand Motion Captioning

Abstract

Understanding human hand motion requires reasoning thatgoes beyond coarse action semantics to capture detailed finger articula-tion and interaction dynamics. However, existing Video-MLLMs primar-ily capture global action intent and often fail to represent fine-grainedfinger-level movements and contact semantics. In this work, we proposeFine-grained Finger-level Hand Motion Captioning (FingerCap), whichaims to generate textual descriptions that capture detailed finger-levelsemantics of hand actions. To support this task, we curate FingerCap-40K, a large-scale corpus of 40K paired hand-motion videos and captionsspanning two complementary sources: concise instruction-style finger mo-tions and diverse, naturalistic hand–object interactions. Furthermore,we design HandJudge, an LLM-based rubric that measures finger-level correctness and motion completeness for effective evaluation. Toestablish a strong yet lightweight baseline under the FingerCap task,we introduce FiGOP (Finger Group-of-Pictures), a module tailoredfor Video-MLLM settings with sparse RGB sampling. FiGOP augmentskeyframes with intermediate 2D hand keypoints, enabling complemen-tary motion cues without increasing the density of RGB frames. Exper-iments on FingerCap-40K show that strong open- and closed-sourceVideo-MLLMs still struggle with finger-level reasoning, while our FiGOP-enhanced model yields consistent gains under both HandJudge and hu-man studies. The dataset and benchmark are available at § FingerCap.