Diffusion Models are Open-World Affordance Learners: Leveraging Generative Priors for 3D Affordance Learning
Authors: Hanqing Wang, Zhenhao Zhang, Kaiyang Ji, Mingyu Liu, Wenti Yin, Yuchao Chen, Zhirui Liu, Xiangyu Zeng, Tianxiang Gui, Hangxing Zhang, Jiahao Yuan, Zhiqing Cui, Jiaxin Liu, Zhiyuan Ma, Hui Xiong
Abstract
3D affordance grounding aims to understand how diverseobjects can be manipulated, making it a cornerstone of embodied inter-action. However, prior works struggle to generalize to out-of-distribution,open-world scenarios, leaving a critical gap between limited dataset per-formance and real-world application needs. Inspired by the saying: “WhatI can not create, I do not understand”, we find generative modelscan generate semantically valid HOI images, which indicates inherent en-coding of affordance concepts. Building on this insight, we propose DAG,the first innovative diffusion-based 3D affordance grounding frameworkthat extracts general affordance knowledge from text-to-image diffusionmodels for 3D affordance prediction. Specifically, we extract the affor-dance priors from a diffusion model to encode HOI priors, and designan affordance block with a multi-source affordance decoder for dense3D affordance prediction. Extensive experiments show that DAG con-sistently outperforms state-of-the-art methods and exhibits strong open-world generalization, even in the challenging one-shot setting. The codeof our method is released on https://github.com/hq-King/DAG.