A4-Agent: An Agentic Framework for Zero-Shot Affordance Reasoning

摘要

Affordance prediction, which identifies interaction regionson objects based on language instructions, is critical for embodied AI.Prevailing end-to-end models couple high-level reasoning and low-levelgrounding into a single monolithic pipeline and rely on training over an-notated datasets, which leads to poor generalization on novel objects andunseen environments. In this paper, we move beyond this paradigm byproposing A4-Agent, a training-free agentic framework that decouplesaffordance prediction into a three-stage pipeline. Our framework coordi-nates specialized foundation models at test time: (1) a Dreamer thatemploys generative models to visualize how an interaction would look;(2) a Thinker that utilizes large vision-language models to decide whatobject part to interact with; and (3) a Spotter that orchestrates vi-sion foundation models to precisely locate where the interaction area is.By leveraging the complementary strengths of pre-trained models with-out any task-specific fine-tuning, our zero-shot framework significantlyoutperforms state-of-the-art supervised methods across multiple bench-marks and demonstrates robust generalization to real-world settings.