OpenGround: Planning-based Online Perception for Open-World 3D Visual Grounding

Abstract

3D visual grounding aims to locate objects based on naturallanguage descriptions in 3D scenes. Existing supervised methods are lim-ited by generalization and recent zero-shot methods typically rely on apredefined Object Lookup Table (OLT) to query Visual Language Mod-els (VLMs) for reasoning about object locations via a single step ground-ing, which limits the applications in scenarios with undefined targets andcomplex queries. To address these problems, we present OpenGround, anovel zero-shot framework for open-world 3D visual grounding that re-mains compatible with recent zero-shot methods. OpenGround integratesTask-Chain Planning to decompose a query into a plan of context-to-target sub-goals for progressive grounding, and Context-Guided Percep-tion to perceive novel objects online under context guidance from thetask chain. We also propose a new dataset named OpenTarget, whichcontains over 7000 object-description pairs to mimic open-world eval-uation. Extensive experiments demonstrate that OpenGround achievescompetitive performance on Nr3D, state-of-the-art on ScanRefer, anddelivers a substantial 17.6% improvement on OpenTarget.