Memory Tree Guided Key Frame Querying for Efficient 3D Question Answering

Abstract

Answering questions accurately and efficiently in embodiedscenarios presents significant challenges due to limited computationaland memory resources for Vision Language Model (VLM) inference. Ex-isting methods adopt visual search key frame retrieval method to se-lect critical question-related key frames for VLM input. However, visualsearch methods are inefficient because they require visual search amongthousands of video frames for each individual user query. In this work, wepropose a memory tree guided key frame selection paradigm for efficient3D question answering in embodied scenarios. Our method leveragesa compact and reusable 3D scene representation, termed MemTree3D,which supports real-time online construction leveraging camera 6-DoFposes. MemTree3D captures multi-level 3D scene information, enabling aLarge Language Model to efficiently query and retrieve question-relevantkey frames through our scoring-based frame selection without repro-cessing the entire video stream. On OpenEQA, our method improvesthe LLM-Match of GPT-4o by 17.4%, LLaVA-OneVision-7B by 5.8%,outperforms existing visual search methods. Our code is available athttps://github.com/hsiangwei0903/MemTree3D.