AMCI: Unlock the Potential of Large Multimodal Models for Fine-grained Open-world Classification via Adaptive Memory Context Injection

摘要

Open World Classification (OWC) with Large MultimodalModels (LMMs) is an emerging and promising task that moves beyondthe closed-world setting by enabling direct answer generation in responseto open-ended queries. However, we observe that the OWC performanceof LMMs significantly degrades on fine-grained tasks. Empirically, wefind that their performance is sensitive to contextual information in thequery. To address this limitation, we propose Adaptive Memory ContextInjection (AMCI), a training-free and model-agnostic framework that dy-namically restores contextual information during OWC inference. AMCImaintains a non-parametric memory that aligns visual indices with theircorresponding attribute-augmented descriptions. By employing an adap-tive similarity threshold to preserve memory diversity and utilizing arefinement LLM to transform accumulated context into a structuredprompt, AMCI transforms the initial zero-shot description into a cali-brated fine-grained prediction without parameter updates. Furthermore,we construct a specialized low-resource fine-grained benchmark to sim-ulate extreme data scarcity and introduce a new metric that penalizesoverly generic predictions, tailored for fine-grained OWC evaluation. Ex-tensive experiments demonstrate that AMCI consistently improves per-formance by a substantial margin while remaining computationally effi-cient and compatible with various LMMs.