Failure is Feedback: History-Aware Backtracking for Agentic Traversal in Multimodal Graphs

摘要

Open-domain multimodal document retrieval aims to retrieve specific components (paragraphs, tables, or images) from large and interconnected document corpora. Existing graph-based retrieval approaches typically rely on a uniform similarity metric that overlooks hop-specific semantics, and their rigid pre-defined plans hinder dynamic error correction. These limitations suggest that a retriever should adapt its reasoning to the evolving context and recover intelligently from dead ends. To address these needs, we propose **Failure is Feedback**, which casts subgraph retrieval as a *sequential decision process* and introduces two key innovations. (i) We introduce a *history-aware backtracking mechanism*; unlike standard backtracking that simply reverts the state, our approach piggybacks on the context of failed traversals, leveraging insights from previous failures. (ii) We implement an *economically-rational agentic workflow*. Unlike conventional agents with static strategies, our orchestrator employs a cost-aware traversal method to dynamically manage the trade-off between retrieval accuracy and inference costs, escalating to intensive LLM-based reasoning only when the prior failure justifies the additional computational investment. The project page is available at [failureisfeedback.github.io](https://failureisfeedback.github.io/).