Magic-MM-Embedding: Towards Visual-Token-Efficient Universal Multimodal Embedding with MLLMs

摘要

Multimodal Large Language Models (MLLMs) have shownimmense promise in universal multimodal retrieval, which aims to findrelevant items of various modalities for a given query. However, theirpractical application is often hindered by the substantial computationalcost incurred from processing a large number of tokens from visual inputs.In this paper, we propose Magic-MM-Embedding, a series of novel modelsthat achieve both high efficiency and state-of-the-art performance inuniversal multimodal embedding. Our approach is built on two synergisticpillars: (1) a highly efficient MLLM architecture incorporating visualtoken compression to drastically reduce inference latency and trainingtime, and (2) a multi-stage progressive training strategy designed tonot only recover but significantly boost performance. This coarse-to-finetraining paradigm begins with extensive continued training to restoremultimodal understanding and generation capabilities, progresses tolarge-scale contrastive pretraining and hard negative mining to enhancediscriminative power, and culminates in a task-aware fine-tuning stageguided by an MLLM-as-a-Judge for precise data curation. Comprehensiveexperiments show that our model outperforms existing methods by alarge margin while being more inference-efficient.