VersaViT: Enhancing MLLM Vision Backbones via Task-Guided Optimization

Abstract

Multimodal Large Language Models (MLLMs) have recentlyachieved remarkable success in visual-language understanding, demon-strating superior high-level semantic alignment within their vision en-coders. An important question thus arises: Can these encoders serve asversatile vision backbones, capable of reliably performing classic vision-centric tasks as well? To address the question, we make the following con-tributions: (i) We identify that the vision encoders within MLLMs exhibitdeficiencies in their dense feature representations, as evidenced by theirsuboptimal performance on dense prediction tasks (e.g., semantic seg-mentation, depth estimation); (ii) We propose VersaViT, a well-roundedvision transformer that instantiates a novel multi-task framework for col-laborative post-training. This framework facilitates the optimization ofthe vision backbone via lightweight task heads with multi-granularity su-pervision; (iii) Extensive experiments across various downstream tasksdemonstrate the effectiveness of our method, yielding a versatile visionbackbone suited for both language-mediated reasoning and pixel-levelunderstanding. The project page is available here.