VIVAS: Vitalizing Visual Perception in VLM Pre-training via Vision-language Unified Autoregressive Supervision

Abstract

While Vision–Language Models (VLMs) demonstrate strongcapabilities, they continue to suffer from a critical limitation: insufficientfine-grained visual perception, which fundamentally limits their mul-timodal understanding. We attribute this bottleneck to text-dominantoptimization biases during pre-training, which encourage the model tooverlook fine-grained visual details, thereby limiting the capability ofmultimodal understanding. We investigate that overcoming this bottle-neck requires two key elements: (1) a unified token space paradigm thatensures stable training dynamics, and (2) a modality-aligned dense vi-sual supervision signal enriched with both structural granularity andsemantic information to capture critical visual representations. Basedon these insights, we propose VIVAS, a framework built upon the uni-fied token space paradigm, which introduces a dense-structural–semanticvision tokenizer, which expands the textual vocabulary into a unified vi-sion–language vocabulary by incorporating a visual vocabulary. Duringpretraining, VIVAS performs vision–language unified autoregressive su-pervision over both visual details and linguistic content, thereby enhanc-ing visual perception to improve multimodal understanding. Trained end-to-end on 12.4T tokens, VIVAS achieves state-of-the-art performanceacross 7 tasks and 39 multimodal benchmarks.