VLZip: Unified Visual and Textual Compression for Interleaved Long-Context Modeling

摘要

Vision Language Models (VLMs) face signi_x001C_cant challengeswith ultra-long, interleaved image-text sequences due to the quadraticcomplexity of self-attention. Current solutions either resort to aggres-sive token pruning, risking irreversible information loss, or adopt e_x001E_-cient but less precise architectures, while largely ignoring the equallyvital textual component. We introduce VLZip, a framework that uni-_x001C_es visual and textual compression for high-_x001C_delity reasoning within apure Transformer. At its core, VLZip hierarchically distills visual andtextual segments into compact, layer-speci_x001C_c "soft pre_x001C_xes" and injectsthem into each decoder layer's hidden states, drastically shortening theattention sequence while preserving _x001C_ne-grained global context. To ad-dress de_x001C_cient evaluations in the _x001C_eld, we also introduce LongVLBench,a new benchmark derived from video narratives that demands holistic,narrative-level reasoning. Extensive experiments show VLZip achievesleading performance on long-context multimodal reasoning, enabling train-ing up to 120K tokens_x0016_a 6× increase over the baseline_x0016_and inferencebeyond 280K tokens with signi_x001C_cantly reduced memory, while demon-strating the memory scalability to handle up to 2M tokens. By excellingat extreme context lengths where existing methods collapse, VLZip es-tablishes an e_x001E_cient and powerful new standard for long-context mul-timodal AI. Code is available at https://github.com/ShareLab-SII/VLZip.