ReasonCLIP-58M: Visually Grounded Commonsense Reasoning Supervision for CLIP

摘要

CLIP and its variants are widely adopted visual backbonesin multimodal systems, but their pretraining remains dominated by de-scriptive image-text alignment. As downstream applications increasinglydemand visually grounded commonsense inference and compositionalreasoning, it remains unclear whether CLIP-style encoders can supportsuch reasoning without architectural changes. To address this, we presentReasonCLIP-58M, a continual pretraining framework that integrates large-scale reasoning supervision into CLIP-style models through our two-stage strategy, which progressively integrates reasoning signals while pre-serving descriptive alignment, followed by category-structured reason-ing supervision. To support this framework, we construct two comple-mentary datasets and a benchmark: ReasonLite-42M, with open-form,visually verifiable reasoning captions; ReasonPro-16M, with category-specific reasoning supervision; and RCLIP-Bench for diagnostic evalua-tion of visually grounded reasoning. We train a family of ReasonCLIPthat improves visually grounded commonsense and compositional reason-ing while also enhancing zero-shot retrieval performance. As a drop-invisual encoder for multimodal large language models such as LLaVA-NeXT, ReasonCLIP delivers consistent gains without additional infer-ence cost, demonstrating that structured reasoning supervision enhancesthe expressive capacity of CLIP-style visual representations. All datasets,models, and training code are available at https://github.com/RISys-Lab/ReasonCLIP.