Geometry-Preserving Unsupervised Alignment for Heterogeneous Foundation Models

摘要

Foundation models have driven rapid progress in computer vision, yet the two dominant paradigms, vision-language foundation models (VLMs) and vision-only foundation models (VFMs), remain only partially compatible. VLMs offer language-grounded semantic alignment but are often visually coarse, while VFMs learn discriminative perceptual geometry but lack semantic grounding. We propose **GPUA**, a *Geometry-Preserving Unsupervised Alignment* framework that integrates the complementary strengths of VFMs and VLMs. Inspired by cross-lingual alignment, **GPUA** treats VFM features as a *visual language* and learns an *orthogonal* mapping that translates the VFM space into the VLM semantic space, preserving geometry and narrowing the modality gap without labels or model parameter updates. **GPUA** is task-agnostic and requires only feature-level access to pretrained models. Experiments across diverse benchmarks demonstrate improved cross-model compatibility and strong gains in downstream zero-shot recognition and segmentation with negligible overhead. Our code is available at: https://github.com/Yuteam14/GPUA.