CrossFeat: Bridging Imaging Modalities in Feature Descriptor Space

摘要

Most advances in keypoint descriptions address monomodalsettings, where image variations arise from viewpoint, illumination, orcontrast changes. Multimodal scenarios involve images produced by fun-damentally different sensing processes, such as multispectral imaging,RGB-depth, satellite imagery, or medical imaging, causing the samestructures to appear differently. A common solution to cross-modal de-scription is to train descriptors for each modality pair, which requires re-training whenever the modalities change, or to train large models, whichincur a significant increase in runtime. Instead, we propose CrossFeat,a framework that enables an existing monomodal descriptor to operateacross modalities. Our method learns a crossing function in descriptorspace that maps features from one modality to a representation com-patible with another. To preserve the structural information capturedby the original descriptor, CrossFeat introduces a geometry–appearancedisentanglement such that only appearance is altered while the geomet-ric properties are preserved. Experiments across multiple domains anddatasets demonstrate improved performance in multimodal matching.Code available here: https://github.com/paulschneider01/CrossFeat