BioMedVR: Confusion-Aware Mixture-of-Prompt Experts for Biomedical Visual Reprogramming

Abstract

Recent advances in vision–language models (VLMs) such asCLIP have demonstrated strong generalization across natural-image do-mains. However, adapting these models to biomedical imaging is non-trivial: full-model fine-tuning is computationally expensive, while medi-cal data are often scarce and exhibit subtle, fine-grained inter-class differ-ences, making parameter-efficient adaptation particularly critical. VisualReprogramming (VR) offers a parameter-efficient alternative by inject-ing learnable perturbations into the input space, but existing VR ap-proaches for VLMs mainly focus on positive class prompts and overlookconfusing negatives, leading to miscalibrated predictions in fine-grainedmedical scenarios. We present BioMedVR, the first VR-based frame-work for biomedical imaging, enabling few-shot adaptation of pretrainedVLMs through compact learnable VR modules. To mitigate class confu-sion, we introduce a Confusion Minimization Mechanism that leveragesLLM-generated confusion-aware attributes together with a Confusion-Suppression Loss to explicitly reduce false-positive alignment. More-over, the designed Mixture-of-Prompt Experts combines a positive ex-pert for main-class discrimination and a negative expert for confusionsuppression, balanced via adaptive gating. Extensive experiments on 18datasets—including 11 biomedical datasets and 7 natural image bench-marks—demonstrate that BioMedVR achieves superior accuracy andgeneralization, effectively bridging VR and VLMs in biomedical domains.Project page: https://jxliu-ai.github.io/biomedvr-page/.