Isotropic Embedding Perturbations for Robust Vision Language Encoders

摘要

Data augmentation is fundamental to training modern deepvision and multimodal models. While individual methods, such as Ran-dAug, CutMix, Mixup, RandErase, and DropPath, offer strong regular-ization effects, their combined use has saturated in performance due tooverlapping functionalities, and aggressive pixel-level manipulations maydisrupt delicate cross-modal alignment. This saturation motivates thesearch for a new augmentation axis within the embedding space ratherthan the input space. We introduce Aether, a simple plug-in method thatapplies diffusion-style random perturbations in the embedding space viacontrolled alpha-mixing, specifically designed to provide isotropic regu-larization that remains semantically consistent. Inspired by feature-spaceperturbations in language models and image degradation in generativepretraining, Aether induces mild yet effective perturbations that smooththe representations without compromising the fine-grained structuralinformation required for strong vision-language encoders. Across diversearchitectures and across multiple recognition tasks, Aether delivers consis-tent gains over the advanced recipe combining CutMix, Mixup, DropPath,and RandAug—a level of improvement rarely observed with modern aug-mentation alternatives. Notably, Aether demonstrates superior effective-ness in multi-modal alignment, succeeding where traditional pixel-spaceaugmentations fail by providing a stable, isotropic regularization signalthat respects the integrity of the high-dimensional feature space. Code isavailable at https://github.com/naver-ai/aether.