DAVE: Distribution-Aware Attribution via ViT Gradient Decomposition

Abstract

Vision Transformers (ViTs) have become a dominant architecture in computer vision, yet producing stable and high-resolution attribution maps remains challenging. Architectural components such as patch embeddings and attention routing often introduce structured artifacts in pixel-level explanations, leading many existing methods to rely on coarse patch-level attributions. We introduce DAVE (Distribution-Aware Attribution via ViT Gradient Decomposition), a mathematically grounded attribution method for ViTs based on a structured decomposition of the input gradient. By exploiting architectural properties of ViTs, DAVE isolates locally equivariant and stable components of the effective input-output mapping while suppressing architecture-induced artifacts and instability. Consequently, DAVE produces robust, precise, and class-consistent attribution maps that highlight model-relevant visual features. Experimental results show that across supervised, self-supervised, and inherently interpretable ViTs, DAVE outperforms prior methods on localization, faithfulness, and user studies.