ConsensusXAI: A Framework to Examine Class-wise Agreement in Medical Imaging

Abstract

Explainable AI (XAI) is essential for trust and transparencyin deep learning, especially in medical imaging.Existing local explanation methods provide per-instance insightsbut fail to show whether similar explanations holdacross samples of the same class. This limits global interpretabilityand demands time-consuming manual reviewby clinicians to trust models in practice. We introduce theConsensus Alignment Score (CAS), a novel metric thatquantifies consistency of explanations at the class level.We also present ConsensusXAI, an open-source, modelandmethod-agnostic framework that evaluates explanationagreement quantitatively (via CAS) and qualitatively(through consensus heatmaps) per class. Unlike priorbenchmarks, ConsensusXAI uses a latent-space clusteringapproach, Latent Consensus, to identify dominant explanationpatterns, exposing biases and inconsistencies towardscertain classes. Evaluated across four benchmark datasetsand two imaging modalities, our method consistently revealsmeaningful class-level insights, outperforming traditionalmetrics like SSIM and IoU, and enabling faster, moreconfident clinical adoption of AI models.