Diffusion Model as a Generalized Segmentation Learner

Abstract

Diffusion models are primarily trained for image synthesis,yet their denoising trajectories encode rich, spatially aligned visual pri-ors. In this paper, we demonstrate that these priors can be utilized fortext-conditioned semantic and open-vocabulary segmentation, and thisapproach can be generalized to various downstream tasks to make ageneral-purpose diffusion segmentation framework. Concretely, we intro-duce DiGSeg (Diffusion Models as a Generalist Segmentation Learner),which repurposes a pretrained diffusion model into a unified segmenta-tion framework. Our approach encodes the input image and ground-truthmask into the latent space and concatenates them as conditioning sig-nals for the diffusion U-Net. A parallel CLIP-aligned text pathway injectslanguage features across multiple scales, enabling the model to align tex-tual queries with evolving visual representations. This design transformsan off-the-shelf diffusion backbone into a universal interface that pro-duces structured segmentation masks conditioned on both appearanceand arbitrary text prompts. Extensive experiments demonstrate state-of-the-art performance on standard semantic segmentation benchmarks, aswell as strong open-vocabulary generalization and cross-domaintransfer to medical, remote sensing, and agricultural scenarios—withoutdomain-specific architectural customization. These results indicate thatmodern diffusion backbones, can serve as generalist segmentation learn-ers rather than pure generators, narrowing the gap between visual gen-eration and visual understanding.