UnGuide: Learning to Forget with LoRA-Guided Diffusion Models

Abstract

Large-scale text-to-image diffusion models pose safety and compliance risks due to their ability to generate harmful or undesirable content. Machine unlearning seeks to remove specific concepts from pretrained models while preserving overall generative quality. Low-Rank Adaptation (LoRA) enables parameter-efficient targeted forgetting, but naive LoRA-based unlearning often induces distributional shift, degrading fidelity and destabilizing sampling. In this paper, we propose UnGuide, a LoRA-based unlearning framework with stability-aware adaptive guidance. Our method dynamically modulates classifier-free guidance (CFG) based on early denoising trajectory variance. By estimating discrepancies between base and LoRA-adapted noise predictions during initial diffusion steps, UnGuide interpolates per prompt between the frozen and adapted models. High-variance trajectories receive stronger adapted-model guidance to enforce forgetting, while low-variance trajectories rely on the base model to preserve fidelity. This mechanism enables controlled concept suppression without prompt embedding modification or external segmentation. Experiments on object erasure and explicit content removal demonstrate improved forgetting efficacy and superior fidelity retention compared to existing LoRA-based approaches.