WACV 2025 论文列表

Recently introduced ControlNet has the ability to steer the text-driven image generation process with geometric input such as human 2D pose or edge representations. While ControlNet provides control over the geometric form of the instances in the generated image it lacks the capability to dictate the visual appearance of each instance. We present FineControlNet to provide fine control over each instance's appearance while maintaining the pose control capability. Specifically we develop and demonstrate FineControlNet with geometric control via human pose images and appearance control via instance-level text prompts. The spatial alignment of instance-specific text prompts and 2D poses in latent space enables the fine control capabilities of FineControlNet. We evaluate the performance of FineControlNet with rigorous comparison against state-of-the-art pose-conditioned text-to-image diffusion models. FineControlNet achieves superior performance in generating high quality images that follow instance-specific controls. We will release the code and the dataset.

Recent work showed that large diffusion models can be reused as highly precise monocular depth estimators by casting depth estimation as an image-conditional image generation task. While the proposed model achieved state-of-the-art results high computational demands due to multi-step inference limited its use in many scenarios. In this paper we show that the perceived inefficiency was caused by a flaw in the inference pipeline that has so far gone unnoticed. The fixed model performs comparably to the best previously reported configuration while being more than 200x faster. To optimize for downstream task performance we perform end-to-end fine-tuning on top of the single-step model with task-specific losses and get a deterministic model that outperforms all other diffusion-based depth and normal estimation models on common zero-shot benchmarks. We surprisingly find that this fine-tuning protocol also works directly on Stable Diffusion and achieves comparable performance to current state-of-the-art diffusion-based depth and normal estimation models calling into question some of the conclusions drawn from prior works.

3D visual grounding consists of identifying the instance in a 3D scene which is referred to by an accompanying language description. While several architectures have been proposed within the commonly employed grounding-by-selection framework the utilized losses are comparatively under-explored. In particular most methods rely on a basic supervised cross-entropy loss on the predicted distribution over candidate instances which fails to model both spatial relations between instances and the internal fine-grained word-level structure of the verbal referral. Sparse attempts to additionally supervise verbal embeddings globally by learning the class of the referred instance from the description or employing verbo-visual contrast to better separate instance embeddings do not fundamentally lift the aforementioned limitations. Responding to these shortcomings we introduce two novel losses for 3D visual grounding: a visual-level offset loss on regressed vector offsets from each instance to the ground-truth referred instance and a language-related span loss on predictions for the word-level span of the referred instance in the description. In addition we equip the verbo-visual fusion module of our new 3D visual grounding architecture AsphaltNet with a top-down bidirectional attentive fusion block which enables the supervisory signals from our two losses to propagate to the respective converse branches of the network and thus aid the latter to learn context-aware instance embeddings and grounding-aware verbal embeddings. AsphaltNet proposes novel auxiliary losses to aid 3D visual grounding with competitive results compared to the state-of-the-art on the ReferIt3D benchmark.

While text-to-video generation shows state-of-the-art results fine-grained output control remains challenging for users relying solely on natural language prompts. In this work we present FACTOR for fine-grained controllable video generation. FACTOR provides an intuitive interface where users can manipulate the trajectory and appearance of individual objects in conjunction with a text prompt. We propose a unified framework to integrate these control signals into an existing text-to-video model. Our approach involves a multimodal condition module with a joint encoder control-attention layers and an appearance augmentation mechanism. This design enables FACTOR to generate videos that closely align with detailed user specifications. Extensive experiments on standard benchmarks and user-provided inputs demonstrate a notable improvement in controllability by FACTOR over competitive baselines.

Detecting and localising unknown or Out-of-distribution (OOD) objects in any scene can be a challenging task in vision particularly in safety-critical cases involving autonomous systems like automated vehicles or trains. Supervised anomaly segmentation or open-world object detection models depend on training on exhaustively annotated datasets for every domain and still struggle in distinguishing between background and OOD objects. In this work we present a plug-and-play framework - PRototype based OOD detection Without Labels (PROWL). It is an inference-based method that does not require training on the domain dataset and relies on extracting relevant features from self-supervised pre-trained models. PROWL can be easily adapted to detect in-domain objects in any operational design domain (ODD) in a zero-shot manner by specifying a list of known classes from this domain. PROWL as a first zero-shot unsupervised method achieves state-of-the-art results on the RoadAnomaly and RoadObstacle datasets provided in road driving benchmarks - SegmentMeIfYouCan (SMIYC) and Fishyscapes as well as comparable performance against existing supervised methods trained without auxiliary OOD data. We also demonstrate its generalisability to other domains such as rail and maritime.

This paper proposes a novel approach to few-shot semantic segmentation for machinery with multiple parts that exhibit spatial and hierarchical relationships. Our method integrates the foundation models CLIPSeg and Segment Anything Model (SAM) with the interest point detector SuperPoint and a graph convolutional network (GCN) to accurately segment machinery parts. By providing 1 to 25 annotated samples our model evaluated on a purely synthetic dataset depicting a truck-mounted loading crane achieves effective segmentation across various levels of detail. Training times are kept under five minutes on consumer GPUs. The model demonstrates robust generalization to real data achieving a qualitative synthetic-to-real generalization with a J&F score of 92.2 on real data using 10 synthetic support samples. When benchmarked on the DAVIS 2017 dataset it achieves a J&F score of 71.5 in semi-supervised video segmentation with three support samples. This method's fast training times and effective generalization to real data make it a valuable tool for autonomous systems interacting with machinery and infrastructure and illustrate the potential of combined and orchestrated foundation models for few-shot segmentation tasks.

Federated- and Continual Learning have been established as approaches to enable privacy-aware learning on continuously changing data as required for deploying AI systems in histopathology images. However data shifts can occur in a dynamic world spatially between institutions and temporally due to changing data over time. This leads to two issues: Client Drift where the central model degrades from aggregating data from clients trained on shifted data and Catastrophic Forgetting from temporal shifts such as changes in patient populations. Both tend to degrade the model's performance of previously seen data or spatially distributed training. Despite both problems arising from the same underlying problem of data shifts existing research addresses them only individually. In this work we introduce a method that can jointly alleviate Client Drift and Catastrophic Forgetting by using our proposed Dynamic Barlow Continuity that evaluates client updates on a public reference dataset and uses this to guide the training process to a spatially and temporally shift-invariant model. We evaluate our approach on the histopathology datasets BCSS and Semicol and prove our method to be highly effective by jointly improving the dice score as much as from 15.8% to 71.6% in Client Drift and from 42.5% to 62.8% in Catastrophic Forgetting. This enables Dynamic Learning by establishing spatio-temporal shift-invariance.

Intracranial Hemorrhage is a potentially lethal condition whose manifestation is vastly diverse and shifts across clinical centers worldwide. Deep-learning-based solutions are starting to model complex relations between brain structures but still struggle to generalize. While gathering more diverse data is the most natural approach privacy regulations often limit the sharing of medical data. We propose the first application of Federated Scene Graph Generation. We show that our models can leverage the increased training data diversity. For Scene Graph Generation they can recall up to 20% more clinically relevant relations across datasets compared to models trained on a single centralized dataset. Learning structured data representation in a federated setting can open the way to the development of new methods that can leverage this finer information to regularize across clients more effectively.

Federated learning (FL) commonly assumes that the server or some clients have labeled data which is often impractical due to annotation costs and privacy concerns. Addressing this problem we focus on a source-free domain adaptation task where (1) the server holds a pre-trained model on labeled source domain data (2) clients possess only unlabeled data from various target domains and (3) the server and clients cannot access the source data in the adaptation phase. This task is known as Federated source-Free Domain Adaptation (FFREEDA). Specifically we focus on classification tasks while the previous work solely studies semantic segmentation. Our contribution is the novel Federated learning with Weighted Cluster Aggregation (FedWCA) method designed to mitigate both domain shifts and privacy concerns with only unlabeled data. Fed-WCA comprises three phases: private and parameter-free clustering of clients to obtain domain-specific global models on the server weighted aggregation of the global models for the clustered clients and local domain adaptation with pseudo-labeling. Experimental results show that Fed-WCA surpasses several existing methods and baselines in FFREEDA establishing its effectiveness and practicality.

Weakly-supervised Semantic Segmentation (WSSS) aims to provide a precise semantic segmentation results without expensive pixel-wise segmentation labels. With the supervision gap between classification and segmentation Image-level WSSS mainly relies on Class Activation Maps (CAMs) from the classification model to emulate the pixel-wise annotations. However CAMs often fail to cover the entire object region because classification models tend to focus on narrow discriminative regions in an object. Towards accurate CAM coverage Existing WSSS methods have tried to boost feature representation learning or impose consistency regularization to the classification models but still there are limitation in activating non-discriminative area where the focus of the models is weak. To tackle this issue we propose FSAE framework which provides explicit supervision of non-discriminative area encouraging the CAMs to activate on various object features. We leverage weak-strong consistency with pseudo-label expansion strategy for reliable supervision and enhance learning of non-discriminative object boundaries. Specifically we use strong perturbation to make challenging inference target and focus on generating reliable pixel-wise supervision signal for broad object regions. Extensive experiments on the WSSS benchmark datasets show that our method boosts initial seed quality and segmentation performance by large margin achieving new state-of-the-art performance on benchmark WSSS datasets. Our public code is available at https://github.com/obeychoi0120/FSAE.

Leveraging a transferability estimation metric facilitates the non-trivial challenge of selecting the optimal model for the downstream task from a pool of pre-trained models. Most existing metrics primarily focus on identifying the statistical relationship between feature embeddings and the corresponding labels within the target dataset but overlook crucial aspect of model robustness. This oversight may limit their effectiveness in accurately ranking pre-trained models. To address this limitation we introduce a feature perturbation method that enhances the transferability estimation process by systematically altering the feature space. Our method includes a Spread operation that increases intra-class variability adding complexity within classes and an Attract operation that minimizes the distances between different classes thereby blurring the class boundaries. Through extensive experimentation we demonstrate the efficacy of our feature perturbation method in providing a more precise and robust estimation of model transferability. Notably the existing LogMe method exhibited a significant improvement showing a 28.84% increase in performance after applying our feature perturbation method. The implementation is available at https://github.com/prafful-kumar/enhancing_TE.git

Unsupervised domain adaptation (UDA) aims to leverage the knowledge learned from labeled source domains to improve performance on the unlabeled target domains. While Convolutional Neural Networks (CNNs) have been dominant in previous UDA methods recent research has shown promise in applying Vision Transformers (ViTs) to this task. In this study we propose a novel Feature Fusion Transferability Aware Transformer (FFTAT) to enhance ViT performance in UDA tasks. Our method introduces two key innovations: First we introduce a patch discriminator to evaluate the transferability of patches generating a transferability matrix. We integrate this matrix into self-attention directing the model to focus on transferable patches. Second we propose a feature fusion technique to fuse embeddings in the latent space enabling each embedding to incorporate information from all others thereby improving generalization. These two components work in synergy to enhance feature representation learning. Extensive experiments on widely used benchmarks demonstrate that our method significantly improves UDA performance achieving state-of-the-art (SOTA) results.

Referring Image Segmentation (RIS) is a task aimed at segmenting objects expressed in natural language within an image. This task requires an understanding of the relationship between vision and language along with precise segmentation capabilities. In the field of computer vision CLIP and Segment anything model (SAM) have gained significant attention for their classification and the segmentation capabilities. Given that both models possess essential skills for RIS combining them seems to be an effective strategy. In this paper we propose a model that integrates CLIP and SAM to enhance RIS. Since SAM lacks classification capabilities we developed a module that supplies the SAM mask decoder with features that specify the target object. We introduce a new module which is trained on additional instance segmentation tasks. The features utilized and derived from this module serve as inputs for the SAM decoder. With these inputs SAM is expected to effectively segment areas corresponding to the given natural language expressions. We conducted experiments using the traditional RefCOCO/+/g as well as the recently introduced gRefCOCO and Ref-zom datasets demonstrating the advantages of our approach. Code will be available on https://github.com/hitachi-rd-cv/dfam.

Test-time adaptation (TTA) allows a model to be adapted to an unseen domain without accessing the source data. Due to the nature of practical environments TTA has a limited amount of data for adaptation. Recent TTA methods further restrict this by filtering input data for reliability making the effective data size even smaller and limiting adaptation potential. To address this issue We propose Feature Augmentation based Test-time Adaptation (FATA) a simple method that fully utilizes the limited amount of input data through feature augmentation. FATA employs Normalization Perturbation to augment features and adapts the model using the FATA loss which makes the outputs of the augmented and original features similar. FATA is model-agnostic and can be seamlessly integrated into existing models without altering the model architecture. We demonstrate the effectiveness of FATA on various models and scenarios on ImageNet-C and Office-Home validating its superiority in diverse real-world conditions. Code is available at https://github.com/RangeWING/FATA.

Segmentation models for brain lesions in MRI are commonly developed for a specific disease and trained on data with a predefined set of MRI modalities. Such models cannot segment the disease using data with a different set of MRI modalities nor can they segment other types of diseases. Moreover this training paradigm prevents a model from using the advantages of learning from heterogeneous databases that may contain scans and segmentation labels for different brain pathologies and diverse sets of MRI modalities. Additionally the confidentiality of patient data often prevents central data aggregation necessitating a decentralized approach. Is it feasible to use Federated Learning (FL) to train a single model on client databases that contain scans and labels of different brain pathologies and diverse sets of MRI modalities? We demonstrate promising results by combining appropriate simple and practical modifications to the model and training strategy: Designing a model with input channels that cover the whole set of modalities available across clients training with random modality drop and exploring the effects of feature normalization methods. Evaluation on 7 brain MRI databases with 5 different diseases shows that this FL framework can train a single model that achieves very promising results in segmenting all disease types seen during training. Importantly it can segment these diseases in new databases that contain sets of modalities different from those in training clients. These results demonstrate for the first time the feasibility and effectiveness of using FL to train a single 3D segmentation model on decentralised data with diverse brain diseases and MRI modalities a necessary step towards leveraging heterogeneous real-world databases. Code: https://github.com/FelixWag/FedUniBrain

Diffusion models have demonstrated remarkable capabilities in text-to-image and text-to-video generation opening up possibilities for video editing based on textual input. However the computational cost associated with sequential sampling in diffusion models poses challenges for efficient video editing. Existing approaches relying on image generation models for video editing suffer from time-consuming one-shot fine-tuning additional condition extraction or DDIM inversion making real-time applications impractical. In this work we propose FastVideoEdit an efficient zero-shot video editing approach inspired by Consistency Models (CMs). By leveraging the self-consistency property of CMs we eliminate the need for time-consuming inversion or additional condition extraction reducing editing time. Our method enables direct mapping from source video to target video with strong preservation ability through attention control. This results in improved speed advantages as fewer sampling steps can be used while maintaining comparable generation quality. Experimental results validate the state-of-the-art performance and speed advantages of FastVideoEdit across evaluation metrics encompassing editing speed temporal consistency and text-video alignment. The source code is available at github.com/youyuan-zhang/FastVideoEdit.

Face recognition and verification are two computer vision tasks whose performances have advanced with the introduction of deep representations. However ethical legal and technical challenges due to the sensitive nature of face data and biases in real-world training datasets hinder their development. Generative AI addresses privacy by creating fictitious identities but fairness problems remain. Using the existing DCFace SOTA framework we introduce a new controlled generation pipeline that improves fairness. Through classical fairness metrics and a proposed in-depth statistical analysis based on logit models and ANOVA we show that our generation pipeline improves fairness more than other bias mitigation approaches while slightly improving raw performance.

Domain generalization(DG) techniques classify data from unseen domains by capitalizing on data from multiple source domains. Most methods in DG focus on improving predictive performance measures on the unseen domain. Recent studies have also attempted to enforce fairness measures on the unseen domain. However these studies assume that every domain has the same sensitive attribute including the unseen domain. In practice each domain may be required to satisfy fairness with respect to its own set of multiple sensitive attributes. Given a set of sensitive attributes (S) current methods need to train 2^n models to ensure fairness with respect to any subset of S where n = |S|. We propose a single-model solution to address this new problem setting. We learn two feature representations one to generalize the model's predictive performance and another to generalize the model's fairness. The first representation is made invariant across domains to generalize predictive performance. The second representation is kept selectively invariant i.e. invariant only across domains having the same sensitive attributes. Our single model exhibits superior predictive performance and fairness measures against the current alternative of 2^n models on unseen domains on multiple real-world vision datasets. Our code is available at https://github.com/ragjapk/SISA.

Facial expression recognition (FER) systems raise significant privacy concerns due to the potential exposure of sensitive identity information. This paper presents a study on removing identity information while preserving FER capabilities. Drawing on the observation that low-frequency components predominantly contain identity information and high-frequency components capture expression we propose a novel two-stream framework that applies privacy enhancement to each component separately. We introduce a controlled privacy enhancement mechanism to optimize performance and a feature compensator to enhance task-relevant features without compromising privacy. Furthermore we propose a novel privacy-utility trade-off providing a quantifiable measure of privacy preservation efficacy in closed-set FER tasks. Extensive experiments on the benchmark CREMA-D dataset demonstrate that our framework achieves 78.84% recognition accuracy with a privacy (facial identity) leakage ratio of only 2.01% highlighting its potential for secure and reliable video-based FER applications.