WACV 2025 Papers

Enabling generative models to decompose visual concepts from a single image is a complex and challenging problem. In this paper we study a new and challenging task customized concept decomposition wherein the objective is to leverage diffusion models to decompose a single image and generate visual concepts from various perspectives. To address this challenge we propose a two-stage framework CusConcept (short for Customized Visual Concept Decomposition) to extract customized visual concept embedding vectors that can be embedded into prompts for text-to-image generation. In the first stage CusConcept employs a vocabulary-guided concept decomposition mechanism to build vocabularies along human-specified conceptual axes. The decomposed concepts are obtained by retrieving corresponding vocabularies and learning anchor weights. In the second stage joint concept refinement is performed to enhance the fidelity and quality of generated images. We further curate an evaluation benchmark for assessing the performance of the open-world concept decomposition task. Our approach can effectively generate high-quality images of the decomposed concepts and produce related lexical predictions as secondary outcomes. Extensive qualitative and quantitative experiments demonstrate the effectiveness of CusConcept. Our code and data are available at https://github.com/xzLcan/CusConcept.

Cryo-electron microscopy (cryo-EM) emerges as a pivotal technology for determining the architecture of cells viruses and protein assemblies at near-atomic resolution. Traditional particle picking a key step in cryo-EM struggles with manual effort and automated methods' sensitivity to low signal-to-noise ratio (SNR) and varied particle orientations. Furthermore existing neural network (NN)-based approaches often require extensive labeled datasets limiting their practicality. To overcome these obstacles we introduce cryoMAE a novel approach based on few-shot learning that harnesses the capabilities of Masked Autoencoders (MAE) to enable efficient selection of single particles in cryo-EM images. Contrary to conventional NN-based techniques cryoMAE requires only a minimal set of positive particle images for training yet demonstrates high performance in particle detection. Furthermore the implementation of a self-cross similarity loss ensures distinct features for particle and background regions thereby enhancing the discrimination capability of cryoMAE. Experiments on large-scale cryo-EM datasets show that cryoMAE outperforms existing state-of-the-art (SOTA) methods improving 3D reconstruction resolution by up to 22.4%. Our code is available at: https://github.com/xulabs/aitom.

A crowd density forecasting task aims to predict how the crowd density map will change in the future from observed past crowd density maps. However the past crowd density maps are often incomplete due to the miss-detection of pedestrians and it is crucial to develop a robust crowd density forecasting model against the miss-detection. This paper presents a MAsked crowd density Completion framework for crowd density forecasting (CrowdMAC) which is simultaneously trained to forecast future crowd density maps from partially masked past crowd density maps (i.e. forecasting maps from past maps with miss-detection) while reconstructing the masked observation maps (i.e. imputing past maps with miss-detection). Additionally we propose Temporal-Density-aware Masking (TDM) which non-uniformly masks tokens in the observed crowd density map considering the sparsity of the crowd density maps and the informativeness of the subsequent frames for the forecasting task. Moreover we introduce multi-task masking to enhance training efficiency. In the experiments CrowdMAC achieves state-of-the-art performance on seven large-scale datasets including SDD ETH-UCY inD JRDB VSCrowd FDST and croHD. We also demonstrate the robustness of the proposed method against both synthetic and realistic miss-detections. The code is released at https://fujiry0.github.io/CrowdMAC-project-page.

We present a comprehensive three-phase study to examine (1) the cultural understanding of Large Multimodal Models (LMMs) by introducing DALLE STREET a large-scale dataset generated by DALL-E 3 and validated by humans containing 9935 images of 67 countries and 10 concept classes; (2) the underlying implicit and potentially stereotypical cultural associations with a cultural artifact extraction task; and (3) an approach to adapt cultural representation in an image based on extracted associations using a modular pipeline CULTUREADAPT. We find disparities in cultural understanding at geographic sub-region levels with both open-source (LLaVA) and closed-source (GPT-4V) models on DALLE STREET and other existing benchmarks which we try to understand using over 18000 artifacts that we identify in association to different countries. Our findings reveal a nuanced picture of the cultural competence of LMMs highlighting the need to develop culture-aware systems.

Aerial imagery analysis is critical for many research fields. However obtaining frequent high-quality aerial images is not always accessible due to its high effort and cost requirements. One solution is to use the Ground-to-Aerial (G2A) technique to synthesize aerial images from easily collectible ground images. However G2A is rarely studied because of its challenges including but not limited to the drastic view changes occlusion and range of visibility. In this paper we present a novel Geometric Preserving Ground-to-Aerial (G2A) image synthesis (GPG2A) model that can generate realistic aerial images from ground images. GPG2A consists of two stages. The first stage predicts the Bird's Eye View (BEV) segmentation (referred to as the BEV layout map) from the ground image. The second stage synthesizes the aerial image from the predicted BEV layout map and text descriptions of the ground image. To train our model we present a new multi-modal cross-view dataset namely VIGORv2 built upon VIGOR with newly collected aerial images maps and text descriptions. Our extensive experiments illustrate that GPG2A synthesizes better geometry-preserved aerial images than existing models. We also present two applications data augmentation for cross-view geo-localization and sketch-based region search to further verify the effectiveness of our GPG2A. The code and dataset are available at https://github.com/AhmadArrabi/GPG2A

Multitask learning (MTL) has become prominent for its ability to predict multiple tasks jointly achieving better per-task performance with fewer parameters than single-task learning. Recently decoder-focused architectures have significantly improved multitask performance by refining task predictions using features from related tasks. However most refinement methods struggle to efficiently capture both local and long-range dependencies between task-specific representations and cross-task patterns. In this paper we introduce the Cross-Task Affinity Learning (CTAL) module a lightweight framework that enhances task refinement in multitask networks. CTAL effectively captures local and long-range cross-task interactions by optimizing task affinity matrices for parameter-efficient grouped convolutions without concern for information loss. Our results demonstrate state-of-the-art MTL performance for both CNN and transformer backbones using significantly fewer parameters than single-task learning.

Prompt Tuning has emerged as a prominent research paradigm for adapting vision-language models to various downstream tasks. However recent research indicates that prompt tuning methods often lead to overfitting due to limited training samples. In this paper we propose a Cross-modal Aligned Feature Tuning (CRAFT) method to address this issue. Cross-modal alignment is conducted by first selecting anchors from the alternative domain and deriving relative representations of the embeddings for the selected anchors. Optimizing for a feature alignment loss over anchor-aligned text and image modalities creates a more unified text-image common space. Overfitting in prompt tuning also deteriorates model performance on out-of-distribution samples. To further improve the prompt model's robustness we propose minimizing Maximum Mean Discrepancy (MMD) over the anchor-aligned feature spaces to mitigate domain shift. The experiment on four different prompt tuning structures consistently shows the improvement of our method with increases of up to 6.1% in the Base-to-Novel generalization task 5.8% in the group robustness task and 2.7% in the out-of-distribution tasks. The code is available at https://github.com/Jingchensun/Craft.

Direct image-to-graph transformation is a challenging task solving object detection and relationship prediction in a single model. Due to this task's complexity large training datasets are rare in many domains making the training of deep-learning methods challenging. This data sparsity necessitates transfer learning strategies akin to the state-of-the-art in general computer vision. In this work we introduce a set of methods enabling cross-domain and cross-dimension learning for image-to-graph transformers. We propose (1) a regularized edge sampling loss to effectively learn object relations in multiple domains with different numbers of edges (2) a domain adaptation framework for image-to-graph transformers aligning image- and graph-level features from different domains and (3) a projection function that allows using 2D data for training 3D transformers. We demonstrate our method's utility in cross-domain and cross-dimension experiments where we utilize labeled data from 2D road networks for simultaneous learning in vastly different target domains. Our method consistently outperforms standard transfer learning and self-supervised pretraining on challenging benchmarks such as retinal or whole-brain vessel graph extraction.

Cross-modal feature extraction and integration have led to steady performance improvements in few-shot learning tasks. However existing multi-modal object detection (MM-OD) methods degrade when facing significant domain shift and are sample insufficient. We hypothesize that rich text information could more effectively help the model to build a knowledge relationship between the vision instance and its language description and can help mitigate domain shift. Specifically we study the Cross-Domain few-shot generalization of MM-OD (CDMM-FSOD) and propose a meta-learning based multi-modal few-shot object detection method that utilizes rich text semantic information as an auxiliary modality to achieve domain adaptation. Our proposed novel neural network contains a multi-modal feature aggregation module that aligns the vision and language support feature embeddings and a rich text semantic rectify module that utilizes bidirectional text feature generation to reinforce multi-modal feature alignment and thus to enhance the model's language understanding capability. We evaluate our model on common standard cross-domain object detection datasets and demonstrate that our approach considerably outperforms existing FSOD methods. Our implementation is publicly available: https://github.com/zshanggu/CDMM

Recent multimodal frameworks often grapple with semantic misalignment and noise impeding effective integration of diverse modalities. In order to solve this problem this study presents CaMN (Cross-aligned Multimodal Network) a framework designed to enhance multimodal understanding through a robust cross-alignment mechanism. Unlike conventional fusion methods our framework aligns features extracted from images text and graphs via a tailored loss function enabling seamless integration and exploitation of complementary information. Leveraging Abstract Meaning Representation (AMR) we extract intricate semantic structures from textual data enriching the multimodal representation with contextual depth. Furthermore to enhance robustness we employ a masked autoencoder to simulate noise-independent feature space. Through comprehensive evaluation on the crisisMMD dataset CaMN demonstrates superior performance in crisis event classification tasks highlighting its potential in advancing multimodal understanding across diverse domains. Our code is available at https://github.com/brillard1/CaMN.

Semi-supervised medical image segmentation aims to leverage a limited set of labeled images alongside a substantial volume of unlabeled images to train semantic segmentation models. Existing studies often employ consistency regularization to maximize the utilization of unlabeled data thereby enhancing the model's robustness and accuracy. However the methods for constructing perturbations at image-level on unlabeled data are typically simplistic involving techniques such as color transformations additive noise which do not adequately leverage the precise and reliable supervisory information available from labeled images. To address this limitation in addition to image perturbation we propose a cross-image feature perturbation approach for semi-supervised medical image segmentation. This method utilizes feature information from labeled images to guide the refinement of ambiguous semantic representations in unlabeled images thereby expanding the perturbation space more effectively. Moreover recognizing the limitations of existing consistency regularization frameworks that rely on confidence thresholds to filter pseudo-labels we introduce an uncertainty-based pseudo-label fusion strategy. This strategy mitigates the effects of unreliable predictions caused by perturbations by calculating the uncertainty and using it as a weight during pseudo-label fusion. We have conducted extensive experiments on the 2D ACDC and 3D LA datasets. The results demonstrate that our approach achieves performance comparable to the current state-of-the-art (SOTA) methods.

Single-source domain generalization attempts to learn a model on a source domain and deploy it to unseen target domains. Limiting access only to source domain data imposes two key challenges - how to train a model that can generalize and how to verify that it does. The standard practice of validation on the training distribution does not accurately reflect the model's generalization ability while validation on the test distribution is a malpractice to avoid. In this work we construct an independent validation set by transforming source domain images with a comprehensive list of augmentations covering a broad spectrum of potential distribution shifts in target domains. We demonstrate a high correlation between validation and test performance for multiple methods and across various datasets. The proposed validation achieves a relative accuracy improvement over the standard validation equal to 15.4% or 1.6% when used for method selection or learning rate tuning respectively. Furthermore we introduce a novel family of methods that increase the shape bias through enhanced edge maps. To benefit from the augmentations during training and preserve the independence of the validation set a k-fold validation process is designed to separate the augmentation types used in training and validation. The method that achieves the best performance on the augmented validation is selected from the proposed family. It achieves state-of-the-art performance on various standard benchmarks. Code at: https://github.com/NikosEfth/crafting-shifts

While research on structural crack segmentation at the image level remains highly active progress beyond two dimensions has been limited. This stagnation is largely due to the lack of available data for crack detection in higher dimensions. To address this limitation we introduce CrackStructures a dataset tailored for real-world 2.5D crack segmentation encompassing 15 segments from five distinct structures. Additionally we present CrackEnsembles a complementary semi-synthetic dataset that combines real textures with synthetic geometry to enhance the development of learning-based algorithms. Coupled with a baseline for multi-view crack instance segmentation this work establishes a solid foundation for advancing algorithms that support real-world structural inspection.

Few-shot Class Incremental Learning (FSCIL) presents a challenging yet realistic scenario which requires the model to continually learn new classes with limited labeled data (i.e. incremental sessions) while retaining knowledge of previously learned base classes (i.e. base sessions). Due to the limited data in incremental sessions models are prone to overfitting new classes and suffering catastrophic forgetting of base classes. To tackle these issues recent advancements resort to prototype-based approaches to constrain the base class distribution and learn discriminative representations of new classes. Despite the progress the limited data issue still induces ill-divided feature space leading the model to confuse the new class with old classes or fail to facilitate good separation among new classes. In this paper we aim to mitigate these issues by directly constraining the span of each class distribution from a covariance perspective. In detail we propose a simple yet effective covariance constraint loss to force the model to learn each class distribution with the same covariance matrix. In addition we propose a perturbation approach to perturb the few-shot training samples in the feature space which encourages the samples to be away from the weighted distribution of other classes. Regarding perturbed samples as new class data the classifier is forced to establish explicit boundaries between each new class and the existing ones. Our approach is easy to integrate into existing FSCIL approaches to boost performance. Experiments on three benchmarks validate the effectiveness of our approach achieving state-of-the-art performance.

Recently there have been significant improvements in the quality and performance of text-to-image generation largely due to the impressive results attained by diffusion models. However text-to-image diffusion models sometimes struggle to create high-fidelity content for the given input prompt. One specific issue is their difficulty in generating the precise number of objects specified in the text prompt. For example when provided with the prompt "five apples and ten lemons on a table" images generated by diffusion models often contain an incorrect number of objects. In this paper we present a method to improve diffusion models so that they accurately produce the correct object count based on the input prompt. We adopt a counting network that performs reference-less class-agnostic counting for any given image. We calculate the gradients of the counting network and refine the predicted noise for each step. To address the presence of multiple types of objects in the prompt we utilize novel attention map guidance to obtain high-quality masks for each object. Finally we guide the denoising process using the calculated gradients for each object. Through extensive experiments and evaluation we demonstrate that the proposed method significantly enhances the fidelity of diffusion models with respect to object count.

In the task of reference-based image inpainting an additional reference image is provided to restore a damaged target image to its original state. The advancement of diffusion models particularly Stable Diffusion allows for simple formulations in this task. However existing diffusion-based methods often lack explicit constraints on the correlation between the reference and damaged images resulting in lower faithfulness to the reference images in the inpainting results. In this work we propose CorrFill a training-free module designed to enhance the awareness of geometric correlations between the reference and target images. This enhancement is achieved by guiding the inpainting process with correspondence constraints estimated during inpainting utilizing attention masking in self-attention layers and an objective function to update the input tensor according to the constraints. Experimental results demonstrate that CorrFill significantly enhances the performance of multiple baseline diffusion-based methods including state-of-the-art approaches by emphasizing faithfulness to the reference images.

Text-to-Video generation has achieved remarkable progress with the rise of diffusion models. In this work we introduce Cached Memory-Guided Video Generation (Corgi) aiming to generate multi-scene videos with arbitrary number of video clips conditioned on input images and instruction prompts. This is a challenging task as traditional T2V methods often struggle to maintain the quality of longer videos due to the difficulties in preserving visual context from earlier scenes. We address this by introducing a cached memory mechanism that stores the key frames. Our multi-scene video generation process is explicitly conditioned on the cached memories to avoid forgetting the visual appearance of target subjects. Corgi shows significant improvement in multi-scene video generation compared to the prior art with up to 59.2% in long-term consistency and 7.6% in diversity.

Reference-guided face restoration can have better identity preservation than non-reference-based methods. However existing methods can (a) easily produce artifacts possibly attributable to inefficient facial priors and (b) do not well preserve fine-grained facial details crucial for identity such as freckles tattoos and scars. In this work we propose solutions for these problems. (1) We incorporate a stronger facial prior generative facial prior (GFP) for reference-based face image restoration. (2) We identify an ambiguity and point out that traditional loss prevents the network from heavily copying facial features from the reference. To address this we set a new goal and come up with a new loss to realize the new goal. More specifically when the ground truth and reference are different (e.g. differences in wrinkles makeup facial hair etc.) which one should the output look like? As a simple example ground truth does not have a mole while reference has one. Traditional loss chose the ground truth which seems natural but then the network also learns to ignore reference's facial features; during testing the network often hesitates. Our new goal is to copy features from the reference as much as possible while maintaining semantic consistency with the degraded input. We propose to use spatial minimum loss and cycle consistency loss to realize the new goal and make the network copy features without hesitation. Using only a single reference image our proposed method is able to restore highly degraded images while accurately capturing fine-grained facial details. To our knowledge we are the first face restoration framework that is able to restore faces at this granularity. Code and data are available at https://github.com/RefineFIR/RefineFIR.

Transformer models have demonstrated remarkable success in many domains such as natural language processing (NLP) and computer vision. With the growing interest in transformer-based architectures they are now utilized for gesture recognition. So we also explore and devise a novel ConvMixFormer architecture for dynamic hand gestures. The transformers use quadratic scaling of the attention features with the sequential data due to which these models are computationally complex and heavy. We have considered this drawback of the transformer and designed a resource-efficient model that replaces the self-attention in the transformer with the simple convolutional layer-based token mixer. The computational cost and the parameters used for the convolution-based mixer are comparatively less than the quadratic self-attention. Convolution-mixer helps the model capture the local spatial features that self-attention struggles to capture due to their sequential processing nature. Further an efficient gate mechanism is employed instead of a conventional feed-forward network in the transformer to help the model control the flow of features within different stages of the proposed model. This design uses fewer learnable parameters which is nearly half the vanilla transformer that helps in fast and efficient training. The proposed method is evaluated on NVidia Dynamic Hand Gesture and Briareo datasets and our model has achieved state-of-the-art results on single and multimodal inputs. We have also shown the parameter efficiency of the proposed ConvMixFormer model compared to other methods. The source code is available at https://github.com/mallikagarg/ConvMixFormer.