WACV 2025 Papers

Generated video scenes for action-centric sequence descriptions such as recipe instructions and do-it-yourself projects often include non-linear patterns where the next video may need to be visually consistent not with the immediately preceding video but with earlier ones. Current multi-scene video synthesis approaches fail to meet these consistency requirements. To address this we propose a contrastive sequential video diffusion method that selects the most suitable previously generated scene to guide and condition the denoising process of the next scene. The result is a multi-scene video that is grounded in the scene descriptions and coherent w.r.t. the scenes that require visual consistency. Experiments with action-centered data from the real world demonstrate the practicality and improved consistency of our model compared to previous work. Code and examples available at https://github.com/novasearch/CoSeD

The spatial information contained in images is of critical importance for many computer vision tasks. Current state-of-the-art approaches dealing with spatially-related tasks such as spatial relationship recognition are typically trained in a supervised manner with semantic information carried by annotations. However datasets containing spatial relations (such as VisualGenome and SpatialSense) contain many errors or ambiguities at the label level (e.g. polysemy of the relations use of different reference frames across relations) which might deteriorate the representation learning step. The representations and image embeddings obtained from this training setup carry poor spatial information as they are entangled with other modalities such as semantic information. To deal with this issue we introduce C-SIP (Contrastive Spatial-Image Pre-training) an approach aiming to learn better spatially-aware image representations more in agreement with human perception of a scene where spatial information is structuring. This training strategy focuses on the alignment of the embeddings of an image encoder and a spatial encoder optimized from the image content in a self-supervised manner. We showcase that training a model with spatial information at its core thanks to C-SIP allows for better spatially-aware image representations on three downstream tasks. These representations can be used in a zero-shot setting such as image retrieval or fine-tuned on semantic tasks such as visual question answering and provide better results compared with supervised counterparts. Our source code can be found at https://github.com/Logan-wilson/CSIP.

Current cardiac cine magnetic resonance image (cMR) studies focus on the end diastole (ED) and end systole (ES) phases while ignoring the abundant temporal information in the whole image sequence. This is because whole sequence segmentation is currently a tedious process and inaccurate. Conventional whole sequence segmentation approaches first estimate the motion field between frames which is then used to propagate the mask along the temporal axis. However the mask propagation results could be prone to error especially for the basal and apex slices where through-plane motion leads to significant morphology and structural change during the cardiac cycle. Inspired by recent advances in video object segmentation (VOS) based on spatio-temporal memory (STM) networks we propose a continuous STM (CSTM) network for semi-supervised whole heart and whole sequence cMR segmentation. Our CSTM network takes full advantage of the spatial scale temporal and through-plane continuity prior of the underlying heart anatomy structures to achieve accurate and fast 4D segmentation. Results of extensive experiments across multiple cMR datasets show that our method can improve the 4D cMR segmentation performance especially for the hard-to-segment regions. Project page is at https://github.com/DeepTag/CSTM.

We introduce a novel continual learning problem: how to sequentially update the weights of a personalized 2D and 3D generative face model as new batches of photos in different appearances styles poses and lighting are captured regularly. We observe that naive sequential fine-tuning of the model leads to catastrophic forgetting of past representations of the individual's face. We then demonstrate that a simple random sampling-based experience replay method is effective at mitigating catastrophic forgetting when a relatively large number of images can be stored and replayed. However for long-term deployment of these models with relatively smaller storage this simple random sampling-based replay technique also forgets past representations. Thus we introduce a novel experience replay algorithm that combines random sampling with StyleGAN's latent space to represent the buffer as an optimal convex hull. We observe that our proposed convex hull-based experience replay is more effective in preventing forgetting than a random sampling baseline and the lower bound. We introduce continual learning datasets for five celebrities along with the evaluation framework metrics and visualizations to examine this problem. See our project page for more details.

We introduce a novel framework for Continual learning in 3D object classification. Our approach is based on the selection of prototypes from each class using spectral clustering. For non-Euclidean data such as point clouds spectral clustering can be employed as long as one can define a distance measure between pairs of samples. Choosing the appropriate distance measure enables us to leverage 3D geometric characteristics to identify representative prototypes for each class. We explore the effectiveness of clustering in the input space (3D points) local feature space (1024-dimensional points) and global feature space. We conduct experiments on the ModelNet40 ShapeNet and ScanNet datasets achieving state-of-the-art accuracy exclusively through the use of input space features. By leveraging the combined input local and global features we have improved the state-of-the-art on ModelNet40 and ShapeNet utilizing nearly half the memory used by competing approaches. For the challenging ScanNet dataset our method enhances accuracy by 4.1% while consuming just 28% of the memory used by our competitors demonstrating the scalability of our approach.

Contextual advertising serves ads that are aligned to the content that the user is viewing. The rapid growth of video content on social platforms and streaming services along with privacy concerns has increased the need for contextual advertising. Placing the right ad in the right context creates a seamless and pleasant ad viewing experience resulting in higher audience engagement and ultimately better ad monetization. From a technology standpoint effective contextual advertising requires a video retrieval system capable of understanding complex video content at a very granular level. Current text-to-video retrieval models based on joint multimodal training demand large datasets and computational resources limiting their practicality and lacking the key functionalities required for ad ecosystem integration. We introduce ContextIQ a multimodal expert-based video retrieval system designed specifically for contextual advertising. ContextIQ utilizes modality-specific experts--video audio transcript (captions) and metadata such as objects actions emotion etc.--to create semantically rich video representations. We show that our system without joint training achieves better or comparable results to state-of-the-art models and commercial solutions on multiple text-to-video retrieval benchmarks. Our ablation studies highlight the benefits of leveraging multiple modalities for enhanced video retrieval accuracy instead of using a vision-language model alone. Furthermore we show how video retrieval systems such as ContextIQ can be used for contextual advertising in an ad ecosystem while also addressing concerns related to brand safety and filtering inappropriate content.

This paper presents a novel approach for robust 3D tracking of multiple birds in an outdoor aviary using a multi-camera system. Our method addresses the significant challenges posed by the visual similarity of birds and the complexities of their rapid movements in three-dimensional space. We leverage environmental context called landmarks to enhance feature matching between cameras leading to more accurate 3D reconstructions and tracking of their positions. Removing outliers in multi-view settings is especially challenging because the objects of interest are visually similar. In our approach outliers are rejected based on their nearest landmark. This leads to constructing precise 3D models of individual birds; additionally it applies these models to track multiple birds simultaneously. By utilizing environmental context our approach significantly improves the differentiation between visually similar birds a key obstacle in existing tracking systems. Experimental results demonstrate the effectiveness of our method showing a 20% elimination of outliers in the 3D reconstruction process with a 97% accuracy in matching. This remarkable accuracy in 3D modeling translates to robust and reliable tracking of multiple birds even in challenging outdoor conditions. Our work not only advances the field of computer vision but also provides a valuable tool for studying bird behavior and movement patterns in natural settings. We also provide a large annotated dataset of 80 birds residing in four enclosures for 20 hours of footage which provides a rich testbed for researchers in computer vision ornithologists and ecologists. Code and the link to the dataset is available at https://github.com/airou-lab/3D_Multi_Bird_Tracking.

Retinal fundus photography offers a non-invasive way to diagnose and monitor a variety of retinal diseases but is prone to inherent quality glitches arising from systemic imperfections or operator/patient-related factors. However high-quality retinal images are crucial for carrying out accurate diagnoses and automated analyses. The fundus image enhancement is typically formulated as a distribution alignment problem by finding a one-to-one mapping between a low-quality image and its high-quality counterpart. This paper proposes a context-informed optimal transport (OT) learning framework for tackling unpaired fundus image enhancement. In contrast to standard generative image enhancement methods which struggle with handling contextual information (e.g. over-tampered local structures and unwanted artifacts) the proposed context-aware OT learning paradigm better preserves local structures and minimizes unwanted artifacts. Leveraging deep contextual features we derive the proposed context-aware OT using the earth mover's distance and show that the proposed context-OT has a solid theoretical guarantee. Experimental results on a large-scale dataset demonstrate the superiority of the proposed method over several state-of-the-art supervised and unsupervised methods in terms of signal-to-noise ratio structural similarity index as well as two downstream tasks. The code is available at https://github.com/Retinal-Research/Contextual-OT.

Audio-driven simultaneous gesture generation is vital for human-computer communication AI games and film production. While previous research has shown promise there are still limitations. Methods based on VAEs are accompanied by issues of local jitter and global instability whereas methods based on diffusion models are hampered by low generation efficiency. This is because the denoising process of DDPM in the latter relies on the assumption that the noise added at each step is sampled from a unimodal distribution and the noise values are small. DDIM borrows the idea from the Euler method for solving differential equations disrupts the Markov chain process and increases the noise step size to reduce the number of denoising steps thereby accelerating generation. However simply increasing the step size during the step-by-step denoising process causes the results to gradually deviate from the original data distribution leading to a significant drop in the quality of the generated actions and the emergence of unnatural artifacts. In this paper we break the assumptions of DDPM and achieves breakthrough progress in denoising speed and fidelity. Specifically we introduce a conditional GAN to capture audio control signals and implicitly match the multimodal denoising distribution between the diffusion and denoising steps within the same sampling step aiming to sample larger noise values and apply fewer denoising steps for high-speed generation. In addition to enable the model to generate high-fidelity global gestures and avoid artifacts we introduce an explicit motion geometric loss to enhance the quality and global stability of the generated gestures. Numerous qualitative and quantitative experiments show that compared to contemporary diffusion-based methods our method offers faster generation speed and higher fidelity and compared to non-diffusion methods it provides a more stable global effect and a more natural user experience.

Concept bottleneck models (CBMs) have emerged as critical tools in domains where interpretability is paramount. These models rely on predefined textual descriptions referred to as concepts to inform their decision-making process and offer more accurate reasoning. As a result the selection of concepts used in the model is of utmost significance. This study proposes Conceptual Learning via Embedding Approximations for Reinforcing Interpretability and Transparency abbreviated as CLEAR a framework for constructing a CBM for image classification. Using score matching and Langevin sampling we approximate the embedding of concepts within the latent space of a vision-language model (VLM) by learning the scores associated with the joint distribution of images and concepts. A concept selection process is then employed to optimize the similarity between the learned embeddings and the predefined ones. The derived bottleneck offers insights into the CBM's decision-making process enabling more comprehensive interpretations. Our approach was evaluated through extensive experiments and achieved state-of-the-art performance on various benchmarks. The code for our experiments is available at https://github.com/clearProject/CLEAR/tree/main.

Medical data often exhibits distribution shifts leading to performance degradation of deep learning models trained using standard supervised learning pipelines. Domain Generalization (DG) addresses this challenge with Single-Domain Generalization (SDG) being notably relevant due to the privacy and logistical constraints often inherent in medical data. Existing disentanglement-based SDG methods heavily rely on structural information from segmentation masks but classification labels do not offer similarly dense information. This work introduces a novel SDG method for medical image classification utilizing channel-wise contrastive disentanglement. The method is further refined with reconstruction-based style regularization to ensure distinct style and structural feature representations are extracted. We evaluate our method on the complex tasks of multicenter histopathology image classification and Diabetic Retinopathy (DR) grading in fundus images benchmarking it against state-of-the-art (SOTA) SDG baselines. Our results demonstrate that our method consistently outperforms the SOTA independently on the choice of the source domain while exhibiting greater performance stability. This study underscores the importance and challenges of exploring SDG frameworks for classification tasks. The code is publicly available at https://github.com/BioMedIA-MBZUAI/ConDiSR

Cardiac image segmentation is essential for automated cardiac function assessment and monitoring of changes in cardiac structures over time. Inspired by coarse-to-fine approaches in image analysis we propose a novel multitask compositional segmentation approach that can simultaneously localize the heart in a cardiac image and perform part-based segmentation of different regions of interest. We demonstrate that this compositional approach achieves better results than direct segmentation of the anatomies. Further we propose a novel Cross-Modal Feature Integration (CMFI) module to leverage the metadata related to cardiac imaging collected during image acquisition. We perform experiments on two different modalities MRI and ultrasound using public datasets Multi-Disease Multi-View and Multi-Centre (M&Ms-2) and Multi-structure Ultrasound Segmentation (CAMUS) data to showcase the efficiency of the proposed compositional segmentation method and Cross-Modal Feature Integration module incorporating metadata within the proposed compositional segmentation network. The source code is available: https://github.com/kabbas570/CompSeg-MetaData.

This work addresses composed image retrieval in the context of domain conversion where the content of a query image is retrieved in the domain specified by the query text. We show that a strong vision-language model provides sufficient descriptive power without additional training. The query image is mapped to the text input space using textual inversion. Unlike common practice that invert in the continuous space of text tokens we use the discrete word space via a nearest-neighbor search in a text vocabulary. With this inversion the image is softly mapped across the vocabulary and is made more robust using retrieval-based augmentation. Database images are retrieved by a weighted ensemble of text queries combining mapped words with the domain text. Our method outperforms prior art by a large margin on standard and newly introduced benchmarks. Code: https://github.com/NikosEfth/freedom

In the era of large-scale pretrained models Knowledge Distillation (KD) serves an important role in transferring the wisdom of computationally-heavy teacher models to lightweight efficient student models while preserving performance. Yet KD settings often assume readily available access to teacher models capable of performing many inferences -- a notion increasingly at odds with the realities of costly large-scale models. Addressing this gap we study an important question: how KD algorithms fare as the number of teacher inferences decreases a setting we term Reduced-Teacher-Inference Knowledge Distillation (RTI-KD). We observe that the performance of prevalent KD techniques and state-of-the-art data augmentation strategies suffers considerably as the number of teacher inferences is reduced. One class of approaches termed "relational" knowledge distillation underperforms the rest yet we hypothesize that they hold promise for reduced dependency on teacher models because they can augment the effective dataset size without additional teacher calls. We find that a simple change -- performing high-dimensional comparisons instead of low-dimensional relations which we term "Comparative Knowledge Distillation" -- vaults performance well over existing KD approaches. We perform empirical evaluation across varied experimental settings and rigorous analysis to understand the learning outcomes of our method. All code is made publicly available.

Current generalizable object pose estimators i.e. approaches that do not need to be trained per object rely on accurate 3D models. Predominantly CAD models are used which can be hard to obtain in practice. At the same time it is often possible to acquire images of an object. Naturally this leads to the question of whether 3D models reconstructed from images are sufficient to facilitate accurate object pose estimation. We aim to answer this question by proposing a novel benchmark for measuring the impact of 3D reconstruction quality on pose estimation accuracy. Our benchmark provides calibrated images suitable for reconstruction and registered with the test images of the YCB-V dataset for pose evaluation under the BOP benchmark format. Detailed experiments with multiple state-of-the-art 3D reconstruction and object pose estimation approaches show that the geometry produced by modern reconstruction methods is often sufficient for accurate pose estimation. Our experiments lead to interesting observations: (1) Standard metrics for measuring 3D reconstruction quality are not necessarily indicative of pose estimation accuracy which shows the need for dedicated benchmarks such as ours. (2) Classical non-learning-based approaches can perform on par with modern learning-based reconstruction techniques and can even offer a better reconstruction time-pose accuracy tradeoff. (3) There is still a sizable gap between performance with reconstructed and with CAD models. To foster research on closing this gap the benchmark is made available at https://github.com/VarunBurde/reconstruction_pose_benchmark.

Unsupervised domain adaptation (UDA) tries to overcome the tedious work of labeling data by leveraging a labeled source dataset and transferring its knowledge to a similar but different target dataset. Meanwhile current vision-language models exhibit remarkable zero-shot prediction capabilities. In this work we combine knowledge gained through UDA with the inherent knowledge of vision-language models. We introduce a strong-weak guidance learning scheme that employs zero-shot predictions to help align the source and target dataset. For the strong guidance we expand the source dataset with the most confident samples of the target dataset. Additionally we employ a knowledge distillation loss as weak guidance. The strong guidance uses hard labels but is only applied to the most confident predictions from the target dataset. Conversely the weak guidance is employed to the whole dataset but uses soft labels. The weak guidance is implemented as a knowledge distillation loss with (adjusted) zero-shot predictions. We show that our method complements and benefits from prompt adaptation techniques for vision-language models. We conduct experiments and ablation studies on three benchmarks (OfficeHome VisDA and DomainNet) outperforming state-of-the-art methods. Our ablation studies further demonstrate the contributions of different components of our algorithm.

Daily monitoring of intra-personal facial changes associated with health and emotional conditions has great potential to be useful for medical healthcare and emotion recognition fields. However the approach for capturing intra-personal facial changes is relatively unexplored due to the difficulty of collecting temporally changing face images. In this paper we propose a facial representation learning method using synthetic images for comparing faces called ComFace which is designed to capture intra-personal facial changes. For effective representation learning ComFace aims to acquire two feature representations i.e. inter-personal facial differences and intra-personal facial changes. The key point of our method is the use of synthetic face images to overcome the limitations of collecting real intra-personal face images. Facial representations learned by ComFace are transferred to three extensive downstream tasks for comparing faces: estimating facial expression changes weight changes and age changes from two face images of the same individual. Our ComFace trained using only synthetic data achieves comparable to or better transfer performance than general pre-training and state-of-the-art representation learning methods trained using real images.

Diffusion models have achieved great success in dual-conditioned image generation. However they still face significant challenges in image-guided sketch colorization where reference and sketch images usually exhibit different spatial structures and semantics. This mismatch termed "distribution shift" in this paper results in various artifacts and degrades the colorization quality. To address this issue we conducted thorough investigations into the image-prompted latent diffusion model and developed a two-stage training framework to mitigate the effects of distribution shift based on our analysis. Comprehensive quantitative comparisons qualitative evaluations and user studies were performed to demonstrate the superiority of our proposed methods. Additionally an ablation study was conducted to assess the impact of the distribution shift and the selection of reference embeddings. Codes are made publicly available at https://github.com/tellurionkanata/colorizeDiffusion.

Finger photo Presentation Attack Detection (PAD) can significantly strengthen smartphone device security. Still these algorithms are trained to detect certain attack types. Furthermore they are designed to operate on images acquired by specific capture devices which leads to poor generalization and a lack of robustness in handling mobile hardware's evolving nature. The proposed investigation is the first to systematically analyze the performance degradation of existing deep learning PAD systems convolutional and transformers in cross-capture device settings. In this paper we introduce the ColFigPhotoAttnNet architecture designed based on window attention on color channels followed by the nested residual network as the predictor to achieve a reliable PAD. Extensive experiments using various capture devices including iPhone13 Pro GooglePixel 3 Nokia C5 and OnePlusOne were carried out to evaluate the performance of proposed and existing methods on three publicly available databases. The findings underscore the effectiveness of our approach.