WACV 2025 Papers

Advancements in vision-language models (VLMs) have propelled the field of computer vision particularly in the zero-shot learning setting. Despite their promise the effectiveness of these models often diminishes due to domain shifts in test environments. To address this we introduce the Test-Time Prototype Shifting (TPS) framework a pioneering approach designed to adapt VLMs to test datasets using unlabeled test inputs. Our method is based on the notion of modulating per-class prototypes in the shared embedding space. By pre-computing and caching prototypes generated with the pre-trained text encoder TPS not only facilitates optimization-free prototype reuse for subsequent predictions but also enables seamless integration with current advancements in prompt engineering. At test-time TPS dynamically learns shift vectors for each prototype based solely on the given test sample effectively bridging the domain gap and enhancing classification accuracy. A notable aspect of our framework is its significantly reduced memory and computational demands when compared to conventional text-prompt tuning methods. Extensive evaluations across 15 image classification datasets involving natural distribution shifts and cross-dataset generalization as well as in context-dependent visual reasoning demonstrate TPS's superior performance achieving state-of-the-art results while reducing resource requirements. Code is available at https://github.com/elaine-sui/TPS.

Recent advances in co-speech gesture and talking head generation have been impressive yet most methods focus on only one of the two tasks. Those that attempt to generate both often rely on separate models or network modules increasing training complexity and ignoring the inherent relationship between face and body movements. To address the challenges in this paper we propose a novel model architecture that jointly generates face and body motions within a single network. This approach leverages shared weights between modalities facilitated by adapters that enable adaptation to a common latent space. Our experiments demonstrate that the proposed framework not only maintains state-of-the-art co-speech gesture and talking head generation performance but also significantly reduces the number of parameters required.

Cryo-Electron Tomography (Cryo-ET) enables detailed 3D visualization of cellular structures in near-native states but suffers from low signal-to-noise ratio due to imaging constraints. Traditional denoising methods and supervised learning approaches often struggle with complex noise patterns and the lack of paired datasets. Self-supervised methods which utilize noisy input itself as a target have been studied; however existing Cryo-ET self-supervised denoising methods face significant challenges due to losing information during training and the learned incomplete noise patterns. In this paper we propose a novel self-supervised learning model that denoises Cryo-ET volumetric images using a single noisy volume. Our method features a U-shape J-invariant blind spot network with sparse centrally masked convolutions dilated channel attention blocks and volume-unshuffle/shuffle technique. The volume-unshuffle/shuffle technique expands receptive fields and utilizes multi-scale representations significantly improving noise reduction and structural preservation. Experimental results demonstrate that our approach achieves superior performance compared to existing methods advancing Cryo-ET data processing for structural biology research.

The proliferation of AI-generated images has intensified the need for robust content authentication methods. We present InvisMark a novel watermarking technique designed for high-resolution AI-generated images. Our approach leverages advanced neural network architectures and training strategies to embed imperceptible yet highly robust watermarks. InvisMark achieves state-of-the-art performance in imperceptibility (PSNR 51 SSIM 0.998) while maintaining over 97% bit accuracy across various image manipulations. Notably we demonstrate the successful encoding of 256-bit watermarks significantly expanding payload capacity while preserving image quality. This enables the embedding of UUIDs with error correction codes achieving near-perfect decoding success rates even under challenging image distortions. We also address potential vulnerabilities against advanced attacks and propose mitigation strategies. By combining high imperceptibility extended payload capacity and resilience to manipulations InvisMark provides a robust foundation for ensuring media provenance in an era of increasingly sophisticated AI-generated content.

The analysis of individual cells is increasingly automated through deep learning techniques. This is particularly relevant for high-resolution whole slide images (WSIs) which can contain thousands of cells making manual evaluation impractical. This increase in automation however requires higher levels of standardisation (with respect to the scanning hardware settings and staining) and is further aggravated by the dynamics of the underlying cellular processes rendering unique cell classifications difficult. To address these difficulties we investigated the entire processing pipeline (from imaging over annotation to model training) and study its underlying trade-offs. In particular we created a new dataset comprising of more than 6300 labelled and 500000 unlabelled cells scanned using two different scan settings resulting in fully registered image pairs with varying level of detail and quality. Using these alternative dataset versions we analysed the impact of inter- and intra-variability between three different annotators and addressed the challenge of limited labelled data by comparing the impact of different self-supervised pretraining strategies. Overall our analyses provide new insights into the dependencies between imaging annotation self-supervision and deep learning-based classification especially in the context of continuously developing cells and demonstrate the beneficial impact of these considerations on the overall classification accuracy. Code is available at https://zivgitlab.uni-muenster.de/cvmls/icdc and the data will be shared upon qualified request due to data privacy laws.

This paper studies the problem of inverting the DDIM image generation process to recover latent variables particularly the initial noise map from a generated image. Existing methods often struggle with accuracy in this task. We propose a novel hybrid approach that combines direct inversion via gradient descent for the first step followed by a fixed-point method for subsequent steps. Empirical evaluations across three datasets demonstrate that our method significantly improves the prediction of initial latent variables while achieving superior reconstruction accuracy. Additionally we introduce a new evaluation called the self-interpolation test which assesses the quality of images generated from interpolated points between the true and predicted latent maps offering deeper insights into performance. Our results reveal that while existing methods perform reasonably well in reconstruction they consistently fail to accurately predict the initial latent variables resulting in poor performance on the self-interpolation test. In contrast our method outperforms all others across all metrics providing valuable insights into diffusion models and enhancing their applications in image generation and editing.

Inverse problems have many applications in science and engineering. In Computer vision several image restoration tasks such as inpainting deblurring and super-resolution can be formally modeled as inverse problems. Recently methods have been developed for solving inverse problems that only leverage a pre-trained unconditional diffusion model and do not require additional task-specific training. In such methods however the inherent intractability of determining the conditional score function during the reverse diffusion process poses a real challenge leaving the methods to settle with an approximation instead which affects their performance in practice. Here we propose a MAP estimation framework to model the reverse conditional generation process of a continuous time diffusion model as an optimization process of the underlying MAP objective whose gradient term is tractable. In theory the proposed framework can be applied to solve general inverse problems using gradient-based optimization methods. However given the highly non-convex nature of the loss objective finding a perfect gradient-based optimization algorithm can be quite challenging nevertheless our framework offers several potential research directions. We use our proposed formulation to develop empirically effective algorithms for image restoration. We validate our proposed algorithms with extensive experiments over multiple datasets across several restoration tasks.

Geometric shape features have been widely used as strong predictors for image classification. Nevertheless most existing classifiers such as deep neural networks (DNNs) directly leverage the statistical correlations between these shape features and target variables. However these correlations can often be spurious and unstable across different environments (e.g. in different age groups certain types of brain changes have unstable relations with neurodegenerative disease); hence leading to biased or inaccurate predictions. In this paper we introduce a novel framework that for the first time develops invariant shape representation learning (ISRL) to further strengthen the robustness of image classifiers. In contrast to existing approaches that mainly derive features in the image space our model ISRL is designed to jointly capture invariant features in latent shape spaces parameterized by deformable transformations. To achieve this goal we develop a new learning paradigm based on invariant risk minimization (IRM) to learn invariant representations of image and shape features across multiple training distributions/environments. By embedding the features that are invariant with regard to target variables in different environments our model consistently offers more accurate predictions. We validate our method by performing classification tasks on both simulated 2D images real 3D brain and cine cardiovascular magnetic resonance images (MRIs). Our code is publicly available at https://github.com/tonmoy-hossain/ISRL.

This paper highlights the potential of a Human-In-the-Loop (HIL) in interactive object detection methods. Although automation in computer vision is advancing rapidly certain critical tasks such as detecting UneXploded Ordnance (UXO) space/marine debris or the generation of new datasets require 100% recall and near-perfect precision. These tasks are often performed manually since automatic methods do not achieve the necessary accuracy. However interactive object detection frameworks can potentially enhance annotation speed while maintaining the recall and accuracy of manual annotation. We propose IRTDETR an interactive and real-time object detection method for very large imagery to address this. Using either point or bounding box annotations provided by a HIL it globally relates the full image with the annotator inputs via a cross-attention-like mechanism employs an attention loss to maximize the classification score based on similarity and reuses portions of the network outputs during iterative refinements to conserve resources. We conduct experiments on five different datasets (Tiny-DOTA CHAI AITOD SarDET and COCO) to verify the efficacy of our approach. Our method surpasses existing interactive annotation approaches achieving a higher mean Average Precision (mAP) with the same number of clicks. Additionally we validate the annotation efficiency of our method in a user study demonstrating it is 2.46x quicker and asks for only 72% of the task load (NASA-TLX) compared to fully manual annotation. The code will is available under https://github.com/mburges-cvl/WACV_IAODF.

Transformer based methods have enabled users to create modify and comprehend text and image data. Recently proposed Large Reconstruction Models (LRMs) further extend this by providing the ability to generate high-quality 3D models with the help of a single object image. These models however lack the ability to manipulate or edit the finer details such as adding standard design patterns or changing the color and reflectance of the generated objects thus lacking fine-grained control that may be very helpful in domains such as augmented reality animation and gaming. Naively training LRMs for this purpose would require generating precisely edited images and 3D object pairs which is computationally expensive. In this paper we propose Instructive3D a novel LRM based model that integrates generation and fine-grained editing through user text prompts of 3D objects into a single model. We accomplish this by adding an adapter that performs a diffusion process conditioned on a text prompt specifying edits in the triplane latent space representation of 3D object models. Our method does not require the generation of edited 3D objects. Additionally Instructive3D allows us to perform geometrically consistent modifications as the edits done through user-defined text prompts are applied to the triplane latent representation thus enhancing the versatility and precision of 3D objects generated. We compare the objects generated by Instructive3D and a baseline that first generates the 3D object meshes using a standard LRM model and then edits these 3D objects using text prompts when images are provided from the Objaverse LVIS dataset. We find that Instructive3D produces qualitatively superior 3D objects with the properties specified by the edit prompts.

Driving is challenging in conditions like night rain and snow. Lack of good labeled datasets has hampered progress in scene understanding under such conditions. Unsupervised Domain Adaptation (UDA) using large labeled clear-day datasets is a promising research direction in such cases. However many UDA methods are trained with dominant scene backgrounds (e.g. roads sky sidewalks) that appear dramatically different across domains. As a result they struggle to learn effective features of smaller and often sparse foreground objects (e.g. people vehicles signs). In this work we improve UDA training by applying in-place image warping to focus on salient objects. We design instance-level saliency guidance to adaptively oversample object regions and undersample background areas which reduces adverse effects from background context and enhances backbone feature learning. Our approach improves adaptation across geographies lighting and weather conditions and is agnostic to the task (segmentation detection) domain adaptation algorithm saliency guidance and underlying model architecture. Result highlights include +6.1 mAP50 for BDD100K Clear to DENSE Foggy +3.7 mAP50 for BDD100K Day to Night +3.0 mAP50 for BDD100K Clear to Rainy and +6.3 mIoU for Cityscapes to ACDC. Besides Our method adds minimal training memory and no additional inference latency. Code is available at https://github.com/ShenZheng2000/Instance-Warp.

Variational channel pruning approaches have obtained impressive results thanks to their stochastic nature well established foundation in information theory and the practically appealing structured sparsity pattern they offer. Despite their success in pruning Plain Networks (PlainNets) their application has faced certain limitations in networks with structurally coupled channels such as ResNets. In such scenarios not only is it required to prune structurally coupled channels together but it is also necessary to ensure that the whole coupled group is irrelevant before pruning is applied. This is an under-investigated problem as most existing methods are designed without taking these couplings into account. In this paper we propose a novel approach based on Information Theoretic Pruning of structurally Coupled Channels (ITPCC) in neural networks. ITPCC allows for learning the probabilistic distribution of coupled channel set importance and prunes the ones withthe least relevant information to the task at hand. Experimental results for image classification on CIFAR10 CIFAR100 and ImageNet datasets show that the proposed method outperforms the state-of-the-art more significantly at high compression rates.

Invoices and receipts submitted by employees are visually rich documents (VRDs) with textual visual and layout information. To protect against the risk of fraud and abuse it is crucial for organizations to efficiently extract desired information from submitted receipts. This helps in the assessment of key factors such as appropriateness of the expense claim adherence to spending and transaction policies the validity of the receipt as well as downstream anomaly detection at various levels. These documents are heterogeneous with multiple formats and languages uploaded with different image qualities and often do not contain ground truth labels for the efficient training of models. In this paper we propose Task Aware Instruction-based Labelling (TAIL) a method for synthetic label generation in VRD corpuses without labels and fine-tune a multimodal Visually Rich Document Understanding Model (VRDU) on TAIL labels using response-based knowledge distillation without using the teacher model's weights or training dataset to conditionally generate annotations in the appropriate format. Using a benchmark external dataset where ground truth labels are available we demonstrate conditions under which our approach performs at par with Claude 3 Sonnet through empirical studies. We then show that the resulting model performs at par or better on the internal expense documents of a large multinational organization than state-of-the-art LMM (large multimodal model) Claude 3 Sonnet while being 85% less costly and 5X faster and outperforms layout-aware baselines by more than 10% in Average Normalized Levenshtein Similarity (ANLS) scores due to its ability to reason and extract information from rare formats. Finally we illustrate the usage of our approach in overpayment prevention.

Abductive reasoning aims to make the most likely inference for a given set of incomplete observations. In this paper we introduce "Abductive Past Action Inference" a novel research task aimed at identifying the past actions performed by individuals within homes to reach specific states captured in a single image using abductive inference. The research explores three key abductive inference problems: past action set prediction past action sequence prediction and abductive past action verification. We introduce several models tailored for abductive past action inference including a relational graph neural network a relational bilinear pooling model and a relational transformer model. Notably the newly proposed object-relational bilinear graph encoder-decoder (BiGED) model emerges as the most effective among all methods evaluated demonstrating good proficiency in handling the intricacies of the Action Genome dataset. The contributions of this research significantly advance the ability of deep learning models to reason about current scene evidence and make highly plausible inferences about past human actions. This advancement enables a deeper understanding of events and behaviors which can enhance decision-making and improve system capabilities across various real-world applications such as Human-Robot Interaction and Elderly Care and Health Monitoring. Code and data available at https://github.com/LUNAProject22/AAR

Despite advancements in human motion generation models their performance drops in infant motion generation due to limited data available and lack of 3D skeleton ground truth. To address this we introduce the infant action generation and classification (InfAGenC) pipeline which combines a transformer-based variational autoencoder (VAE) with a spatial-temporal graph convolutional network (STGCN) to create synthetic infant action samples. By iterative refinement of the generative model with diverse and accurate data we improve the realism of synthetic data leading to more precise infant action recognition models. Our results show significant improvements in action recognition performance on real-world data demonstrating that synthetic data can enhance small training datasets and advance infant action recognition. Our pipeline increases action recognition accuracy up to 88.58% on the infant action dataset and up to 98% on an adult action dataset.

Understanding the progress of a task allows humans to not only track what has been done but also to better plan for future goals. We demonstrate TaKSIE a novel framework that incorporates task progress knowledge into visual subgoal generation for robotic manipulation tasks. We jointly train a recurrent network with a latent diffusion model to generate the next visual subgoal based on the robot's current observation and the input language command. At execution time the robot leverages a visual progress representation to monitor the task progress and adaptively samples the next visual subgoal from the model to guide the manipulation policy. We train and validate our model in simulated and real-world robotic tasks achieving state-of-the-art performance on the CALVIN manipulation benchmark. We find that the inclusion of task progress knowledge can improve the robustness of trained policy for different initial robot poses or various movement speeds during demonstrations. The project page is available at https://live-robotics-uva.github.io/TaKSIE/.

In this paper we introduce InDistill a method that serves as a warmup stage for enhancing Knowledge Distillation (KD) effectiveness. InDistill focuses on transferring critical information flow paths from a heavyweight teacher to a lightweight student. This is achieved via a training scheme based on curriculum learning that considers the distillation difficulty of each layer and the critical learning periods when the information flow paths are established. This procedure can lead to a student model that is better prepared to learn from the teacher. To ensure the applicability of InDistill across a wide range of teacher-student pairs we also incorporate a pruning operation when there is a discrepancy in the width of the teacher and student layers. This pruning operation reduces the width of the teacher's intermediate layers to match those of the student allowing direct distillation without the need for an encoding stage. The proposed method is extensively evaluated using various pairs of teacher-student architectures on CIFAR-10 CIFAR-100 and ImageNet datasets demonstrating that preserving the information flow paths consistently increases the performance of the baseline KD approaches on both classification and retrieval settings. The code is available at https://github.com/gsarridis/InDistill.

Detecting object-level changes between two images across possibly different views (Fig. 1) is a core task in many applications that involve visual inspection or camera surveillance. Existing change-detection approaches suffer from three major limitations: (1) lack of evaluation on image pairs that contain no changes leading to unreported false positive rates; (2) lack of correspondences (i.e. localizing the regions before and after a change); and (3) poor zero-shot generalization across different domains. To address these issues we introduce a novel method that leverages change correspondences (a) during training to improve change detection accuracy and (b) at test time to minimize false positives. That is we harness the supervision labels of where an object is added or removed to supervise change detectors improving their accuracy over previous work [21] by a large margin. Our work is also the first to predict correspondences between pairs of detected changes using estimated homography and the Hungarian algorithm. Our model demonstrates superior performance over existing methods achieving state-of-the-art results in change detection and change correspondence accuracy across both in-distribution and zero-shot benchmarks.

AI-driven second-opinion systems play a crucial role in decision-making especially in medicine where accurate predictions guide clinicians. However quantifying uncertainty in deep learning is challenging as current methods often rely on hard class labels which do not reflect true prediction confidence. This often results in overconfident predictions and slow convergence to true probabilities. To address this we suggest a new method that separates uncertainty into two types: epistemic and aleatoric. We estimate these uncertainties using hard and soft confidence labels with experts providing confidence levels that indicate the likelihood of misclassification. We release an updated blood typing dataset consisting of 3139 images with soft labels of uncertainty annotations from six experts and hard labels collected from medical records. Proposed approach improves SotA uncertainty estimation quality by two times for blood typing (classification) and by 62% for histology (segmentation). The code is available at: https://github.com/createcolor/confidence-aware-uncertainty.