WACV 2025 论文列表

RegistFormer our novel reference-augmented image synthesis framework generates aligned pseudo-CT images (with respect to MR) from misaligned MR and CT pairs. RegistFormer addresses the limitations of intensity-based registration methods which often fail due to dissimilar image features and complex deformation fields. Unlike conventional image-to-image (I2I) translation methods our method uses a misaligned CT scan as an auxiliary input to guide the synthesis task through the Deformation-Aware Cross-Attention (DACA) mechanism. DACA integrates the deformation field from a registration method to aggregate spatially matched features from the misaligned CT into MR spatial coordinates. Additionally we propose a novel combination of loss functions for training with datasets of misaligned MR-CT pairs in a self-supervised manner eliminating the need for pre-aligned training data. Experiments were conducted with the synthRAD2023 MR-CT pelvis pair dataset. RegistFormer outperforms past state-of-the-art methods including I2I registration and hybrid (registration + I2I) across metrics evaluating both structure alignment and distribution similarity. Moreover RegistFormer demonstrates superior performance in zero-shot segmentation downstream tasks highlighting its clinical value. Source code: https://github.com/danny4159/RegistFormer

Text-to-Image diffusion models have shown remarkable capabilities in generating high-quality images. However current models often struggle to adhere to the complete set of conditions specified in the input text and return unfaithful generations. Existing works address this problem by either fine-tuning the base model or modifying the latent representations during the inference stage with gradient-based updates. Not only are these approaches computationally expensive but also they usually only improve limited kinds of errors (e.g. the count of objects). In this work we propose an intervention-based mechanism to enhance the faithfulness of diffusion models by controlling the denoising process. Starting with layout-conditional diffusion models our approach first detects incorrectly-generated/missing objects during denoising steps. Next a layout is constructed from the erroneous objects (feedback). Finally we return to an earlier denoising step. The new layout is fed to the diffusion model to obtain its latent representation. Correction is applied by composing the new latents with the original ones and continuing the generation process thereby driving the generation away from erroneous directions. As additional feedback and correction strategy we also explore retrieval-augmented generation to help the model recover missing objects. We conduct experiments on VPEval and HRS-Bench datasets and measure faithfulness across four dimensions; presence of objects object counts scale of objects and spatial relations between objects. Compared to GLIGEN the state-of-the-art model on the VPEval dataset our approach significantly improves on all metrics (+6.7% average accuracy increase). On HRS-Bench dataset it also outperforms existing models in count and scale metrics.

Pluralistic Image Inpainting (PII) offers multiple plausible solutions for restoring missing parts of images and has been successfully applied to various applications including image editing and object removal. Recently VQGAN-based methods have been proposed and have shown that they significantly improve the structural integrity in the generated images. Nevertheless the state-of-the-art VQGAN-based model PUT faces a critical challenge: degradation of detail quality in output images due to feature quantization. Feature quantization restricts the latent space and causes information loss which negatively affects the detail quality essential for image inpainting. To tackle the problem we propose the FDM (Feature Dequantization Module) specifically designed to restore the detail quality of images by compensating for the information loss. Furthermore we develop an efficient training method for FDM which drastically reduces training costs. We empirically demonstrate that our method significantly enhances the detail quality of the generated images with negligible training and inference overheads. The code is available at https://github.com/hyudsl/FDM

Deep visual detectors are known to be vulnerable to adversarial attacks raising concerns about their real-world applications (e.g. self-driving perception). We argue that this vulnerability arises from the spurious dependency of final detections on irrelevant/loophole latent dimensions. The greater the number of such dimensions the higher the likelihood of the detector being compromised by adversarial attacks making it more susceptible to input perturbations. To enhance detection robustness we propose Detection-related Discriminant Maximization and Reorganization (DDMR) condensing the detection utility to a compressed number of relevant dimensions while deactivating the influence of irrelevant ones. This approach also alleviates the misalignment issue between the two task domains in visual detection and consequently their gradients. This enables the generation of more potent adversarial attacks and defenses for visual detectors within the adversarial training framework. Extensive experiments conducted with four cutting-edge visual detectors on the KITTI and COCO datasets showcase the efficacy of the proposed approach in improving the adversarial robustness of deep visual detectors against both white-box and black-box attacks. For example on the KITTI dataset our method demonstrates an increase in robustness of up to 12.4% and 28.0% without and with adversarial training respectively.

Conditional diffusion probabilistic models can model the distribution of natural images and can generate diverse and realistic samples based on given conditions. However oftentimes their results can be unrealistic with observable color shifts and textures. We believe that this issue results from the divergence between the probabilistic distribution learned by the model and the distribution of natural images. The delicate conditions gradually enlarge the divergence during each sampling timestep. To address this issue we introduce a new method that brings the predicted samples to the training data manifold using a pretrained unconditional diffusion model. The unconditional model acts as a regularizer and reduces the divergence introduced by the conditional model at each sampling step. We perform comprehensive experiments to demonstrate the effectiveness of our approach on super-resolution colorization turbulence removal and image-deraining tasks. The improvements obtained by our method suggest that the priors can be incorporated as a general plugin for improving conditional diffusion models.

Efficient visual trackers overfit to their training distributions and lack generalization abilities resulting in them performing well on their respective in-distribution (ID) test sets and not as well on out-of-distribution (OOD) sequences imposing limitations to their deployment in-the-wild under constrained resources. We introduce SiamABC a highly efficient Siamese tracker that significantly improves tracking performance even on OOD sequences. SiamABC takes advantage of new architectural designs in the way it bridges the dynamic variability of the target and of new losses for training. Also it directly addresses OOD tracking generalization by including a fast backward-free dynamic test-time adaptation method that continuously adapts the model according to the dynamic visual changes of the target. Our extensive experiments suggest that SiamABC shows remarkable performance gains in OOD sets while maintaining accurate performance on the ID benchmarks. SiamABC outperforms MixFormerV2-S by 7.6% on the OOD AVisT benchmark while being 3x faster (100 FPS) on a CPU. Our code and models are available at https://wvuvl.github.io/SiamABC/.

Recently transformers have gained prominence in video action recognition due to their ability to capture spatio-temporal dependencies. Despite their effectiveness the interpretability of their self-attention mechanisms remains limited posing obstacles in understanding model decisions impacting transparency and bias identification. Additionally the computational demands of transformer architectures particularly the self-attention mechanism present practical difficulties. To tackle both challenges we adapt existing interpretability techniques and introduce a layer pruning method guided by importance metrics. In the context of driver action recognition our findings highlight the efficacy of the applied head importance metrics in pinpointing crucial attention heads and identifying key visual cues essential for recognizing driver behavior. Experimental results conducted on three mainstream video transformers demonstrate the effectiveness of the proposed pruning technique with significantly reduced computational costs and only slight performance degradation by removing low-relevance layers. Specifically on our DriverActionInsight (DAI) dataset we achieve a 23.5% FLOPs saving in compressing Video Swin with less than a 1% decrease in Top-1 accuracy.

Accurate retinal vessel (RV) segmentation is a crucial step in the quantitative assessment of retinal vasculature which is needed for the early detection of retinal diseases and other conditions. Numerous studies have been conducted to tackle the problem of segmenting vessels automatically using a pixel-wise classification approach. The common practice of creating ground truth labels is to categorize pixels as foreground and background. This approach is however biased and it ignores the uncertainty of a human annotator when it comes to annotating e.g. thin vessels. In this work we propose a simple and effective method that casts the RV segmentation task as an image-level regression. For this purpose we first introduce a novel Segmentation Annotation Uncertainty-Aware (SAUNA) transform which adds pixel uncertainty to the ground truth using the pixel's closeness to the annotation boundary and vessel thickness. To train our model with soft labels we generalize the earlier proposed Jaccard metric loss to arbitrary hypercubes for soft Jaccard index (Intersection-over-Union) optimization. Additionally we employ a stable version of the Focal-L1 loss for pixel-wise regression. We conduct thorough experiments and compare our method to a diverse set of baselines across 5 retinal image datasets. Our empirical results indicate that the integration of the SAUNA transform and these segmentation losses led to significant performance boosts for different segmentation models. Particularly our methodology enables UNet-like architectures to substantially outperform computational-intensive baselines. Our implementation is available at https://github.com/Oulu-IMEDS/SAUNA.

Recent advances in vision-language models have combined contrastive approaches with generative methods to achieve state-of-the-art (SOTA) on downstream inference tasks like zero-shot image classification. However a persistent issue of these models for image classification is their out-of-distribution (OOD) generalization capabilities. We first show that when an OOD datapoint is misclassified the correct class can be typically found in the Top-K predicted classes. In order to steer the model prediction toward the correct class within the top predicted classes we propose the Image-Caption Encoding (ICE) method a straightforward approach that directly enforces consistency between the image-conditioned and caption-conditioned predictions at evaluation time only. Intuitively we take advantage of unique properties of the generated captions to guide our local search for the correct class label within the Top-K predicted classes. We show that our method can be easily combined with other SOTA methods to enhance Top-1 OOD accuracies by 0.5% on average and up to 3% on challenging datasets.

Colour Vision Deficiency (CVD) occurs when anomalous retinal cone spectral responses impact the ability to distinguish between certain colours. To enhance image quality and viewing experience recolouring algorithms seek to modify pixel values so that this does not lead to a loss of detail or image quality. Recent approaches to recolouring for CVD viewers employ neural models which exploit higher order features to direct colour adaptation. In this work we build upon the idea that visual neural models exhibit emergent behaviour which mimics the human visual system. We make use of these learned behaviours to guide the colour adaptation process by considering regions of the image that are the most semantically meaningful for a non-CVD viewer and compensate for them appropriately if they are absent or distorted in a CVD-simulated version of the image. We find that a minimal algorithm built atop a pre-trained model produces results that substantially boost contrast and salience for viewers affected by CVD. We also investigate a few cases where modifications are absent indicating that a neurally guided salience-based model may also provide a means of determining when recolouring is not necessary. Additionally we introduce a novel metric that quantifies the contrast increase or decrease under changes in image colour.

Uncertainty estimation is crucial in safety-critical applications where robust out-of-distribution (OOD) detection is essential. Traditional Bayesian methods though effective are often hindered by high computational demands. As an alternative Laplace approximation offers a more practical and efficient approach to uncertainty estimation. In this paper we introduce the Identity Curvature Laplace Approximation (ICLA) a novel method that challenges the conventional posterior covariance formulation by using identity curvature and optimizing prior precision. This innovative design significantly enhances OOD detection performance on well-known datasets such as CIFAR-10 CIFAR-100 and ImageNet while maintaining calibration scores. We attribute this improvement to the alignment issues between typical feature embeddings and curvature as measured by the Fisher information matrix. Our findings are further supported by demonstrating that incorporating Fisher penalty or sharpness-aware minimization techniques can greatly enhance the uncertainty estimation capabilities of standard Laplace approximation.

Backdoor attacks present a significant threat to the robustness of Federated Learning (FL) due to their stealth and effectiveness. They maintain both the main task of the FL system and the backdoor task simultaneously causing malicious models to appear statistically similar to benign ones which enables them to evade detection by existing defense methods. We find that malicious parameters in backdoored models are inactive on the main task resulting in a significantly large empirical loss during the machine unlearning process on clean inputs. Inspired by this we propose MASA a method that utilizes individual unlearning on local models to identify malicious models in FL. To improve the performance of MASA in challenging non-independent and identically distributed (non-IID) settings we design pre-unlearning model fusion that integrates local models with knowledge learned from other datasets to mitigate the divergence in their unlearning behaviors caused by the non-IID data distributions of clients. Additionally we propose a new anomaly detection metric with minimal hyperparameters to filter out malicious models efficiently. Extensive experiments on IID and non-IID datasets across six different attacks validate the effectiveness of MASA. To the best of our knowledge this is the first work to leverage machine unlearning to identify malicious models in FL. Code is available at https://github.com/JiiahaoXU/MASA.

Point cloud visibility estimation is fundamental as it is useful for many computer vision applications including surface reconstruction 3D segmentation from paired images and point densification. Previous works showed outstanding results on simple object and outdoor datasets. However unlike the previously studied scenes the most challenging environments are those providing a high amount of object points in the same direction typically in complex indoor scenes. In this kind of environments due to the lack of real data ground truth quantitative analysis are either missing or based on simulated data. In this work we present IRIS-VIS (Industrial Room In Saclay - VISibility) a new dataset for point visibility estimation in an indoor environment. It is a high complexity scene due to the large variety in the shape size and orientation of the objects. To our knowledge this is the first dataset on real indoor data providing a dense LiDAR station-based point cloud along with a well-fitted CAD model. The latter is useful to compute automatically quickly and accurately the visibility from any given viewpoint enabling evaluations under infinite conditions. We propose new metrics for the visibility estimation task and evaluate state-of-the-art methods in both sparse and dense conditions with the proposed dataset.

In this study we introduce UJ-AQA-CricketVision a dataset comprising 8540 video clips of cricket strokes each annotated with detailed phase breakdowns. We develop a novel multi-variate approach for Action Quality Assessment (AQA) at a body level that leverages an Autoencoder for extracting sophisticated feature representations from video frames and pose estimated keypoints. These features are subsequently utilised by a multi-layer perceptron regression-based model to accurately predict the quality of cricket actions in terms of their head shoulder hands hips and feet. Our approach is benchmarked against contemporary state-of-the-art AQA methods and achieves a Spearman Rank Correlation score of 0.84. The performance highlights the significance of integrating pose keypoint and frame data for the nuanced analysis of short and complex action sequences in sports such as cricket. This work aims to foster the development of accurate Action Quality Assessment methods on Cricket Video data. The dataset can be found here: https://github.com/dvanderhaar/uj-aqa-cricketvision

Visual framing analysis is a key method in social sciences for determining common themes and concepts in a given discourse. To reduce manual effort image clustering can significantly speed up the annotation process. In this work we phrase the clustering task as a Minimum Cost Multicut Problem [MP]. Solutions to the MP have been shown to provide clusterings that maximize the posterior probability solely from provided local pairwise probabilities of two images belonging to the same cluster. We discuss the efficacy of numerous embedding spaces to detect visual frames and show its superiority over other clustering methods. To this end we employ the climate change dataset ClimateTV which contains images commonly used for visual frame analysis. For broad visual frames DINOv2 is a suitable embedding space while ConvNeXt V2 returns a larger number of clusters which contain fine-grain differences i.e. speech and protest. Our insights into embedding space differences in combination with the optimal clustering - by definition - advances automated visual frame detection. Our code can be found at https://github.com/KathPra/MP4VisualFrameDetection.

Inpainting focuses on filling missing or corrupted regions of an image to blend seamlessly with its surrounding content and style. While conditional diffusion models have proven effective for text-guided inpainting we introduce the novel task of multi-mask inpainting where multiple regions are simultaneously inpainted using distinct prompts. Furthermore we design a fine-tuning procedure for multimodal LLMs such as LLaVA to generate multi-mask prompts automatically using corrupted images as inputs. These models can generate helpful and detailed prompt suggestions for filling the masked regions. The generated prompts are then fed to Stable Diffusion which is fine-tuned for the multi-mask inpainting problem using rectified cross-attention enforcing prompts onto their designated regions for filling. Experiments on digitized paintings from WikiArt and the Densely Captioned Images dataset demonstrate that our pipeline delivers creative and accurate inpainting results. Our code data and trained models are available at https://cilabuniba.github.io/i-dream-my-painting.

Many computer vision applications rely on interpretable machine learning algorithms to analyze the data collected from various sources. We leverage Hyperdimensional Computing (HDC) as an innovative computational model that mimics key brain functionalities to achieve efficient and robust cognitive learning. We propose HDlm a novel HDC-based cognitive representation capable of adaptive information association and memorization. HDlm first theoretically expands HDC mathematics to support selective information association and adaptive memorization. Then it exploits the proposed operations to support cognitive operations including set membership information retrieval and item comparison. We evaluated our solution for a selection of applications related to visual data representation and sequence matching analysis. Our evaluation shows that HDlm provides more adaptive similarity metrics between objects that lead to better task performance.

We propose HYBRIDDEPTH a robust depth estimation pipeline that addresses key challenges in depth estimation including scale ambiguity hardware heterogeneity and generalizability. HYBRIDDEPTH leverages focal stack data conveniently accessible in common mobile devices to produce accurate metric depth maps. By incorporating depth priors afforded by recent advances in single-image depth estimation our model achieves a higher level of structural detail compared to existing methods. We test our pipeline as an end-to-end system with a newly developed mobile client to capture focal stacks which are then sent to a GPU-powered server for depth estimation. Comprehensive quantitative and qualitative analyses demonstrate that HYBRIDDEPTH outperforms state-of-the-art (SOTA) models on common datasets such as DDFF12 and NYU Depth V2. HYBRIDDEPTH also shows strong zero-shot generalization. When trained on NYU Depth V2 HYBRIDDEPTH surpasses SOTA models in zero-shot performance on ARKitScenes and delivers more structurally accurate depth maps on Mobile Depth. The code is available at https://github.com/cake-lab/HybridDepth/.

Assistive embodied agents that can be instructed in natural language to perform tasks in open-world environments have the potential to significantly impact labor tasks like manufacturing or in-home care - benefiting the lives of those who come to depend on them. In this work we consider how this benefit might be hijacked by local modifications in the appearance of the agent's operating environment. Specifically we take the popular Vision-andLanguage Navigation (VLN) task as a representative setting and develop a whitebox adversarial attack that optimizes a 3D attack object's appearance to induce desired behaviors in pretrained VLN agents that observe it in the environment. We demonstrate that the proposed attack can cause VLN agents to ignore their instructions and execute alternative actions after encountering the attack object - even for instructions and agent paths not considered when optimizing the attack. For these novel settings we find our attacks can induce early-termination behaviors or divert an agent along an attacker-defined multi-step trajectory. Under both conditions environmental attacks significantly reduce agent capabilities to successfully follow user instructions.