WACV 2025 Papers

In this work we present Multiformer a novel approach to depth-aware video panoptic segmentation (DVPS) based on the mask transformer paradigm. Our method learns object representations that are shared across segmentation monocular depth estimation and object tracking subtasks. In contrast to recent unified approaches that progressively refine a common object representation we propose a hybrid method using task-specific branches within each decoder block ultimately fusing them into a shared representation at the block interfaces. Extensive experiments on the Cityscapes-DVPS and SemKITTI-DVPS datasets demonstrate that Multiformer achieves state-of-the-art performance across all DVPS metrics outperforming previous methods by substantial margins. With a ResNet-50 backbone Multiformer surpasses the previous best result by 3.0 DVPQ points while also improving depth estimation accuracy. Using a Swin-B backbone Multiformer further improves performance by 4.0 DVPQ points. Multiformer also provides valuable insights into the design of multi-task decoder architectures.

Video moment retrieval (VMR) identifies a specific moment in an untrimmed video for a given natural language query. This task is prone to suffer the weak alignment problem innate in video datasets. Due to the ambiguity a query does not fully cover the relevant details of the corresponding moment or the moment may contain misaligned and irrelevant frames potentially limiting further performance gains. To tackle this problem we propose a background-aware moment detection transformer (BM-DETR). Our model adopts a contrastive approach carefully utilizing the negative queries matched to other moments in the video. Specifically our model learns to predict the target moment from the joint probability of each frame given the positive query and the complement of negative queries. This leads to effective use of the surrounding background improving moment sensitivity and enhancing overall alignments in videos. Extensive experiments on four benchmarks demonstrate the effectiveness of our approach. Our code is available at https://github.com/minjoong507/BM-DETR.

Individuals who are blind or have low vision (BLV) are at a heightened risk of sharing private information if they share photographs they have taken. To facilitate developing technologies that can help them preserve privacy we introduce BIV-Priv-Seg the first localization dataset originating from people with visual impairments that shows private content. It contains 1028 images with segmentation annotations for 16 private object categories. We first characterize BIV-Priv-Seg and then evaluate modern models' performance for locating private content in the dataset. We find modern models struggle most with locating private objects that are not salient small and lack text as well as recognizing when private content is absent from an image. We facilitate future extensions by sharing our new dataset with the evaluation server at https://vizwiz.org/tasks-and-datasets/object-localization/.

Reconstruction of deformable scenes from endoscopic videos is important for many applications such as intraoperative navigation surgical visual perception and robotic surgery. It is a foundational requirement for realizing autonomous robotic interventions for minimally invasive surgery. However previous approaches in this domain have been limited by their modular nature and are confined to specific camera and scene settings. Our work adopts the Neural Radiance Fields (NeRF) approach to learning 3D implicit representations of scenes that are both dynamic and deformable over time and furthermore with unknown camera poses. This work removes the constraints of known camera poses and overcomes the drawbacks of the state-of-the-art unstructured dynamic scene reconstruction technique which relies on the static part of the scene for accurate reconstruction. Through several experimental datasets we demonstrate the versatility of our proposed model to adapt to diverse camera and scene settings and show its promise for both current and future robotic surgical systems.

Extracting multiscale contextual information and higher-order correlations among skeleton sequences using Graph Convolutional Networks (GCNs) alone is inadequate for effective action classification. Hypergraph convolution addresses the above issues but cannot harness the long-range dependencies. The transformer proves to be effective in capturing these dependencies and making complex contextual features accessible. We propose an Autoregressive Adaptive HyperGraph Transformer (AutoregAd-HGformer) model for in-phase (autoregressive and discrete) and out-phase (adaptive) hypergraph generation. The vector quantized in-phase hypergraph equipped with powerful autoregressive learned priors produces a more robust and informative representation suitable for hyperedge formation. The out-phase hypergraph generator provides a model-agnostic hyperedge learning technique to align the attributes with input skeleton embedding. The hybrid (supervised and unsupervised) learning in AutoregAd-HGformer explores the action-dependent feature along spatial temporal and channel dimensions. The extensive experimental results and ablation study indicate the superiority of our model over state-of-the-art hypergraph architectures on the NTU RGB+D NTU RGB+D 120 and NW-UCLA datasets.

Positioning patients for scanning and interventional procedures is a critical task that requires high precision and accuracy. The conventional workflow involves manually adjusting the patient support to align the center of the target body part with the laser projector or other guiding devices. This process is not only time-consuming but also prone to inaccuracies. In this work we propose an automated patient positioning system that utilizes a camera to detect specific hand gestures from technicians allowing users to indicate the target patient region to the system and initiate automated positioning. Our approach relies on a novel multi-stage pipeline to recognize and interpret the technicians' gestures translating them into precise motions of medical devices. We evaluate our proposed pipeline during actual MRI scanning procedures using RGB-Depth cameras to capture the process. Results show that our system achieves accurate and precise patient positioning with minimal technician intervention. Furthermore we validate our method on HaGRID a large-scale hand gesture dataset demonstrating its effectiveness in hand detection and gesture recognition.

Large Vision-Language Models (LVLMs) have received widespread attentions for advancing the interpretable self-driving. Existing evaluations of LVLMs primarily focus on multi-faceted capabilities in natural circumstances lacking automated and quantifiable assessment for self-driving let alone the severe road corner cases. In this work we propose CODA-LM the very first benchmark for the automatic evaluation of LVLMs for self-driving corner cases. We adopt a hierarchical data structure and prompt powerful LVLMs to analyze complex driving scenes and generate high-quality pre-annotations for the human annotators while for LVLM evaluation we show that using the text-only large language models (LLMs) as judges reveals even better alignment with human preferences than the LVLM judges. Moreover with our CODA-LM we build CODA-VLM a new driving LVLM surpassing all open-sourced counterparts on CODA-LM. Our CODA-VLM performs comparably with GPT-4V even surpassing GPT-4V by +21.42% on the regional perception task. We hope CODA-LM can become the catalyst to promote interpretable self-driving empowered by LVLMs.

Segment Anything Model (SAM) is one of the pioneering prompt-based foundation models for image segmentation and has been rapidly adopted for various medical imaging applications. However in clinical settings creating effective prompts is notably challenging and time-consuming requiring the expertise of domain specialists such as physicians. This requirement significantly diminishes SAM's primary advantage--its interactive capability with end users--in medical applications. Moreover recent studies have indicated that SAM originally designed for 2D natural images performs suboptimally on 3D medical image segmentation tasks. This subpar performance is attributed to the domain gaps between natural and medical images and the disparities in spatial arrangements between 2D and 3D images particularly in multi-organ segmentation applications. To overcome these challenges we present a novel technique termed AutoProSAM. This method automates 3D multi-organ CT-based segmentation by leveraging SAM's foundational model capabilities without relying on domain experts for prompts. The approach utilizes parameter-efficient adaptation techniques to adapt SAM for 3D medical imagery and incorporates an effective automatic prompt learning paradigm specific to this domain. By eliminating the need for manual prompts it enhances SAM's capabilities for 3D medical image segmentation and achieves state-of-the-art (SOTA) performance in CT-based multi-organ segmentation tasks. The code is in this link.

Image attribute editing is a widely researched area fueled by the recent advancements in deep generative models. Existing methods treat semantic attributes as binary and do not allow the user to generate multiple variations of the attribute edits. This limits the applications of editing methods in the real world e.g. exploring multiple eyeglass variations on an e-commerce platform. In this work we present a technique to generate a collection of diverse attribute edits and a principled way to explore them. Generation and controlled exploration of attribute variations is challenging as it requires fine control over the attribute styles while preserving other attributes and the identity of the subject. Capitalizing on the attribute disentanglement property of the latent spaces of pretrained GANs we represent the attribute edits in this space. Next we train a diffusion model to model these latent directions of edits. We propose a coarse-to-fine sampling strategy to explore these variations in a controlled manner. Extensive experiments on various datasets establish the effectiveness and generalization of the proposed approach for the generation and controlled exploration of diverse attribute edits. Code is available at - rishubhpar.github.io/attributediffusion

Masked autoencoders (MAEs) have established themselves as a powerful pre-training method for computer vision tasks. While vanilla MAEs put equal emphasis on reconstructing the individual parts of the image we propose to inform the reconstruction process through an attention-guided loss function. By leveraging advances in unsupervised object discovery we obtain an attention map of the scene which we employ in the loss function to put increased emphasis on reconstructing relevant objects. Thus we incentivize the model to learn improved representations of the scene for a variety of tasks. Our evaluations show that our pre-trained models produce off-the-shelf representations more effective than the vanilla MAE for such tasks demonstrated by improved linear probing and k-NN classification results on several benchmarks while at the same time making ViTs more robust against varying backgrounds and changes in texture.

Domain adaptation methods for object detection (OD) strive to mitigate the impact of distribution shifts by promoting feature alignment across source and target domains. Multi-source domain adaptation (MSDA) allows leveraging multiple annotated source datasets and unlabeled target data to improve the accuracy and robustness of the detection model. Most state-of-the-art MSDA methods for OD perform feature alignment in a class-agnostic manner. This is challenging since the objects have unique modality information due to variations in object appearance across domains. A recent prototype-based approach proposed a class-wise alignment yet it suffers from error accumulation caused by noisy pseudo-labels that can negatively affect adaptation with imbalanced data. To overcome these limitations we propose an attention-based class-conditioned alignment method for MSDA designed to align instances of each object category across domains. In particular an attention module combined with an adversarial domain classifier allows learning domain-invariant and class-specific instance representations. Experimental results on multiple benchmarking MSDA datasets indicate that our method outperforms state-of-the-art methods and exhibits robustness to class imbalance achieved through a conceptually simple class-conditioning strategy. Our code is available at: https://github.com/imatif17/ACIA.

This paper proposes a new dataset of archaeological artefacts for evaluating 3D reconstruction methods and questions the notion of ground truth. Indeed 3D reconstruction of archaeological objects can be carried out using either scanners or photographic methods. It turns out that multi-view stereo (MVS) faithfully reconstructs the overall shape of an object on a par with a hand-held scanner while calibrated photometric stereo (CPS) reveals relief details. The restitution of low- and high-frequencies is therefore the prerogative of distinct methods which indicates that the ground truth and the metric used for evaluation should be chosen in view of the target frequencies. This observation led us to combine MVS and CPS using MVS to calibrate the illumination used by CPS. We demonstrate on our dataset of archaeological objects that this original 3D reconstruction method indeed combines the advantages of MVS and CPS. Our proposed dataset can be accessed here: https://github.com/BenjaminCoupry/the-MAD-project.

Neural Networks (NNs) have surpassed human accuracy in image classification on ImageNet yet they often lack robustness against image corruption i.e. corruption robustness with such robustness being seemingly effortless for human perception. In this paper we propose visually-continuous corruption robustness (VCR) - an extension of corruption robustness to allow assessing it over the wide and continuous range of changes that correspond to the human perceptive quality (i.e. from the original image to the full distortion of all perceived visual information) along with two novel human-aware metrics for NN evaluation. To compare VCR of NNs with human perception we conducted extensive experiments on 14 commonly used image corruptions with 7718 human participants and state-of-the-art robust NN models with different training objectives (e.g. standard adversarial corruption robustness) different architectures (e.g. convolution NNs vision transformers) and different amounts of training data augmentation. Our study showed that: 1) assessing robustness against continuous corruption can reveal insufficient robustness undetected by existing benchmarks; as a result 2) the gap between NN and human robustness is larger than previously known; and finally 3) some image corruptions have a similar impact on human perception offering opportunities for more cost-effective robustness assessments.

Class Incremental Learning (CIL) models are designed to continuously learn new classes without forgetting previously learned ones often relying on an exemplar set to retain a portion of knowledge from previously learned classes. However their vulnerability to adversarial attacks under novel and unexplored conditions remains unstudied. In this work we are the first to evaluate the robustness of exemplar-based CIL models using a non-overlapping dataset where the dataset is independent of the training and test sets of the target model. We propose and implement a novel black-box attack framework targeting the exemplar set of class incremental learning models using zero-overlapping data. Specifically we focus on scenarios where the target model provides only hard-label predictions without interactive access. Our experimental evaluation covers a range of exemplar-based incremental learning algorithms different surrogate models and black-box attack options. Our findings reveal significant vulnerabilities in exemplar-based CIL models to poisoning-based attacks using a non-overlapping dataset.

The increasing capabilities of deep neural networks for re-identification combined with the rise in public surveillance in recent years pose a substantial threat to individual privacy. Event cameras were initially considered as a promising solution since their output is sparse and therefore difficult for humans to interpret. However recent advances in deep learning proof that neural networks are able to reconstruct high-quality grayscale images and re-identify individuals using data from event cameras. In our paper we contribute a crucial ethical discussion on data privacy and present the first event anonymization pipeline to prevent re-identification not only by humans but also by neural networks. Our method effectively introduces learnable data-dependent noise to cover personally identifiable information in raw event data reducing attackers' re-identification capabilities by up to 60% while maintaining substantial information for the performing of downstream tasks. Moreover our anonymization generalizes well on unseen data and is robust against image reconstruction and inversion attacks. Code: https://github.com/dfki-av/AnonyNoise

Recent advances in multimodal foundation models have set new standards in few-shot anomaly detection. This paper explores whether high-quality visual features alone are sufficient to rival existing state-of-the-art vision-language models. We affirm this by adapting DINOv2 for one-shot and few-shot anomaly detection with a focus on industrial applications. We show that this approach does not only rival existing techniques but can even outmatch them in many settings. Our proposed vision-only approach AnomalyDINO follows the well-established patch-level deep nearest neighbor paradigm and enables both image-level anomaly prediction and pixel-level anomaly segmentation. The approach is methodologically simple and training-free and thus does not require any additional data for fine-tuning or meta-learning. Despite its simplicity AnomalyDINO achieves state-of-the-art results in one- and few-shot anomaly detection (e.g. pushing the one-shot performance on MVTec-AD from an AUROC of 93.1% to 96.6%). The reduced overhead coupled with its outstanding few-shot performance makes AnomalyDINO a strong candidate for fast deployment e.g. in industrial contexts.

Recent advancements in computer vision have led to a renewed interest in developing assistive technologies for individuals with visual impairments. Although extensive research has been conducted in the field of computer vision-based assistive technologies most of the focus has been on understanding contexts in images rather than addressing their physical safety and security concerns. To address this challenge we propose the first step towards detecting anomalous situations for visually impaired people by observing their entire surroundings using an egocentric 360-degree camera. We first introduce a novel egocentric 360-degree video dataset called VIEW360 (Visually Impaired Equipped with Wearable 360-degree camera) which contains abnormal activities that visually impaired individuals may encounter such as shoulder surfing and pickpocketing. Furthermore we propose a new architecture called the FDPN (Frame and Direction Prediction Network) which facilitates frame-level prediction of abnormal events and identifying of their directions. Finally we evaluate our approach on our VIEW360 dataset and the publicly available UCF-Crime and Shanghaitech datasets demonstrating state-of-the-art performance.

Video generation has drawn significant interest recently pushing the development of large-scale models capable of producing realistic videos with coherent motion. Due to memory constraints these models typically generate short video segments that are then combined into long videos. The merging process poses a significant challenge as it requires ensuring smooth transitions and overall consistency. In this paper we introduce Anchored Diffusion a novel method for synthesizing relatively long and seamless videos. We extend Diffusion Transformers (DiTs) to incorporate temporal information creating our sequence-DiT (sDiT) model for generating short video segments. Unlike previous works we train our model on video sequences with random non-uniform temporal spacing and incorporate temporal information via external guidance increasing flexibility and allowing it to capture both short and long-term relationships. Furthermore during inference we leverage the transformer architecture to modify the diffusion process generating a batch of non-uniform sequences anchored to a common frame ensuring consistency regardless of temporal distance. To demonstrate our method we focus on face reenactment a task of transforming the action from the driving video to the source face. Through comprehensive experiments we show our approach outperforms current techniques in producing longer consistent high-quality videos while offering editing capabilities.

With the advances in generative adversarial networks (GANs) and neural rendering 3D relightable face generation has received significant attention. Among the existing methods a particularly successful technique uses an implicit lighting representation and generates relit images through the product of synthesized albedo and light-dependent shading images. While this approach produces high-quality results with intricate shading details it often has difficulty producing relit images with consistent skin tones particularly when the lighting condition is extracted from images of individuals with dark skin. Additionally this technique is biased towards producing albedo images with lighter skin tones. Our main observation is that this problem is rooted in the biased spherical harmonics (SH) coefficients used during training. Following this observation we conduct an analysis and demonstrate that the bias appears not only in band 0 (DC term) but also in the other bands of the estimated SH coefficients. We then propose a simple but effective strategy to mitigate the problem. Specifically we normalize the SH coefficients by their DC term to eliminate the inherent magnitude bias while statistically align the coefficients in the other bands to alleviate the directional bias. We also propose a scaling strategy to match the distribution of illumination magnitude in the generated images with the training data. Through extensive experiments we demonstrate the effectiveness of our solution in increasing the skin tone consistency and mitigating bias.