WACV 2025 论文列表

Models for egocentric 3D and 4D reconstruction including few-shot interpolation and extrapolation settings can benefit from having images from exocentric viewpoints as supervision signals. No existing dataset provides the necessary mixture of complex dynamic and multi-view data. To facilitate the development of 3D and 4D reconstruction methods in the autonomous driving context we propose a Synthetic Ego-Exo Dynamic 4D (SEED4D) data generator and dataset. We present a customizable easy-to-use data generator for spatio-temporal multi-view data creation. Our open-source data generator allows the creation of synthetic data for camera setups commonly used in the NuScenes KITTI360 and Waymo datasets. Additionally SEED4D encompasses two large-scale multi-view synthetic urban scene datasets. Our static (3D) dataset encompasses 212k inward- and outward-facing vehicle images from 2k scenes while our dynamic (4D) dataset contains 16.8M images from 10k trajectories each sampled at 100 points in time with egocentric images exocentric images and LiDAR data. The datasets and the data generator can be found here: https://seed4d.github.io

Compositional image retrieval (CIR) is a multimodal learning task where a model combines a query image with a user-provided text modification to retrieve a target image. CIR finds applications in a variety of domains including product retrieval (e-commerce) and web search. Existing methods primarily focus on fully-supervised learning wherein models are trained on datasets of labeled triplets such as FashionIQ and CIRR. This poses two significant challenges: (i) curating such triplet datasets is labor intensive; and (ii) models lack generalization to unseen objects and domains. In this work we propose SCOT (Self-supervised COmpositional Training) a novel zero-shot compositional pretraining strategy that combines existing large image-text pair datasets with the generative capabilities of large language models to contrastively train an embedding composition network. Specifically we show that the text embedding from a large-scale contrastively-pretrained vision-language model can be utilized as proxy target supervision during compositional pretraining replacing the target image embedding. In zero-shot settings this strategy surpasses SOTA zero-shot compositional retrieval methods as well as many fully-supervised methods on standard benchmarks such as FashionIQ and CIRR.

Vision is essential for human navigation. The World Health Organization (WHO) estimates that 43.3 million people were blind in 2020 and this number is projected to reach 61 million by 2050. Modern scene understanding models could empower these people by assisting them with navigation obstacle avoidance and visual recognition capabilities. The research community needs high quality datasets for both training and evaluation to build these systems. And while datasets for autonomous vehicles are abundant there is a critical gap in datasets tailored for outdoor human navigation. This gap poses a major obstacle to the development of computer vision based Assistive Technologies. To overcome this obstacle we present SANPO a large-scale egocentric video dataset designed for dense prediction in outdoor human navigation environments. SANPO contains 701 stereo videos of 30+ seconds captured in diverse real-world outdoor environments across four geographic locations in the USA. Every frame has a high resolution depth map and 112K frames were annotated with temporally consistent dense video panoptic segmentation labels. The dataset also includes 1961 high-quality synthetic videos with pixel accurate depth and panoptic segmentation annotations to balance the noisy real world annotations with the high precision synthetic annotations. SANPO is already publicly available and is being used by applications like Project Guideline to train mobile models that help low-vision users run independently. To preserve anonymization during peer review a link to the dataset will be provided upon acceptance.

We propose Spatially-Aware open-set Network Dissection (SAND) a technique to identify and label the learned representation of the neurons of deep vision networks. Contrary to earlier open-vocabulary neuron explanation methods we also leverage a neuron's spatial pattern of activation to guide our predictions towards more accurate and relevant concepts while avoiding being misled by confounding visual information. We highlight important regions for a neuron through image masking which has the advantage of being able to block out irrelevant concepts from an image handling irregularly shaped activation regions and revealing the visual concepts that a neuron learns in order to identify objects. We use CLIP to connect highly activating image regions with descriptive concepts and measure the quality of our results through human evaluation. Further since such manual evaluation can be highly time consuming costly and unscalable we also propose an automated approach which uses image generation to get quantitative feedback on the generated concepts. Finally as an application of our interpretability method we demonstrate how it can be tuned to the medical domain. Our code is available at https://github.com/Trustworthy-ML-Lab/SAND.

Polyp segmentation in colonoscopy images is crucial for detecting colorectal cancer but is challenging due to variations in the structure color and size of polyps as well as the lack of clear boundaries with surrounding tissues. Traditional segmentation models based on Convolutional Neural Networks (CNNs) struggle to capture detailed patterns and global context limiting their performance. Vision Transformer (ViT)-based models address some of these issues but have difficulties in capturing local context and lack strong zero-shot generalization. To this end we propose the Mamba-guided Segment Anything Model (SAM-Mamba) for efficient polyp segmentation. Our approach introduces a Mamba-Prior module in the encoder to bridge the gap between the general pre-trained representation of SAM and polyp-relevant trivial clues. It injects salient cues of polyp images into the SAM image encoder as a domain prior while capturing global dependencies at various scales leading to more accurate segmentation results. Extensive experiments on five benchmark datasets show that SAM-Mamba outperforms traditional CNN ViT and Adapter-based models in both quantitative and qualitative measures. Additionally SAM-Mamba demonstrates excellent adaptability to unseen datasets making it highly suitable for real-time clinical use.

This paper addresses the domain adaptation challenge for semantic segmentation in medical imaging. Despite the impressive performance of recent foundational segmentation models like SAM on natural images they struggle with medical domain images. Beyond this recent approaches that perform end-to-end fine-tuning of models are simply not computationally tractable. To address this we propose a novel SAM adapter approach that minimizes the number of trainable parameters while achieving comparable performances to full fine-tuning. The proposed SAM adapter is strategically placed in the mask decoder offering excellent and broad generalization capabilities and improved segmentation across both fully supervised and test-time domain adaptation tasks. Extensive validation on four datasets showcases the adapter's efficacy outperforming existing methods while training less than 1% of SAM's total parameters.

Managing chronic wounds is a global challenge that can be alleviated by the adoption of automatic systems for clinical wound assessment from consumer-grade videos. While 2D image analysis approaches are insufficient for handling the 3D features of wounds existing approaches utilizing 3D reconstruction methods have not been thoroughly evaluated. To address this gap this paper presents a comprehensive study on 3D wound reconstruction from consumer-grade videos. Specifically we introduce the SALVE dataset comprising video recordings of realistic wound phantoms captured with different cameras. Using this dataset we assess the accuracy and precision of state-of-the-art methods for 3D reconstruction ranging from traditional photogrammetry pipelines to advanced neural rendering approaches. In our experiments we observe that photogrammetry approaches do not provide smooth surfaces suitable for precise clinical measurements of wounds. Neural rendering approaches show promise in addressing this issue advancing the use of this technology in wound care practices.

Decentralized training enables learning with distributed datasets generated at different locations without relying on a central server. In realistic scenarios the data distribution across these sparsely connected learning agents can be significantly heterogeneous leading to local model over-fitting and poor global model generalization. Another challenge is the high communication cost of training models in such a peer-to-peer fashion without any central coordination. In this paper we jointly tackle these two-fold practical challenges by proposing SADDLe a set of sharpness-aware decentralized deep learning algorithms. SADDLe leverages Sharpness-Aware Minimization (SAM) to seek a flatter loss landscape during training resulting in better model generalization as well as enhanced robustness to communication compression. We present two versions of our approach and demonstrate its effectiveness through extensive experiments on various Computer Vision datasets (CIFAR-10 CIFAR-100 Imagenette and ImageNet) model architectures and graph topologies. Our results show that SADDLe leads to 1-20% improvement in test accuracy as compared to existing techniques while incurring a minimal accuracy drop ( 1%) in the presence of up to 4x compression.

Temporal Action Localization (TAL) is a complex task that poses relevant challenges particularly when attempting to generalize on new - unseen - domains in real-world applications. These scenarios despite realistic are often neglected in the literature exposing these solutions to important performance degradation. In this work we tackle this issue by introducing for the first time an approach for Unsupervised Domain Adaptation (UDA) in sparse TAL which we refer to as Semantic Adversarial unsupervised Domain Adaptation (SADA). Our contributions are threefold: (1) we pioneer the development of a domain adaptation model that operates on realistic sparse action detection benchmarks; (2) we tackle the limitations of global-distribution alignment techniques by introducing a novel adversarial loss that is sensitive to local class distributions ensuring finer-grained adaptation; and (3) we present a novel set of benchmarks based on EpicKitchens100 and CharadesEgo that evaluate multiple domain shifts in a comprehensive manner. Our experiments indicate that SADA improves the adaptation across domains when compared to fully supervised state-of-the-art and alternative UDA methods attaining a performance boost of up to 6.14% mAP. The code is publicly available at https://github.com/davidpujol/SADA.

Recent self-supervised clustering-based pre-training techniques like DINO and CriBo have shown impressive results for downstream detection and segmentation tasks. However real-world applications such as autonomous driving face challenges with imbalanced object class and size distributions and complex scene geometries. In this paper we propose S3PT a novel scene semantics and structure guided clustering to provide more scene-consistent objectives for self-supervised training. Specifically our contributions are threefold: First we incorporate semantic distribution consistent clustering to encourage better representation of rare classes such as motorcycles or animals. Second we introduce object diversity consistent spatial clustering to handle imbalanced and diverse object sizes ranging from large background areas to small objects such as pedestrians and traffic signs. Third we propose a depth-guided spatial clustering to regularize learning based on geometric information of the scene thus further refining region separation on the feature level. Our learned representations significantly improve performance in downstream semantic segmentation and 3D object detection tasks on the nuScenes nuImages and Cityscapes datasets and show promising domain translation properties.

Figure skating automatic scoring is the task of estimating the competition score of a performance video. The technical element score (TES) aggregates the technical quality (grade of execution) and difficulty (base value) scores for each element. Most prior work adapted from short-term action quality assessment entangle difficulty and quality and compute TES for the entire video reducing interpretability for athletes. This is mainly due to a lack of element segmentation and difficulty annotations in existing datasets. Motivated by increasing interpretability we propose a novel method that implicitly segments a video to produce element-level representations and uses adherence with a natural language rubric to score each element without needing additional annotations. We compute element-level representations using learnable element queries in a transformer and propose implicit segmentation regularization to encourage element queries to attend to elements rather than background transitions between elements (most of video). Additionally we use the element list (sequence of elements) to isolate difficulty just like judges who receive the routine list in advance so we can focus on the more critical problem of how well elements are done. These components significantly improve interpretability scoring precision and ranking capability. Code is released at https://arushirai1.github.io/rcs-project.

We present RopeTP a novel framework that combines Robust pose estimation with a diffusion Trajectory Prior to reconstruct global human motion from videos. At the heart of RopeTP is a hierarchical attention mechanism that significantly improves context awareness which is essential for accurately inferring the posture of occluded body parts. This is achieved by exploiting the relationships with visible anatomical structures enhancing the accuracy of local pose estimations. The improved robustness of these local estimations allows for the reconstruction of precise and stable global trajectories. Additionally RopeTP incorporates a diffusion trajectory model that predicts realistic human motion from local pose sequences. This model ensures that the generated trajectories are not only consistent with observed local actions but also unfold naturally over time thereby improving the realism and stability of 3D human motion reconstruction. Extensive experimental validation shows that RopeTP surpasses current methods on two benchmark datasets particularly excelling in scenarios with occlusions. It also outperforms methods that rely on SLAM for initial camera estimates and extensive optimization delivering more accurate and realistic trajectories.

High-quality portrait photography has become an essential function in our daily lives. However due to the limited aperture and focal length of a smartphone camera images captured by a smartphone cannot match the same level of bokeh effect by a digital single-lens reflex camera. A typical solution on a smartphone is to simulate out-of-focus effects from an all-in-focus image where the key is robust depth estimation and portrait matting. To achieve this we propose a multi-stage multi-branch matting network to estimate a strand-level portrait alpha mask which is then used to refine the coarse depth map from the pre-trained model. Combining the input portrait image with the estimated depth map and alpha mask we propose a learning-free optimization mechanism to construct a multi-plane image (MPI) representation for depth-of-field synthesis. The MPI consists of multiple layers of disk-blurred images with kernel size proportional to the absolute depth distance to the focus layer. Then a depth-aware blurring process is applied to enforce the bokeh effect. Besides each MPI layer has an alpha channel controlling the visibility according to the corresponding depth. Finally an image with bokeh is rendered by compositing all MPI layers. We conduct comprehensive experiments to evaluate our method which demonstrates that our method can generate more accurate alpha masks and more realistic images with bokeh compared to prior work.

Novelty detection seeks to identify samples deviating from a known distribution yet data shifts in a multitude of ways and only a few consist of relevant changes. Aligned with out-of-distribution generalization literature we advocate for a formal distinction between task-relevant semantic or content changes and irrelevant style changes. This distinction forms the basis for robust novelty detection emphasizing the identification of semantic changes resilient to style distributional shifts. To this end we introduce Stylist a method that utilizes pretrained large-scale model representations to selectively discard environment-biased features. By computing per-feature scores based on feature distribution distances between environments Stylist effectively eliminates features responsible for spurious correlations enhancing novelty detection performance. Evaluations on adapted domain generalization datasets and a synthetic dataset demonstrate Stylist's efficacy in improving novelty detection across diverse datasets with stylistic and content shifts. We make our code available at https://github.com/bit-ml/Stylist.

Advances in machine learning algorithms for sensor fusion have significantly improved the detection and prediction of other road users thereby enhancing safety. However even a small angular displacement in the sensor's placement can cause significant degradation in output especially at long range. In this paper we demonstrate a simple yet generic and efficient multi-task learning approach that not only detects misalignment between different sensor modalities but is also robust against them for long-range perception. Along with the amount of misalignment our method also predicts calibrated uncertainty which can be useful for filtering and fusing predicted misalignment values over time. In addition we show that the predicted misalignment parameters can be used for self-correcting input sensor data further improving the perception performance under sensor misalignment.

The ability of robots to manipulate objects relies heavily on their aptitude for visual perception. In domains characterized by cluttered scenes and high object variability most methods call for vast labeled datasets laboriously hand-annotated with the aim of training capable models. Once deployed the challenge of generalizing to unfamiliar objects implies that the model must evolve alongside its domain. To address this we propose a novel framework that combines Semi-Supervised Learning (SSL) with Learning Through Interaction (LTI) allowing a model to learn by observing scene alterations and leverage visual consistency despite temporal gaps without requiring curated data of interaction sequences. As a result our approach exploits partially annotated data through self-supervision and incorporates temporal context using pseudo-sequences generated from unlabeled still images. We validate our method on two common benchmarks ARMBench mix-object-tote and OCID where it achieves state-of-the-art performance. Notably on ARMBench we attain an AP50 of 86.37 almost a 20% improvement over existing work and obtain remarkable results in scenarios with extremely low annotation achieving an AP50 score of 84.89 with just 1% of annotated data compared to previous state of the art of 82 which targeted the fully annotated dataset.

Covariance matrices outperform first-order features in many tasks attracting considerable attention from the computer vision research community. Covariance matrices encode second-order statistics between features at the same time it is robust to noise. Based on this we propose representing images by covariance matrices and defining a loss function that measures the distance between them through the Riemannian distance. Motivated by the robustness and invariance properties of the affine invariant Riemannian metric the proposed method was validated in printer-proof data transmission which is a challenging task due to the trade-off between image quality and message recovery capabilities after printing and digitization procedures. The effectiveness of this approach was systematically assessed using MS COCO and IMM Face datasets. The results demonstrated that the proposed approach outperforms conventional methods that use Euclidean distance generating encoded images with better quality and achieving higher recovery accuracy in printed images. Additionally a broader application of the proposed loss was successfully tested in image generation tasks using generative adversarial networks (GANs).

For object segmentation in medical images deep neural networks (DNNs) typically perform poorly on out-of-distribution (OOD) images stemming from the large variability in image-acquisition equipment and protocols across sites. However compared to such variability in the acquired medical images we observe that the variability in the underlying object-segmentation maps is far lower. Thus we propose a novel DNN framework to model this variability in segmentation maps and leverage it to revive poor segmentations produced by existing DNNs on OOD images. Our DNN framework (i) learns the principal modes of variation in a class of segmentation maps (ii) models each segmentation map using a low-dimensional mixture-of-modes latent representation on a simplex (iii) enables sampling-free variational learning and uncertainty estimation and (iv) trains using small in-distribution image sets. When OOD-image segmentations are extremely poor we propose a novel human-in-the-loop method needing minuscule human intervention. Results using 6 publicly-available datasets and 8 existing DNN segmenters show the benefits of our framework in OOD-image object segmentation.

Machine unlearning an emerging research topic focusing on data privacy compliance enables trained models to erase information learned from specific data. While many existing methods indirectly address this issue by intentionally injecting incorrect supervision they often result in drastic and unpredictable changes to decision boundaries and feature spaces leading to training instability and undesired side effects. To address this challenge more fundamentally we first analyze the changes in latent feature spaces between the original and retrained models and observe that the feature representations of samples not included in training are closely aligned with the feature manifolds of previously seen samples. Building on this insight we introduce a novel evaluation metric for machine unlearning coined dimensional alignment which measures the alignment between the eigenspaces of the forget and retain sets. We incorporate this metric as a regularizer loss to develop a robust and stable unlearning framework which is further enhanced by a self-distillation loss and an alternating training scheme. Our framework effectively eliminates information from the forget set while preserving knowledge from the retain set. Finally we identify critical flaws in existing evaluation metrics for machine unlearning and propose new tools that more accurately capture its fundamental objectives.