WACV 2025 Papers

3D models particularly AI-generated ones have witnessed a recent surge across various industries such as entertainment. Hence there is an alarming need to protect the intellectual property and avoid the misuse of these valuable assets. As a viable solution to address these concerns we rigorously define the novel task of automated 3D visible watermarking in terms of two competing aspects: watermark quality and asset utility. Moreover we propose a method of embedding visible watermarks that automatically determines the right location orientation and number of watermarks to be placed on arbitrary 3D assets for high watermark quality and asset utility. Our method is based on a novel rigid-body optimization that uses back-propagation to automatically learn transforms for ideal watermark placement. In addition we propose a novel curvature-matching method for fusing the watermark into the 3D model that further improves readability and security. Finally we provide a detailed experimental analysis on two benchmark 3D datasets validating the superior performance of our approach in comparison to baselines. Code and demo are available here.

Neural networks are prone to be vulnerable to adversarial attacks and domain shifts. Adversarial-driven methods including adversarial training and adversarial augmentation have been frequently proposed to improve the model's robustness against adversarial attacks and distribution-shifted samples. Nonetheless recent research on adversarial attacks has cast a spotlight on the robustness lacuna against attacks targeted at deep semantic layers. Our analysis reveals that previous adversarial-driven methods tend to generate overpowering perturbations in deep semantic layers leading to distortion of the training for these layers. This can be primarily attributed to the exclusive utilization of loss functions on the output layer for adversarial gradient generation. This inherent practice projects an excessive adversarial impact on the deep semantic layers elevating the difficulty of training such layers. Therefore from the standing point of relaxing the excessive perturbations in the deep semantic layer and diversifying the adversarial gradients to ensure robust training for deep semantic layers this paper proposes a novel Stochastic Loss Integration Method (SLIM) which can be instantiated into the existing adversarial-driven methods in a plug-and-play manner. Experimental results across diverse tasks including classification and segmentation as well as various areas such as adversarial robustness and domain generalization validate the effectiveness of our proposed method. Furthermore we provide an in-depth analysis to offer a comprehensive understanding of layer-wise training involving various loss terms.

In this paper we address the challenge of performing open-vocabulary video instance segmentation (OV-VIS) in real-time. We analyze the computational bottlenecks of state-of-the-art foundation models that performs OV-VIS and propose a new method TROY-VIS that significantly improves processing speed while maintaining high accuracy. We introduce three key techniques: (1) Decoupled Attention Feature Enhancer to speed up information interaction between different modalities and scales; (2) Flash Embedding Memory for obtaining fast text embeddings of object categories; and (3) Kernel Interpolation for exploiting the temporal continuity in videos. Our experiments demonstrate that TROY-VIS achieves the best trade-off between accuracy and speed on two large-scale OV-VIS benchmarks BURST and LV-VIS running 20x faster than GLEE-Lite (25 FPS v.s. 1.25 FPS) with comparable or even better accuracy. These results demonstrate TROY-VIS's potential for real-time applications in dynamic environments such as mobile robotics and augmented reality. Code and model will be released at https://github.com/google-research/troyvis.

With the emerging trend of large generative models ControlNet is introduced to enable users to fine-tune pre-trained models with their own data for various use cases. A natural question arises: how can we train ControlNet models while ensuring users' data privacy across distributed devices? We first propose a new distributed learning structure that eliminates the need for the server to send gradients based on split learning. We discover that in the context of fine-tuning ControlNet with split learning most existing attacks are ineffective except for two mentioned in previous literature. To counter these threats we leverage the properties of diffusion models and design a new timestep sampling policy during forward processes. We also propose a privacy-preserving activation function and a method to prevent private text prompts from leaving clients tailored for image generation with diffusion models. Our experimental results demonstrate that our algorithms and systems greatly enhance the efficiency of distributed fine-tuning for ControlNet while ensuring users' data privacy without compromising image generation quality.

As the diversity of real-world object categories increases the need for sophisticated classification methods also grows. However Novel Category Discovery (NCD) which aims to predict unseen categories often falls short in scenarios where new categories are constantly updated and data distributions are potentially biased. In addition existing dynamic NCD approaches assume that incremental stages introduce a fixed number of new classes and often overlook distributional biases in real-world classes. To address these limitations we propose a novel framework Novel Category Discovery for Dynamic Long-Tailed distribution (NCD-DLT) which deals with the more realistic and challenging scenario where imbalanced unlabeled data are introduced incrementally and sporadically over time. Unlike conventional methods requiring k-means clustering on all test samples our approach identifies novel categories on-the-fly predicting categories for individual data points as they arrive. We propose an advanced hash-based clustering technique leveraging a double-hashing strategy to mitigate collisions and incorporating a greedy hash regularization loss for sparse representations to enhance clustering capabilities. Furthermore we implement distillation losses during training to preserve the model's discriminative power across stages without forgetting prior knowledge. Finally we introduce a novel graph merging algorithm based on the Hash Hamming Graph revealing the dataset's clustering structure. It serves as a mechanism for pseudo-labeling in training and acts as a post-processing tool reallocating less confident samples to more appropriate clusters. Our comprehensive approach addresses the limitations of existing NCD methods in the dynamic scenario of novel category discovery in long-tailed distributions demonstrating improved accuracy for both uniform and long-tailed scenarios.

We introduce an industrial Head Blending pipeline for the task of seamlessly integrating an actor's head onto a target body in digital content creation. The key challenge stems from discrepancies in head shape and hair structure which lead to unnatural boundaries and blending artifacts. Existing methods treat foreground and background as a single task resulting in suboptimal blending quality. To address this problem we propose CHANGER a novel pipeline that decouples background integration from foreground blending. By utilizing chroma keying for artifact-free background generation and introducing Head shape and long Hair augmentation (H2 augmentation) to simulate a wide range of head shapes and hair styles CHANGER improves generalization on innumerable various real-world cases. Furthermore our Foreground Predictive Attention Transformer (FPAT) module enhances foreground blending by predicting and focusing on key head and body regions. Quantitative and qualitative evaluations on benchmark datasets demonstrate that our CHANGER outperforms state-of-the-art methods delivering high-fidelity industrial-grade results.

The generalization capability of a face anti-spoofing (FAS) model is critical to its practicality in the real world. Recent studies have theoretically and empirically uncovered that neural networks tend to exploit easy-to-learn frequency sets for decisions. These simplicity-biased representations depending on what best simplifies the training objective may hamper generalization. This paper thus focuses on mitigating the frequency shortcut learning of prior FAS models for improved generalization. Specifically we introduce a frequency-aware autoencoder to retain more frequency details in intermediate features via reconstruction facilitating comprehensive judgment of FAS. Based on the encoder output we propose a dynamic frequency masking mechanism to select and suppress the probable shortcut bands during training enabling broader horizons on under-explored frequencies. Moreover we employ a style inhibited modulation to weaken stylized information in frequency space to reduce the reliance on spurious style features. Experiment results on generalized FAS benchmarks verify the superiority of our framework over existing methods. Our code has been integrated into this project: https://github.com/VISION-SJTU/UniDefense.

Unsupervised anomaly detection (UAD) from images strives to model normal data distributions creating discriminative representations to distinguish and precisely localize anomalies. Despite recent advancements in the efficient and unified one-for-all scheme challenges persist in accurately segmenting anomalies for further monitoring. Moreover this problem is obscured by the widely-used AUROC metric under imbalanced UAD settings. This motivates us to emphasize the significance of precise segmentation of anomaly pixels using pAP and DSC as metrics. To address the unsolved segmentation task we introduce the Unified Anomaly Segmentation (UniAS). UniAS presents a multi-level hybrid pipeline that progressively enhances normal information from coarse to fine incorporating a novel multi-granularity gated CNN (MGG-CNN) into Transformer layers to explicitly aggregate local details from different granularities. UniAS achieves state-of-the-art anomaly segmentation performance attaining 65.12/59.33 and 40.06/32.50 in pAP/DSC on the MVTec-AD and VisA datasets respectively surpassing previous methods significantly. The codes are shared at https://github.com/Mwxinnn/UniAS.

Text-Video Retrieval (TVR) methods typically match query-candidate pairs by aligning text and video features in coarse-grained fine-grained or combined (coarse-to-fine) manners. However these frameworks predominantly employ a one(query)-to-one(candidate) alignment paradigm which struggles to discern nuanced differences among candidates leading to frequent mismatches. Inspired by Comparative Judgement in human cognitive science where decisions are made by directly comparing items rather than evaluating them independently we propose TokenBinder. This innovative two-stage TVR framework introduces a novel one-to-many coarse-to-fine alignment paradigm imitating the human cognitive process of identifying specific items within a large collection. Our method employs a Focused-view Fusion Network with a sophisticated cross-attention mechanism dynamically aligning and comparing features across multiple videos to capture finer nuances and contextual variations. Extensive experiments on six benchmark datasets confirm that TokenBinder substantially outperforms existing state-of-the-art methods. These results demonstrate its robustness and the effectiveness of its fine-grained alignment in bridging intra- and inter-modality information gaps in TVR tasks.

We propose Vision Token Turing Machines (ViTTM) an efficient low-latency memory-augmented Vision Transformer (ViT). Our approach builds on Neural Turing Machines (NTM) and Token Turing Machines (TTM) which were applied to NLP and sequential visual understanding tasks. ViTTMs are designed for non-sequential computer vision tasks such as image classification and segmentation. Our model creates two sets of tokens: process tokens and memory tokens; process tokens pass through encoder blocks and read-write from memory tokens at each encoder block in the network allowing them to store and retrieve information from memory. By ensuring that there are fewer process tokens we are able to reduce the inference time of the network while maintaining its accuracy. On ImageNet-1K the state-of-the-art ViT-B has median latency of 529.5ms and 81.0% accuracy while our ViTTM-B is 56% faster (234.1ms) with 2.4x fewer FLOPs with an accuracy of 82.9%. On ADE20K semantic segmentation ViT-B achieves 45.65mIoU at 13.8 frame-per-second (FPS) whereas our ViTTM-B model acheives a 45.17 mIoU with 26.8 FPS (+94%).

Recent research in Vision Language Navigation (VLN) has overlooked the development of agents' inquisitive abilities which allow them to ask clarifying questions when instructions are incomplete. This paper addresses how agents can recognize "when" they lack sufficient information without focusing on "what" is missing particularly in VLN tasks with vague instructions. Equipping agents with this ability enhances efficiency by reducing potential digressions and seeking timely assistance. The challenge in identifying such uncertain points is balancing between being overly cautious (high recall) and overly confident (high precision). We propose an attention-based instruction-vagueness estimation module that learns associations between instructions and the agent's trajectory. By leveraging instruction-to-path alignment information during training the module's vagueness estimation performance improves by around 52% in terms of precision-recall balance. In our ablative experiments we also demonstrate the effectiveness of incorporating this additional instruction-to-path attention network alongside the cross-modal attention networks within the navigator module. Our results show that the attention scores from the instruction-to-path attention network serve as better indicators for estimating vagueness.

Timber represents an increasingly valuable and versatile resource. However forestry operations such as harvesting handling and measuring logs still require substantial human labor in remote environments posing significant safety risks. Progressively automating these tasks has the potential of increasing their efficiency as well as safety but requires an accurate detection of individual logs as well as live trees and their context. Although initial approaches have been proposed for this challenging application domain specialized data and algorithms are still too scarce to develop robust solutions. To mitigate this gap we introduce the TimberVision dataset consisting of more than 2k annotated RGB images containing a total of 51k trunk components including cut and lateral surfaces thereby surpassing any existing dataset in this domain in terms of both quantity and detail by a large margin. Based on this data we conduct a series of ablation experiments for oriented object detection and instance segmentation and evaluate the influence of multiple scene parameters on model performance. We introduce a generic framework to fuse the components detected by our models for both tasks into unified trunk representations. Furthermore we automatically derive geometric properties and apply multi-object tracking to further enhance robustness. Our detection and tracking approach provides highly descriptive and accurate trunk representations solely from RGB image data even under challenging environmental conditions. Our solution is suitable for a wide range of application scenarios and can be readily combined with other sensor modalities.

Recent extended reality headsets and field robots have adopted covers to protect the front-facing cameras from environmental hazards and falls. The surface irregularities on the cover can lead to optical aberrations like blurring and non-parametric distortions. Novel view synthesis methods like NeRF and 3D Gaussian Splatting are ill-equipped to synthesize from sequences with optical aberrations. To address this challenge we introduce SynthCover to enable novel view synthesis through protective covers for downstream extended reality applications. SynthCover employs a Refractive Field that estimates the cover's geometry enabling precise analytical calculation of refracted rays. Experiments on synthetic and real-world scenes demonstrate our method's ability to accurately model scenes viewed through protective covers achieving a significant improvement in rendering quality compared to prior methods. We also show that the model can adjust well to various cover geometries with synthetic sequences captured with covers of different surface curvatures. To motivate further studies on this problem we provide the benchmarked dataset containing real and synthetic walkable scenes captured with protective cover optical aberrations.

In the past decade state-of-the-art deep learning models have shown impressive performance in many computer vision tasks by learning from large and diverse image datasets. Most of these datasets consist of web-scraped image collections. This approach however makes it very challenging to obtain desirable data such as multiple views of the same object 3D geometric information or camera parameters for a large-scale image dataset. In this paper we propose a 3D-scanned multi-view 2D image dataset of fine-grained category instances with accurate camera calibration parameters. We describe our bi-directional multi-camera and 3D scanning system and the data collection pipeline. Our target objects are relatively small highly-detailed fine-grained category instances such as insects. We present this dataset as a contribution to fine-grained visual categorization 3D representation learning and for use in other computer vision tasks. The final version of the FineView dataset is available at: https://github.com/byu-vision/fineview

Sequential DeepFake detection is an emerging task that predicts the manipulation sequence in order. Existing methods typically formulate it as an image-to-sequence problem employing conventional Transformer architectures. However these methods lack dedicated design and consequently result in limited performance. As such this paper describes a new Transformer design called TSOM by exploring three perspectives: Texture Shape and Order of Manipulations. Our method features four major improvements: we describe a new texture-aware branch that effectively captures subtle manipulation traces with a Diversiform Pixel Difference Attention module. Then we introduce a Multi-source Cross-attention module to seek deep correlations among spatial and sequential features enabling effective modeling of complex manipulation traces. To further enhance the cross-attention we describe a Shape-guided Gaussian mapping strategy providing initial priors of the manipulation shape. Finally observing that the subsequent manipulation in a sequence may influence traces left in the preceding one we intriguingly invert the prediction order from forward to backward leading to notable gains as expected. Extensive experimental results demonstrate that our method outperforms others by a large margin highlighting the superiority of our method.

This paper addresses the challenge of developing robust Face Anti-Spoofing (FAS) models for face recognition systems. Traditional FAS protocols are limited by a lack of diversity in subject identities and environmental conditions restricting generalization to real-world scenarios. Recent advancements in spoof image synthesis have mitigated data scarcity but still fail to capture the full range of facial attributes and environmental variability needed for effective domain generalization. To address this we propose a novel framework capable of generating diverse realistic facial images with text-guided control. We fine-tune Stable Diffusion to extract real facial features and specifically train LoRA layers to capture detailed spoof patterns. Additionally the text-guided control of attributes helps overcome the lack of diversity seen in previous methods. Extensive experiments demonstrate that our text-to-image-based synthetic data generation significantly enhances the robustness of FAS models establishing a new benchmark for domain-independent and reliable anti-spoofing systems.

Document comparison typically relies on optical character recognition (OCR) as its core technology. However OCR requires the selection of appropriate language models for each document and the performance of multilingual or hybrid models remains limited. To overcome these challenges we propose text change detection (TCD) using an image comparison model tailored for multilingual documents. Unlike OCR-based approaches our method employs word-level text image-to-image comparison to detect changes. Our model generates bidirectional change segmentation maps between the source and target documents. To enhance performance without requiring explicit text alignment or scaling preprocessing we employ correlations among multi-scale attention features. We also construct a benchmark dataset comprising actual printed and scanned word pairs in various languages to evaluate our model. We validate our approach using our benchmark dataset and public benchmarks Distorted Document Images and the LRDE Document Binarization Dataset. We compare our model against state-of-the-art semantic segmentation and change detection models as well as to conventional OCR-based models.

The conventional modus operandi for adapting pre-trained vision-language models (VLMs) during test-time involves tuning learnable prompts i.e. test-time prompt tuning. This paper introduces Test-Time Low-rank adaptation (TTL) as an alternative to prompt tuning for zero-shot generalization of large-scale VLMs. Taking inspiration from recent advancements in efficiently fine-tuning large language models TTL offers a test-time parameter-efficient adaptation approach that updates the attention weights of the transformer encoder by maximizing prediction confidence. The self-supervised confidence maximization objective is specified using a weighted entropy loss that enforces consistency among predictions of augmented samples. TTL introduces only a small amount of trainable parameters for low-rank adapters in the model space while keeping the prompts and backbone frozen. Extensive experiments on a variety of natural distribution and cross-domain tasks show that TTL can outperform other techniques for test-time optimization of VLMs in strict zero-shot settings. Specifically TTL outperforms test-time prompt tuning baselines with a significant improvement on average. Our code is available at https://github.com/Razaimam45/TTL-Test-Time-Low-Rank-Adaptation.

Test-time adaptation (TTA) of 3D point clouds is crucial for mitigating discrepancies between training and testing samples in real-world scenarios particularly when handling corrupted point clouds. LiDAR data for instance can be affected by sensor failures or environmental factors causing domain gaps. Adapting models to these distribution shifts online is crucial as training for every possible variation is impractical. Existing methods often focus on fine-tuning pre-trained models based on self-supervised learning or pseudo-labeling which can lead to forgetting valuable source domain knowledge over time and reduce generalization on future tests. In this paper we introduce a novel 3D test-time adaptation method termed 3DD-TTA which stands for 3D Denoising Diffusion Test-Time Adaptation. This method uses a diffusion strategy that adapts input point cloud samples to the source domain while keeping the source model parameters intact. The approach uses a Variational Autoencoder (VAE) to encode the corrupted point cloud into a shape latent and latent points. These latent points are corrupted with Gaussian noise and subjected to a denoising diffusion process. During this process both the shape latent and latent points are updated to preserve fidelity guiding the denoising toward generating consistent samples that align more closely with the source domain. We conduct extensive experiments on the ShapeNet dataset and investigate its generalizability on ModelNet40 and ScanObjectNN achieving state-of-the-art results. The code has been released at https://github.com/hamidreza-dastmalchi/3DD-TTA.