WACV 2025 论文列表

Despite recent significant advancements in Handwritten Document Recognition (HDR) the efficient and accurate recognition of text against complex backgrounds diverse handwriting styles and varying document layouts remains a practical challenge. Moreover this issue is seldom addressed in academic research particularly in scenarios with minimal annotated data available. In this paper we introduce the DocTTT framework to address these challenges. The key innovation of our approach is that it uses test-time training to adapt the model to each specific input during testing. We propose a novel Meta-Auxiliary learning approach that combines Meta-learning and self-supervised Masked Autoencoder(MAE). During testing we adapt the visual representation parameters using a self-supervised MAE loss. During training we learn the model parameters using a meta-learning framework so that the model parameters are learned to adapt to a new input effectively. Experimental results show that our proposed method significantly outperforms existing state-of-the-art approaches on benchmark datasets.

Document image dewarping is a crucial step in the digitization of physical documents as it aims to remove the distortions induced by challenging environment settings and document sheet deformations often encountered when using smartphone cameras for image capture. Recently deep learning-based methods were combined with knowledge about the expected document structure also known as a template at inference time to improve the dewarping results. Our contributions in this work are threefold: (1) we propose a novel document image dewarping approach that leverages the prior knowledge about the document structure effectively by detecting and matching lines from the warped and the template domain and (2) we introduce a novel evaluation metric called matched normalized character error rate (mnCER) to overcome the limitations of existing metrics in evaluating the dewarping process. (3) Finally we evaluate our approach on the Inv3DReal dataset and show that our approach outperforms the state-of-the-art methods in terms of visual and text-based metrics. Our approach improves upon the state-of-the-art methods by 32.6% in Local Distortion and 40.2% in mnCER. Our code and models are available at https://felixhertlein.github.io/doc-matcher.

Diabetic Retinopathy (DR) constitutes 5% of global blindness cases. While numerous deep learning approaches have sought to enhance traditional DR grading methods they often falter when confronted with new out-of-distribution data thereby impeding their widespread application. In this study we introduce a novel deep learning method for achieving domain generalization (DG) in DR grading and make the following contributions. First we propose a new way of generating image-to-image diagnostically relevant fundus augmentations conditioned on the grade of the original fundus image. These augmentations are tailored to emulate the types of shifts in DR datasets thus increase the model's robustness. Second we address the limitations of the standard classification loss in DG for DR fundus datasets by proposing a new DG-specific loss - domain alignment loss; which ensures that the feature vectors from all domains corresponding to the same class converge onto the same manifold for better domain generalization. Third we tackle the coupled problem of data imbalance across DR domains and classes by proposing to employ Focal loss which seamlessly integrates with our new alignment loss. Fourth due to inevitable observer variability in DR diagnosis that induces label noise we propose leveraging self-supervised pretraining. This approach ensures that our DG model remains robust against early susceptibility to label noise even when only a limited dataset of non-DR fundus images is available for pretraining. Our method demonstrates significant improvements over the strong Empirical Risk Minimization baseline and other recently proposed state-of-the-art DG methods for DR grading. Code is available at https://github.com/sharonchokuwa/dg-adr.

Text-to-Avatar generation has recently made significant strides due to advancements in diffusion models. However most existing work remains constrained by limited diversity producing avatars with subtle differences in appearance for a given text prompt. We design DivAvatar a novel framework that generates diverse avatars empowering 3D creatives with a multitude of distinct and richly varied 3D avatars from a single text prompt. Different from most existing work that exploits scene-specific 3D representations such as NeRF DivAvatar finetunes a 3D generative model (i.e. EVA3D) allowing diverse avatar generation from simply noise sampling in inference time. DivAvatar has two key designs that help achieve generation diversity and visual quality. The first is a noise sampling technique during training phase which is critical in generating diverse appearances. The second is a semantic-aware zoom mechanism and a novel depth loss the former producing appearances of high textual fidelity by separate fine-tuning of specific body parts and the latter improving geometry quality greatly by smoothing the generated mesh in the features space. Extensive experiments show that DivAvatar is highly versatile in generating avatars of diverse appearances.

Domain adaptive semantic segmentation aims to transfer a model proficient in dense image classification from a source domain to a target domain. While various transfer methods have been explored in previous studies we argue that the modeling of categories within the model significantly affects its transferability. Building on the Gaussian Hypothesis which posits that each category in the feature space adheres to a multidimensional Gaussian distribution we propose a Class-Aware Variational Inference (CAVI) training method. This approach normalizes features of different categories into distinct multidimensional Gaussian distributions. To further learn domain-independent feature distributions we optimize the feature space using a Gaussian-based alignment strategy and incorporate Gaussian-based contrastive learning. Experimental results demonstrate that our method achieves state-of-the-art performance on the GTAV-to-Cityscapes and Synthia-to-Cityscapes benchmarks.

Video anomaly detection aims to develop automated models capable of identifying abnormal events in surveillance videos. The benchmark setup for this task is extremely challenging due to: i) the limited size of the training sets ii) weak supervision provided in terms of video-level labels and iii) intrinsic class imbalance induced by the scarcity of abnormal events. In this work we show that distilling knowledge from aggregated representations of multiple backbones into a single-backbone Student model achieves state-of-the-art performance. In particular we develop a bi-level distillation approach along with a novel disentangled cross-attention-based feature aggregation network. Our proposed approach DAKD (Distilling Aggregated Knowledge with Disentangled Attention) demonstrates superior performance compared to existing methods across multiple benchmark datasets. Notably we achieve significant improvements of 1.36% 0.78% and 7.02% on the UCF-Crime ShanghaiTech and XD-Violence datasets respectively.

Semantic correspondence the task of determining relationships between different parts of images underpins various applications including 3D reconstruction image-to-image translation object tracking and visual place recognition. Recent studies have begun to explore representations learned in large generative image models for semantic correspondence demonstrating promising results. Building on this progress current state-of-the-art methods rely on combining multiple large models resulting in high computational demands and reduced efficiency. In this work we address this challenge by proposing a more computationally efficient approach. We propose a novel knowledge distillation technique to overcome the problem of reduced efficiency. We show how to use two large vision foundation models and distill the capabilities of these complementary models into one smaller model that maintains high accuracy at reduced computational cost. Furthermore we demonstrate that by incorporating 3D data we are able to further improve performance without the need for human-annotated correspondences. Overall our empirical results demonstrate that our distilled model with 3D data augmentation achieves performance superior to current state-of-the-art methods while significantly reducing computational load and enhancing practicality for real-world applications such as semantic video correspondence. Our code and weights are publicly available on our project page.

Recent research has unveiled the development of customizing large-scale text-to-image models. These models bind a unique subject desired by a user to a specific token using the token to generate the subject in various contexts. However models from previous studies also bind elements unrelated to the subject's identity such as common backgrounds or poses in the reference images. This often leads to conflicts between the token and the context of text prompts during inference causing the model to fail to generate both the subject and the prompted context. In this work we approach this issue from a data scarcity perspective and propose to augment the number of reference images through a novel self-distillation framework. Our framework selects high-quality samples from images generated by a teacher model and uses them in student training. Our framework can be applied to any models that suffer from the conflicts and we demonstrate that our framework most effectively resolves the issue through comprehensive evaluations.

Weakly supervised video object segmentation (WSVOS) enables the identification of segmentation maps without requiring extensive annotations of object masks relying instead on coarse video labels indicating object presence. Weakly supervised semantic segmentation of objects in surgical videos is however more challenging due to a complex interaction of multiple transient objects such as surgical tools moving in and out of the surgical field. In this scenario state-of-the-art WSVOS methods struggle to learn accurate segmentation maps. We address this problem by introducing ViDeo Spatio-Temporal disentanglement Networks (VDST-Net) a framework to disentangle complex spatiotemporal object interactions using semi-decoupled knowledge distillation to predict high-quality class activation maps (CAMs). A teacher network is designed to help a temporal-reasoning student network resolve activation conflicts as the student leverages temporal dependencies when specifics about object location and timing in the video are not provided. We demonstrate the efficacy of our framework on a challenging surgical video dataset where objects are on average present in less than 60% of annotated frames and compare our method to state-of-the-art methods on surgical data and on a public dataset commonly used to benchmark WSVOS. Our method outperforms state-of-the-art techniques and generates accurate segmentation masks under video-level weak supervision. Our code is available at: https://github.com/PCASOlab/VDST-net.

Disentangled representation learning (DRL) aims to break down observed data into core intrinsic factors for a profound understanding of the data. In real-world scenarios manually defining and labeling these factors are non-trivial making unsupervised methods attractive. Recently there have been limited explorations of utilizing diffusion models (DMs) which are already mainstream in generative modeling for unsupervised DRL. They implement their own inductive bias to ensure that each latent unit input to the DM expresses only one distinct factor. In this context we design Dynamic Gaussian Anchoring to enforce attribute-separated latent units for more interpretable DRL. This unconventional inductive bias explicitly delineates the decision boundaries between attributes while also promoting the independence among latent units. Additionally we also propose Skip Dropout technique which easily modifies the denoising U-Net to be more DRL-friendly addressing its uncooperative nature with the disentangling feature extractor. Our methods which carefully consider the latent unit semantics and the distinct DM structure enhance the practicality of DM-based disentangled representations demonstrating state-of-the-art disentanglement performance on both synthetic and real data as well as advantages in downstream tasks.

Continual Test-Time Adaptation (CoTTA) task is proposed to tackle the challenges of constant domain shifts during testing. The goals are twofold: 1) to preserve the knowledge from the source domain without source data and 2) to effectively extract target knowledge using unlabeled target domain data. Existing works primarily focus on either source or target knowledge attempting to learn both in a mixed manner. We argue that this may harm the source knowledge preservation and target knowledge extraction. To this end this paper proposes a Source and Target knowledge Disentangle Transformer (SoTa-DiT) with the prompting mechanism. Specifically in a vision transformer (ViT) we incorporate source and target prompts supervised by two groups of deliberately designed loss functions to learn source and target knowledge separately. The source prompt focuses on anti-source-forgetting by extracting and preserving knowledge from the source model while the target prompt focuses on pro-target-extracting using target data contrastive learning. With comprehensive evaluations across various datasets using different ViT backbones we demonstrate that this dual-prompt architecture of SoTa-DiT is effective and that disentangling knowledge with the prompts benefits CoTTA. As a result SoTa-DiT significantly improves image classification accuracy under the CoTTA setting.

Weakly supervised video anomaly detection (WSVAD) often relies on Multiple Instance Learning (MIL). However selecting only the most discriminative segments for training limits the model's ability to comprehensively detect anomalous events particularly hard anomalies. To overcome this limitation we propose the Discriminative Score Suppression (DSS) module. This module suppresses the discriminative scores of the most prominent anomalies shifting the model's attention to less obvious but important hard anomalies. This approach guides the model to learn the critical features of hard anomalies enabling a more comprehensive detection of anomalous events. Additionally the Anomaly Score Refinement (ASR) module constructs a dissimilarity-based classifier by storing normal patterns as prototypes and integrates this with a neural network classifier. By combining the anomaly scores from both classifiers more accurate detection of true hard anomalies is achieved. A score-sensitive inner-bag loss function not only adjusts penalties based on anomaly scores but also ensures that the model avoids erroneous selections. Our method accurately detects various anomalies including challenging and multi-segment anomalies while minimizing false positives for normal events. Extensive experiments show that the proposed framework outperforms state-of-the-art methods on the UCF-Crime and XD-Violence datasets.

Generating realistic one-shot emotional talking head animation on arbitrary faces is a challenging problem as it requires realistic emotions head movements identity preservation and accurate lip sync. Existing emotional talking face generation methods either fail to retain the identity information of arbitrary subjects owing to the limited variability of existing emotional datasets or they fail to capture emotions accurately even if they preserve identity of arbitrary faces. Moreover most of the methods rely on additional input videos for driving poses and/or or expressions on the generated video. For practical applications it is infeasible to obtain driving videos of the same or different subject with variations in head pose expressions etc. In this paper we propose a novel approach for Audio-driven Emotional Talking Head generation from a single image with emotion-controllable head pose generation. Unlike existing methods our method does not require a driving video either for pose or emotions and can generate different emotions and diverse head pose variations from input speech and a single image of an arbitrary subject in neutral emotion. Our method overcomes the limitations of existing emotional audio-visual datasets by learning a disentangled approach for optical flow computation approach for pose and expression. Using our proposed method of independently computing pose-driven and expression-driven optical flow our image generation network can be pre-trained on a large dataset with greater pose variability but lacking emotion annotations. The expression flow generation branch is fine-tuned on a smaller emotional dataset to accurately capture different emotions not present in the original dataset while retaining the pose variability from the original dataset. We present extensive experiments to demonstrate the superiority of our proposed method in generating talking head animation with accurate emotions diverse head movements and generalization to arbitrary faces.

Actionable part perception for robotic object manipulation needs to perceive parts over open-world object categories within 3D space which is challenging as the appearance of the same part on different objects varies greatly. It is frequently observed that despite the huge intra-class difference in appearance the parts share common interactive functions over different objects i.e. common affordance. According to this observation we propose DisCo a novel technique that Discovers Common affordance information from powerful large models for guiding the actionable part perception across open-world objects. Specifically we first use a large language model to identify the object names that each part potentially belongs to and a text-to-image generative model to generate image examples for the queried objects constructing image-text paired data that indicate visual and semantic information of common affordance. Then our model encodes the common affordance information by learning to pair the object-part images with their text descriptions. Subsequently the 2D-pixel features are distilled into 3D space thus the 3D point features are enriched with not only the semantic information of open-set objects but also the common affordance information which is highly generalizable. Finally a segmentation head and a pose regression network are developed to predict more accurate results of part segmentation and pose estimation improving the success rate of robotic object manipulation. Extensive experiments show that our method outperforms existing methods on the part instance and semantic segmentation by significant margins of 4.8% mAP 5.4% AP50 and 3.9% mIoU on the unseen object categories.

Visual anagrams are images that change appearance upon transformation like flipping or rotation. With the advent of diffusion models generating such optical illusions can be achieved by averaging noise across multiple views during the reverse denoising process. However we observe two critical failure modes in this approach: (i) concept segregation where concepts in different views are independently generated which can not be considered a true anagram and (ii) concept domination where certain concepts overpower others. In this work we cast the visual anagram generation problem in a multi-task learning setting where different viewpoint prompts are analogous to different tasks and derive denoising trajectories that align well across tasks simultaneously. At the core of our designed framework are two newly introduced techniques where (i) an anti-segregation optimization strategy that promotes overlap in cross-attention maps between different concepts and (ii) a noise vector balancing method that adaptively adjusts the influence of different tasks. Additionally we observe that directly averaging noise predictions yields suboptimal performance because statistical properties may not be preserved prompting us to derive a noise variance rectification method. Extensive qualitative and quantitative experiments demonstrate our method's superior ability to generate visual anagrams spanning diverse concepts.

The Bird-Eye-View (BEV) is one of the most widely-used scene representations for visual perception in Autonomous Vehicles (AVs) due to its well suited compatibility to downstream tasks. For the enhanced safety of AVs modeling perception uncertainty in BEV is crucial. Recent diffusion-based methods offer a promising approach to uncertainty modeling for visual perception but fail to effectively detect small objects in the large coverage of the BEV. Such degradation of performance can be attributed primarily to the specific network architectures and the matching strategy used when training. Here we address this problem by combining the diffusion paradigm with current state-of-the-art 3D object detectors in BEV. We analyze the unique challenges of this approach which do not exist with deterministic detectors and present a simple technique based on object query interpolation that allows the model to learn positional dependencies even in the presence of the diffusion noise. Based on this we present a diffusion-based DETR model for object detection that bears similarities to particle methods. Abundant experimentation on the NuScenes dataset shows equal or better performance for our generative approach compared to deterministic state-of-the-art methods. The source code is at https://github.com/insait-institute/ParticleDETR.

Ensuring the quality and quantity of labeled training data has long been a challenge in training deep neural networks for discriminative tasks. One solution to this problem is to use a generative model to augment training data and learn a discriminative model with it. For image classification with the recent development of diffusion models it has become possible to generate a variety synthetic images and there are high expectations for their use as training data. However to obtain high-quality labeled synthetic images the hyperparameters and prompts often need to be manually tuned and the accuracy of the trained image classification model is highly dependent on them. To address this issue this paper proposes diffusion-based generative regularization a supervised discriminative learning framework that utilizes a diffusion-based image generation model as a regularizer to robustly learn discriminative representations without the need to synthesize images. Our experiments using vision transformers and stable diffusion models on ImageNet-1k demonstrate that the proposed framework improves classification accuracy on both in-distribution and distribution-shifted data.

We present a simple but effective training-free approach for text-driven image-to-image translation based on a pretrained text-to-image diffusion model. Our goal is to generate an image that aligns with the target task while preserving the structure and background of a source image. To this end we derive the representation guidance with a combination of two objectives: maximizing the similarity to the target prompt based on the CLIP score and minimizing the structural distance to the source latent variable. This guidance improves the fidelity of the generated target image to the given target prompt while maintaining the structure integrity of the source image. To incorporate the representation guidance component we optimize the target latent variable of diffusion model's reverse process with the guidance. Experimental results demonstrate that our method achieves outstanding image-to-image translation performance on various tasks when combined with the pretrained Stable Diffusion model.

Despite the remarkable progress in generative modelling current diffusion models lack a quantitative approach to assess image quality. To address this limitation we propose to estimate the pixel-wise aleatoric uncertainty during the sampling phase of diffusion models and utilise the uncertainty to improve the sample generation quality. The uncertainty is computed as the variance of the denoising scores with a perturbation scheme that is specifically designed for diffusion models. We then show that the aleatoric uncertainty estimates are related to the second-order derivative of the diffusion noise distribution. We evaluate our uncertainty estimation algorithm and the uncertainty-guided sampling on the ImageNet and CIFAR- 10 datasets. In our comparisons with the related work we demonstrate promising results in filtering out low quality samples. Furthermore we show that our guided approach leads to better sample generation in terms of FID scores