WACV 2025 Papers

Camera relocalization methods range from dense image alignment to direct camera pose regression from a query image. Among these sparse feature matching stands out as an efficient versatile and generally lightweight approach with numerous applications. However feature-based methods often struggle with significant viewpoint and appearance changes leading to matching failures and inaccurate pose estimates. To overcome these limitations we propose a novel approach that leverages a globally sparse but locally dense 3D representation of 2D features. By tracking and triangulating landmarks over a sequence of frames we construct a sparse voxel map optimized to render image patch descriptors observed during tracking. Given an initial pose estimate we first synthesize descriptors from the voxels using volumetric rendering and then perform feature matching to estimate the camera pose. This method enables the generation of descriptors for unseen views enhancing robustness to viewpoint changes. We evaluate our method on the 7-Scenes and Cambridge Landmarks datasets. Our results show that our approach significantly outperforms existing state-of-the-art feature representation techniques in indoor environments achieving up to a 39% improvement in median translation error. Additionally our approach yields comparable results to other methods for outdoor scenes but with lower computational and memory footprints.

While the mainstream research in anomaly detection has mainly followed the one-class classification practical industrial environments often incur noisy training data due to annotation errors or lack of labels for new or refurbished products. To address these issues we propose a novel learning-based approach for fully unsupervised anomaly detection with unlabeled and potentially contaminated training data. Our method is motivated by two observations that i) the pairwise feature distances between the normal samples are on average likely to be smaller than those between the anomaly samples or heterogeneous samples and ii) pairs of features mutually closest to each other are likely to be homogeneous pairs which hold if the normal data has smaller variance than the anomaly data. Building on the first observation that nearest-neighbor distances can distinguish between confident normal samples and anomalies we propose a pseudo-labeling strategy using an iteratively reconstructed memory bank (IRMB). The second observation is utilized as a new loss function to promote class-homogeneity between mutually closest pairs thereby reducing the ill-posedness of the task. Experimental results on two public industrial anomaly benchmarks and semantic anomaly examples validate the effectiveness of FUN-AD across different scenarios and anomaly-to-normal ratios. Our code is available at https://github.com/HY-Vision-Lab/FUNAD.

Talking face generation has gained immense popularity in the computer vision community with various applications including AR VR teleconferencing digital assistants and avatars. Traditional methods are mainly audio-driven which have to deal with the inevitable resource-intensive nature of audio storage and processing. To address such a challenge we propose FT2TF - First-Person Statement Text-To-Talking Face Generation a novel one-stage end-to-end pipeline for talking face generation driven by first-person statement text. Different from previous work our model only leverages visual and textual information without any other sources (e.g. audio/landmark/pose) during inference. Extensive experiments are conducted on LRS2 and LRS3 datasets and results on multi-dimensional evaluation metrics are reported. Both quantitative and qualitative results showcase that FT2TF outperforms existing relevant methods and reaches the state-of-the-art. This achievement highlights our model's capability to bridge first-person statements and dynamic face generation providing insightful guidance for future work.

Due to the widespread impact of social media efficiently detecting out-of-context misinformation where a real image is paired with a fake caption has become imperative. Towards this goal we propose a novel framework FRAUD-Net which incorporates several unexplored aspects of this task in the model formulation. Keeping in mind that the image and textual evidences collected using the input image-text pair by web search play a crucial role in this task we build a generalized model which can handle missing or variable number of evidences as expected in real-world scenarios. We achieve this by effectively utilizing the common semantic latent space of Visual Language Models and transformer attention blocks. In addition we observe that explicit domain information not only allows the model to learn better but also dynamically adjusts to the diverse domains (like politics healthcare etc.) during testing. We also propose a mechanism to handle noisy training data by analyzing prediction consistency which helps in model training. Extensive experiments on the large-scale NewsClippings and Verite benchmark datasets showcase the effectiveness of the proposed framework compared to the state-of-the-art techniques for this challenging task.

As working with large datasets becomes standard the task of accurately retrieving images containing objects of interest by an open set textual query gains practical importance. The current leading approach utilizes a pre-trained CLIP model without any adaptation to the target domain balancing accuracy and efficiency through additional post-processing. In this work we propose FOR: Finetuning for Object-centric Open-vocabulary Image Retrieval which allows finetuning on a target dataset using closed-set labels while keeping the visual-language association crucial for open vocabulary retrieval. FOR is based on two design elements: a specialized decoder variant of the CLIP head customized for the intended task and its coupling within a multi-objective training framework. Together these design choices result in a significant increase in accuracy show-casing improvements of up to 8 mAP@50 points over SoTA across three datasets. Additionally we demonstrate that FOR is also effective in a semi-supervised setting achieving impressive results even when only a small portion of the dataset is labeled.

Medical datasets are often large and contain sensitive information presenting significant challenges for data sharing and storage. To address these issues this paper introduces a novel method called Fused Matching Distillation (FMD) which combines multiple dataset distillation techniques to achieve both data compression and enhanced privacy. FMD synthesizes representative subsets that capture the essential information from the original dataset while anonymizing sensitive details during the distillation process. The proposed approach integrates two complementary methods: parameter matching a technique that aligns the training trajectories of a teacher network trained on real data with those of a student network trained on synthetic data and feature distribution matching which ensures that the synthetic dataset closely approximates the feature distribution of the original data. By fusing these techniques FMD maximizes the information density within each pixel of the distilled dataset achieving a balance between compression and performance. Experimental evaluations on medical datasets including COVID chest X-ray and Pancreas cancer CT demonstrate that FMD achieves superior accuracy and privacy compared to existing methods. Furthermore the proposed method is evaluated using metrics for both model performance and anonymity showing that FMD not only maintains high diagnostic accuracy but also effectively anonymizes the data. This makes FMD a promising tool for secure and efficient medical data sharing.

Face video restoration (FVR) is a challenging but important problem where one seeks to recover a perceptually realistic face videos from a low-quality input. While diffusion probabilistic models (DPMs) have been shown to achieve remarkable performance for face image restoration they often fail to preserve temporally coherent high-quality videos compromising the fidelity of reconstructed faces. We present a new conditional diffusion framework called FLAIR for FVR. FLAIR ensures improved temporal alignments across frames in a computationally efficient fashion by converting a traditional image DPM into a video DPM. The proposed conversion uses a recurrent video refinement layer and a temporal self-attention at different scales. FLAIR also uses a conditional iterative refinement process to balance the perceptual and distortion quality during inference. This process consists of two key components: a data-consistency module that analytically ensures that the generated video precisely matches its degraded observation and a coarse-to-fine image enhancement module specifically for facial regions. Our extensive experiments show superiority of FLAIR over the current state-of-the-art (SOTA) for video super-resolution deblurring JPEG restoration and space-time frame interpolation on two high-quality face video datasets.

Domain Generalization techniques aim to enhance model robustness by simulating novel data distributions during training typically through various augmentation or stylization strategies. However these methods frequently suffer from limited control over the diversity of generated images and lack assurance that these images span distinct distributions. To address these challenges we propose FDS Feedback-guided Domain Synthesis a novel strategy that employs diffusion models to synthesize novel pseudo-domains by training a single model on all source domains and performing domain mixing based on learned features. By incorporating images that pose classification challenges to models trained on original samples alongside the original dataset we ensure the generation of a training set that spans a broad distribution spectrum. Our comprehensive evaluations demonstrate that this methodology sets new benchmarks in domain generalization performance across a range of challenging datasets effectively managing diverse types of domain shifts. The code can be found at https://github.com/Mehrdad-Noori/FDS.

Scene Text Editing (STE) is a challenging research problem that primarily aims towards modifying existing texts in an image while preserving the background and the font style of the original text. Despite its utility in numerous real-world applications existing style-transfer-based approaches have shown sub-par editing performance due to (1) complex image backgrounds (2) diverse font attributes and (3) varying word lengths within the text. To address such limitations in this paper we propose a novel font-agnostic scene text editing and rendering framework named FASTER for simultaneously generating text in arbitrary styles and locations while preserving a natural and realistic appearance and structure. A combined fusion of target mask generation and style transfer units with a cascaded self-attention mechanism has been proposed to focus on multi-level text region edits to handle varying word lengths. Extensive evaluation on a real-world database with further subjective human evaluation study indicates the superiority of FASTER in both scene text editing and rendering tasks in terms of model performance and efficiency. The code and pre-trained models have been released in our Github repo.

Face recognition systems are widely used for identity verification in various fields. However recent studies have highlighted bias issues related to demographic and non-demographic attributes such as accessories haircolor ethnicity or gender. These biases lead to higher error rates for specific attribute subgroups. This is especially problematic in critical areas like forensics where these systems are deployed. Addressing this issue requires a solution that reduces bias without compromising accuracy. Existing methods focus on learning less biased face representations but they are often difficult to integrate into current systems or negatively impact overall recognition performance. This work introduces FALCON (Fair Adaptation Through Local Optimal Normalization) an effective method to increase fairness in face recognition systems. FALCON operates in an unsupervised manner addressing bias without requiring demographic labels and can be easily integrated as a post-processing step. It treats individuals with similar traits similarly reducing bias in face recognition by processing each image individually. The proposed method is rigorously tested across various face recognition models and datasets and compared with four other fairness post-processing methods. Results show that FALCON significantly enhances both fairness and accuracy. Unlike other methods it allows seamlessly adjusting the fairness-accuracy trade-off while effectively addressing bias.

Deep neural networks are susceptible to adversarial attacks and common corruptions which undermine their robustness. In order to enhance model resilience against such challenges Adversarial Training (AT) has emerged as a prominent solution. Nevertheless adversarial robustness is often attained at the expense of model fairness during AT i.e. disparity in class-wise robustness of the model. While distinctive classes become more robust towards such adversaries hard to detect classes suffer. Recently research has focused on improving model fairness specifically for perturbed images overlooking the accuracy of the most likely non-perturbed data. Additionally despite their robustness against the adversaries encountered during model training state-of-the-art adversarial trained models have difficulty maintaining robustness and fairness when confronted with diverse adversarial threats or common corruptions. In this work we address the above concerns by introducing a novel approach called Fair Targeted Adversarial Training (FAIR-TAT). We show that using targeted adversarial attacks for adversarial training (instead of untargeted attacks) can allow for more favorable trade-offs with respect to adversarial fairness. Empirical results validate the efficacy of our approach.

Recent progress in image deblurring techniques focuses mainly on operating in both frequency and spatial domains using the Fourier transform (FT) properties. However their performance is limited due to the dependency of FT on stationary signals and its lack of capability to extract spatial-frequency properties. In this paper we propose a novel approach based on the Fractional Fourier Transform (FRFT) a unified spatial-frequency representation leveraging both spatial and frequency components simultaneously making it ideal for processing non-stationary signals like images. Specifically we introduce a Fractional Fourier Transformer (F2former) where we combine the classical fractional Fourier based Wiener deconvolution (F2WD) as well as a multi-branch encoder-decoder transformer based on a new fractional frequency aware transformer block (F2TB). We design F2TB consisting of a fractional frequency aware self-attention (F2SA) to estimate element-wise product attention based on important frequency components and a novel feed-forward network based on frequency division multiplexing (FM-FFN) to refine high and low frequency features separately for efficient latent clear image restoration. Experimental results for the cases of both motion deblurring as well as defocus deblurring show that the performance of our proposed method is superior to other state-of-the-art (SOTA) approaches.

Frozen Section (FS) technique is a rapid method taking only 15-30 minutes to prepare slides for pathologists' evaluation during surgery enabling immediate surgical decisions. However the FS process often introduces artifacts and distortions like folds and ice-crystal effects. In contrast these artifacts are absent in higher-quality formalin-fixed paraffin-embedded (FFPE) slides which take 2-3 days to prepare. Generative Adversarial Network (GAN)-based methods have been used to translate FS to FFPE images but they may leave morphological inaccuracies or introduce new artifacts reducing translation quality for clinical assessments. In this study we benchmark recent generative models focusing on GANs and Latent Diffusion Models (LDMs) to address these limitations. We introduce a novel approach combining LDMs with Histopathology Pre-Trained Embeddings to enhance FS image restoration. Our framework uses LDMs conditioned by both text and pre-trained embeddings to learn meaningful features of FS and FFPE images. Through diffusion and denoising processes our approach preserves essential diagnostic attributes like color staining and tissue morphology using DDIM Inversion with FS representation. Additionally it enhances the histological quality of translated images by predicting the targeted FFPE representation via an innovative embedding translation mechanism improving morphological details and reducing FS artifacts. As a result this work significantly improves classification performance with the Area Under the Curve rising from 81.99% to 94.64% and achieves the best results in user study. This work establishes a new benchmark for FS to FFPE image translation quality promising enhanced reliability and accuracy in histopathology FS image analysis. Our work is available at https://minhmanho.github.io/f2f_ldm/.

Continual learning (CL) has emerged as a critical area in machine learning enabling neural networks to learn from evolving data distributions while mitigating catastrophic forgetting. However recent research has identified the stability gap - a phenomenon where models initially lose performance on previously learned tasks before partially recovering during training. Such learning dynamics are contradictory to the intuitive understanding of stability in continual learning where one would expect the performance to degrade gradually instead of rapidly decreasing and then partially recovering later. To better understand and alleviate the stability gap we investigate it at different levels of the neural network architecture particularly focusing on the role of the classification head. We introduce the nearest-mean classifier (NMC) as a tool to attribute the influence of the backbone and the classification head on the stability gap. Our experiments demonstrate that NMC not only improves final performance but also significantly enhances training stability across various continual learning benchmarks including CIFAR100 ImageNet100 CUB-200 and FGVC Aircrafts. Moreover we find that NMC also reduces task-recency bias. Our analysis provides new insights into the stability gap and suggests that the primary contributor to this phenomenon is the linear head rather than the insufficient representation learning.

The vocabulary size in temporal action localization (TAL) is limited by the scarcity of large-scale annotated datasets. To overcome this recent works integrate vision-language models (VLMs) such as CLIP for open-vocabulary TAL (OV-TAL). However despite the success of VLMs trained on extensive datasets existing OV-TAL methods still rely on human-labeled TAL datasets of limited size to train action localizers limiting their generalizability. In this paper we explore the scalability of self-training with unlabeled YouTube videos for OV-TAL. Our approach consists of two stages: (1) a class-agnostic action localizer is trained on a human-labeled TAL dataset to generate pseudo-labels for unlabeled videos and (2) the large-scale pseudo-labeled dataset is then used to train the localizer. Extensive experiments demonstrate that leveraging web-scale videos in self-training significantly enhances the generalizability of an action localizer. Additionally we identify limitations in existing OV-TAL evaluation schemes and propose a new benchmark for thorough assessment. Finally we showcase the TAL performance of the large multimodal model Gemini-1.5 on our new benchmark. Code is released at https://github.com/HYUNJS/STOV-TAL.

Action detection aims to detect (recognize and localize) human actions spatially and temporally in videos. Existing approaches focus on the closed-set setting where an action detector is trained and tested on videos from a fixed set of action categories. However this constrained setting is not viable in an open world where test videos inevitably come beyond the trained action categories. In this paper we address the practical yet challenging Open-Vocabulary Action Detection (OVAD) problem. It aims to detect any action in test videos while training a model on a fixed set of action categories. To achieve such an open-vocabulary capability we propose a novel method OpenMixer that exploits the inherent semantics and localizability of large vision-language models (VLM) within the family of query-based detection transformers (DETR). Specifically the OpenMixer is developed by spatial and temporal OpenMixer blocks (S-OMB and T-OMB) and a dynamically fused alignment (DFA) module. The three components collectively enjoy the merits of strong generalization from pre-trained VLMs and end-to-end learning from DETR design. Moreover we established OVAD benchmarks under various settings and the experimental results show that the OpenMixer performs the best over baselines for detecting seen and unseen actions. We release the codes models and dataset splits at: https://github.com/Cogito2012/OpenMixer.

Detecting out-of-distribution (OOD) data is essential for safe deployment of deep neural networks (DNNs). This problem becomes particularly challenging in the presence of long-tailed in-distribution (ID) datasets often leading to high false positive rates (FPR) and low tail-class ID classification accuracy. In this paper we demonstrate that exploiting inter-sample relationships using a graph-based representation can significantly improve OOD detection in long-tailed recognition of vision datasets. To this end we use the feature space of a pre-trained model to initialize our graph structure. We account for the differences between the activation layer distribution of the pre-training vs. training data and actively introduce Gaussianization to alleviate any deviations from a standard normal distribution in the activation layers of the pre-trained model. We then refine this initial graph representation using graph convolutional networks (GCNs) to arrive at a feature space suitable for long-tailed OOD detection. This leads us to address the inferior performance observed in ID tail-classes within existing OOD detection methods. Experiments over three benchmarks CIFAR10-LT CIFAR100-LT and ImageNet-LT demonstrate that our method outperforms the state-of-the-art approaches by a large margin in terms of FPR and tail-class ID classification accuracy.

Nowadays many videos contain graphic elements such as logos subtitles and user interfaces. These overlayed elements exhibit discontinuous motions characterized by static or instantaneous motions that are neither spatially nor temporally coherent. As existing Video Frame Interpolation (VFI) methods rely on motion-compensation techniques they work best for videos with continuous motions but face limitations against videos with discontinuous motion. In this paper we propose a simple framework to enhance the robustness of existing VFI models against discontinuous motion. We first identify key properties that distinguish discontinuous from continuous motion. They are then leveraged by the Discontinuity map (D-map) estimator to explicitly guide the classification of continuous and discontinuous areas through a coherence mask and an additional supervisory signal. Our framework separately interpolates the predicted continuous and discontinuous regions to achieve state-of-the-art performance against synthetic discontinuous motions while also generalizing well to real-world discontinuous motions. Moreover our framework's `plug-and-play' design enables easy application to existing VFI models without the need for retraining and maintains strong performance on continuous motion.

We propose a novel benchmark for cross-view knowledge transfer of dense video captioning adapting models from web instructional videos with exocentric views to an egocentric view. While dense video captioning (predicting time segments and their captions) is primarily studied with exocentric videos (e.g. YouCook2) benchmarks with egocentric videos are restricted due to data scarcity. To overcome the limited video availability transferring knowledge from abundant exocentric web videos is demanded as a practical approach. However learning the correspondence between exocentric and egocentric views is difficult due to their dynamic view changes. The web videos contain shots showing either full-body or hand regions while the egocentric view is constantly shifting. This necessitates the in-depth study of cross-view transfer under complex view changes. To this end we first create a real-life egocentric dataset (EgoYC2) whose captions follow the definition of YouCook2 captions enabling transfer learning between these datasets with access to their ground-truth. To bridge the view gaps we propose a view-invariant learning method using adversarial training which consists of pre-training and fine-tuning stages. Our experiments confirm the effectiveness of overcoming the view change problem and knowledge transfer to egocentric views. Our benchmark pushes the study of cross-view transfer into a new task domain of dense video captioning and envisions methodologies that describe egocentric videos in natural language.