WACV 2025 Papers

With the rise of marine exploration underwater imaging has gained significant attention as a research topic. Underwater video enhancement has become crucial for real-time computer vision tasks in marine exploration. However most existing methods focus on enhancing individual frames and neglect video temporal dynamics leading to visually poor enhancements. Furthermore the lack of ground-truth references limits the use of abundant available underwater video data in many applications. To address these issues we propose a two-stage framework for enhancing underwater videos. The first stage uses a denoising diffusion probabilistic model to learn a generative prior from unlabeled data capturing robust and descriptive feature representations. In the second stage this prior is incorporated into a physics-based image formulation for spatial enhancement while also enforcing temporal consistency between video frames. Our method enables real-time and computationally-efficient processing of high-resolution underwater videos at lower resolutions and offers efficient enhancement in the presence of diverse water-types. Extensive experiments on four datasets show that our approach generalizes well and outperforms existing enhancement methods. Our code is available at github.com/suhas-srinath/undive.

3D Gaussian splatting (3DGS) offers the capability to achieve real-time high quality 3D scene rendering. However 3DGS assumes that the scene is in a clear medium environment and struggles to generate satisfactory representations in underwater scenes where light absorption and scattering are prevalent and moving objects are involved. To overcome these we introduce a novel Gaussian Splatting-based method UW-GS designed specifically for underwater applications. It introduces a color appearance that models distance-dependent color variation employs a new physics-based density control strategy to enhance clarity for distant objects and uses a binary motion mask to handle dynamic content. Optimized with a well-designed loss function supporting for scattering media and strengthened by pseudo-depth maps UW-GS outperforms existing methods with PSNR gains up to 1.26dB. To fully verify the effectiveness of the model we also developed a new underwater dataset S-UW with dynamic object masks. The code of UW-GS and S-UW will be available at https://github.com/WangHaoran16/UW-GS.

Transformer-based methods have shown great promise in underwater image enhancement (UIE) tasks due to their capability to model long-range dependencies which are vital for reconstructing clear images. While numerous effective attention mechanisms have been devised to handle the computational requirements of transformers they frequently incorporate redundant information and noisy interactions from irrelevant regions. Additionally the current methods focusing solely on the raw pixel space constrains the exploration of the underwater image frequency dynamics thus hindering the models from fully leveraging their potential for producing high-quality images. To address these challenges we propose USWformer an efficient UIE Sparse Wavelet Transformer Network (1.19 M parameters) to eliminate the redundant features in both the spatial and frequency domains. The USWformer consists of two fundamental components: a Sparse Wavelet Self-Attention (SWSA) block and a Multi-scale Wavelet Feed-Forward Network (MWFN). The SWSA block selectively preserves essential attention scores from the keys corresponding to each query adjusting the feature details. MWFN further diminishes the feature redundancy in the aggregated features thereby improving the enhancement of the underwater images. We assess the efficacy of our approach across benchmark datasets comprising synthetic and real-world underwater images showcasing its superiority via thorough ablation studies and comparative analyses.

Universal Cross-Domain Retrieval (UCDR) retrieves relevant images from unseen domains and classes without semantic labels ensuring robust generalization. Existing methods commonly employ prompt tuning with pre-trained vision-language models but are inherently limited by static prompts reducing adaptability. We propose UCDR-Adapter which enhances pre-trained models with adapters and dynamic prompt generation through a two-phase training strategy. First Source Adapter Learning integrates class semantics with domain-specific visual knowledge using a Learnable Textual Semantic Template and optimizes Class and Domain Prompts via momentum updates and dual loss functions for robust alignment. Second Target Prompt Generation creates dynamic prompts by attending to masked source prompts enabling seamless adaptation to unseen domains and classes. Unlike prior approaches UCDR-Adapter dynamically adapts to evolving data distributions enhancing both flexibility and generalization. During inference only the image branch and generated prompts are used eliminating reliance on textual inputs for highly efficient retrieval. Extensive benchmark experiments show that UCDR-Adapter consistently outperforms ProS in most cases and other state-of-the-art methods on UCDR U^cCDR and U^dCDR settings.

Localizing unusual activities in videos such as abnormal behaviors or traffic incidents holds practical significance. However pretrained foundation models struggle with localizing diverse unusual events likely because of their insufficient representation in the models' pretraining datasets. To explore foundation models' capability in localizing unusual activities we introduce UAL-Bench a comprehensive benchmark for unusual activity localization featuring three video datasets (UAG-OOPS UAG-SSBD and UAG-FunQA) and an instruction-tuning dataset (OOPS-UAG-Instruct) to improve model capabilities. We also introduce a new metric R@1 TD <= p as an auxiliary metric to reasonably consider detections as true positive if their starting and ending timestamps are within a threshold. On UAL-Bench we evaluate three approaches: Video-Language Models (Vid-LLMs) instruction-tuned Vid-LLMs and a novel integration of Vision-Language Models and Large Language Models (VLM-LLM). Our results show the VLM-LLM approach excels in localizing short-span unusual events and predicting their onset (start time) more accurately than Vid-LLMs. Our findings highlight the challenges posed by long-duration videos particularly in autism diagnosis scenarios and the need for further advancements in localization techniques. Our work not only provides a benchmark for unusual activity localization but also outlines the key challenges for existing foundation models suggesting future research directions on this important task.

Semantic segmentation has witnessed remarkable advancements with the adaptation of the Transformer architecture. Parallel to the strides made by the Transformer CNN-based U-Net has seen significant progress especially in high-resolution medical imaging and remote sensing. This dual success inspired us to merge both strengths leading to the inception of a U-Net-based vision transformer decoder tailored for efficient contextual encoding. Here we propose a novel transformer decoder U-MixFormer built upon the U-Net structure designed for efficient semantic segmentation. Our approach distinguishes itself from the previous transformer methods by leveraging lateral connections between the encoder and decoder stages as feature queries for the attention modules apart from the traditional reliance on skip connections. Moreover we innovatively mix hierarchical feature maps from various encoder and decoder stages to form a unified representation for keys and values giving rise to our unique mix-attention module. Our approach demonstrates state-of-the-art performance across various configurations. Extensive experiments show that U-MixFormer outperforms SegFormer FeedFormer and SegNeXt by a large margin. For example U-MixFormer-B0 surpasses SegFormer-B0 and FeedFormer-B0 with 3.8% and 2.0% higher mIoU and 27.3% and 21.8% less computation and outperforms SegNext with 3.3% higher mIoU with MSCAN-T encoder on ADE20K. Code available at https://github.com/julian-klitzing/u-mixformer.

In recent times contrastive learning based loss functions have become increasingly popular for visual self-supervised representation learning owing to their state-of-the-art (SOTA) performance. Most of the modern contrastive learning methods generalize only to one positive and multiple negatives per anchor in a batch. A recent state-of-the-art contrastive loss called supervised contrastive (SupCon) loss extends self-supervised contrastive learning to supervised setting by generalizing to multiple positives and negatives in a batch and improves upon the cross-entropy loss. In this paper we propose a novel contrastive loss function -- Tuned Contrastive Learning (TCL) loss that generalizes to multiple positives and negatives in a batch and offers parameters to tune and improve the gradient responses from hard positives and hard negatives. We provide theoretical analysis of our loss function's gradient response and show mathematically how it is better than that of SupCon loss. We empirically compare our loss function with SupCon loss and cross-entropy loss in supervised setting on multiple classification-task datasets to show its effectiveness. We also show the stability of our loss function to a range of hyper-parameter settings. Unlike SupCon loss which is only applied to supervised setting we show how to extend TCL to self-supervised setting and empirically compare it with various SOTA self-supervised learning methods. Hence we show that TCL loss achieves performance on par with SOTA methods in both supervised and self-supervised settings.

Due to privacy concerns obtaining large datasets is challenging in medical image analysis especially with 3D modalities like Computed Tomography (CT) and Magnetic Resonance Imaging (MRI). Existing generative models developed to address this issue often face limitations in output diversity and thus cannot accurately represent 3D medical images. We propose a tumor-generation model that utilizes radiomics features as generative conditions. Radiomics features are high-dimensional handcrafted semantic features that are biologically well-grounded and thus are good candidates for conditioning. Our model employs a GAN-based model to generate tumor masks and a diffusion-based approach to generate tumor texture conditioned on radiomics features. Our method allows the user to generate tumor images according to user-specified radiomics features such as size shape and texture at an arbitrary location. This enables the physicians to easily visualize tumor images to better understand tumors according to changing radiomics features. Our approach allows for the removal manipulation and repositioning of tumors generating various tumor types in different scenarios. The model has been tested on tumors in four different organs (kidney lung breast and brain) across CT and MRI. The synthesized images are shown to effectively aid in training for downstream tasks and their authenticity was also evaluated through expert evaluations. Our method has potential usage in treatment planning with diverse synthesized tumors. Our code is available at github.com/jongdory/TS-Radiomics.

Accurate estimation of plant skeletal structure (e.g. branching structure) from images is essential for smart agriculture and plant science. Unlike human skeletons with fixed topology plant skeleton estimation presents a unique challenge i.e. estimating arbitrary tree graphs from images. While recent graph generation methods successfully infer thin structures from images it is challenging to constrain the output graph strictly to a tree structure. To this problem we present TreeFormer a plant skeleton estimator via tree-constrained graph generation. Our approach combines learning-based graph generation with traditional graph algorithms to impose the constraints during the training loop. Specifically our method projects an unconstrained graph onto a minimum spanning tree (MST) during the training loop and incorporates this prior knowledge into the gradient descent optimization by suppressing unwanted feature values. Experiments show that our method accurately estimates target plant skeletal structures for multiple domains: Synthetic tree patterns real botanical roots and grapevine branches. Our implementations are available at https://github.com/huntorochi/TreeFormer/.

We present TabGuard a privacy-preserving framework for an end-to-end secure Table Structure Recognition. TabGuard masks all the contents of the table locally and utilizes the masked table image for structure recognition. Our method is simple yet effective for detecting table cells while preserving the inherent table alignment characteristics to reconstruct tables. Our approach benefits from inductive bias expressed through an approximated table grid which helps alleviate challenges in the detection of cells that are small or have extreme aspect ratios. Experimental results demonstrate that our solution not only establishes a new state-of-the-art on several benchmark datasets but also effectively addresses long-standing challenges associated with dense tables having complex layouts. We make our code publically available at https://github.com/sachinraja13/TabGuard.

We consider the problem of few-viewpoint 3D surface reconstruction using raw measurements from a lidar system. Lidar captures 3D scene geometry by emitting pulses of light to a target and recording the speed-of-light time delay of the reflected light. However conventional lidar systems do not output the raw captured waveforms of backscattered light; instead they preprocess these data into a 3D point cloud. Since this procedure typically does not accurately model the noise statistics of the system exploit spatial priors or incorporate information about downstream tasks it ultimately discards useful information that is encoded in raw measurements of backscattered light. Here we propose to leverage raw measurements captured with a single-photon lidar system from multiple viewpoints to optimize a neural surface representation of a scene. The measurements consist of time-resolved photon count histograms or transients which capture information about backscattered light at picosecond time scales. Additionally we develop new regularization strategies that improve robustness to photon noise enabling accurate surface reconstruction with as few as 10 photons per pixel. Our method outperforms other techniques for few-viewpoint 3D reconstruction based on depth maps point clouds or conventional lidar as demonstrated in simulation and with captured data.

Improving the generalization capabilities of general-purpose robotic manipulation in real world has long been a significant challenge. Existing approaches often rely on collecting large-scale robotic data which is costly and time-consuming. However due to insufficient diversity of data they typically suffer from limiting their capability in open-domain scenarios with new objects and diverse environments. In this paper we propose a novel paradigm that effectively leverages language-reasoning segmentation mask generated by internet-scale foundation models to condition robot manipulation tasks. By integrating the mask modality which incorporates semantic geometric and temporal correlation priors derived from vision foundation models into the end-to-end policy model our approach can effectively and robustly perceive object pose and enable sample-efficient generalization learning including new object instances semantic categories and unseen backgrounds. We first introduce a series of foundation models to ground natural language demands across multiple tasks. Secondly we develop a two-stream 2D policy model based on imitation learning which processes raw images and object masks to predict robot actions with a local-global perception manner. Extensive real-world experiments conducted on a Franka Emika robot and a low-cost dual-arm robot demonstrate the effectiveness of our proposed paradigm and policy. Demos can be found in link1 or link2 and our code will be released at https://github.com/MCG-NJU/TPM.

Unsupervised domain adaptation (UDA) in videos is a challenging task that remains not well explored compared to image-based UDA techniques. Although vision transformers (ViT) achieve state-of-the-art performance in many computer vision tasks their use in video UDA has been little explored. Our key idea is to use transformer layers as a feature encoder and incorporate spatial and temporal transferability relationships into the attention mechanism. A Transferable-guided Attention (TransferAttn) framework is then developed to exploit the capacity of the transformer to adapt cross-domain knowledge across different backbones. To improve the transferability of ViT we introduce a novel and effective module named Domain Transferable-guided Attention Block (DTAB). DTAB compels ViT to focus on the spatio-temporal transferability relationship among video frames by changing the self-attention mechanism to a transferability attention mechanism. Extensive experiments were conducted on UCF-HMDB Kinetics-Gameplay and Kinetics-NEC Drone datasets with different backbones like ResNet101 I3D and STAM to verify the effectiveness of TransferAttn compared with state-of-the-art approaches. Also we demonstrate that DTAB yields performance gains when applied to other state-of-the-art transformer-based UDA methods from both video and image domains. Our code is available at https://github.com/Andre-Sacilotti/transferattn-project-code.

In medical imaging inverse problems aim to infer high-fidelity images from incomplete noisy measurements minimizing expenses and risks to patients in clinical settings. Diffusion models have recently emerged as a promising solution to such practical challenges proving particularly useful for the training-free inference of images from partially acquired measurements in Magnetic Resonance Imaging (MRI) and Computed Tomography (CT). A central challenge however is how to guide an unconditional denoised estimate to conform to the measurement information. Existing methods often employ either deficient projection or inefficient posterior approximation leading to suboptimal performance. In this paper we propose Bi-level Guided Diffusion Models (BGDM) a zero-shot imaging framework that efficiently steers the image generation process through a bi-level guidance strategy. Specifically BGDM first approximates an inner-level conditional posterior mean to establish an initial measurement-consistent prediction and then solves an outer-level proximal optimization objective to reinforce the measurement consistency. Our experimental findings leveraging publicly available MRI and CT datasets indicate that BGDM is more effective and efficient compared to baseline methods consistently generating high-fidelity medical images and significantly reducing hallucinatory artifacts in cases of sparse measurements. Code https://github.com/hosseinaskari-cs/BGDM.

Despite remarkable achievements in video synthesis achieving granular control over complex dynamics such as nuanced movement among multiple interacting objects still presents a significant hurdle for dynamic world modeling compounded by the necessity to manage appearance and disappearance drastic scale changes and ensure consistency for instances across frames. These challenges hinder the development of video generation that can faithfully mimic real-world complexity limiting utility for applications requiring high-level realism and controllability including advanced scene simulation and training of perception systems. To address that we propose TrackDiffusion a novel video generation framework affording fine-grained trajectory-conditioned motion control via diffusion models which facilitates the precise manipulation of the object trajectories and interactions overcoming the prevalent limitation of scale and continuity disruptions. A pivotal component of TrackDiffusion is the instance enhancer which explicitly ensures inter-frame consistency of multiple objects a critical factor overlooked in the current literature. Moreover we demonstrate that generated video sequences by our TrackDiffusion can be used as training data for visual perception models. To the best of our knowledge this is the first work to apply video diffusion models with tracklet conditions and demonstrate that generated frames can be beneficial for improving the performance of object trackers.

Text driven diffusion models have shown remarkable capabilities in editing images. However when editing 3D scenes existing works mostly rely on training a NeRF for 3D editing. Recent NeRF editing methods leverages edit operations by deploying 2D diffusion models and project these edits into 3D space. They require strong positional priors alongside text prompt to identify the edit location. These methods are operational on small 3D scenes and are more generalized to particular scene. They require training for each specific edit and cannot be exploited in real-time edits. To address these limitations we propose a novel method FreeEdit to make edits in training free manner using mesh representations as a substitute for NeRF. Training-free methods are now a possibility because of the advances in foundation model's space. We leverage these models to bring a training-free alternative and introduce solutions for insertion replacement and deletion. We consider insertion replacement and deletion as basic blocks for performing intricate edits with certain combinations of these operations. Given a text prompt and a 3D scene our model is capable of identifying what object should be inserted/replaced or deleted and location where edit should be performed. We also introduce a novel algorithm as part of FreeEdit to find the optimal location on grounding object for placement. We evaluate our model by comparing it with baseline models on a wide range of scenes using quantitative and qualitative metrics and showcase the merits of our method with respect to others. Project page: https://vivekmadhavaram.github.io/FreeEdit_page/1.

3D anomaly detection and localization is of great significance for industrial inspection. Prior 3D anomaly detection and localization methods focus on the setting that the testing data share the same category as the training data which is normal. However in real-world applications the normal training data for the target 3D objects can be unavailable due to issues like data privacy or export control regulation. To tackle these challenges we identify a new task - zero-shot 3D anomaly detection and localization where the training and testing classes do not overlap. To this end we design 3DzAL a novel patch-level contrastive learning framework based on pseudo anomalies generated using the inductive bias from task-irrelevant 3D xyz data to learn more representative feature representations. Furthermore we train a normalcy classifier network to classify the normal patches and pseudo anomalies and utilize the classification result jointly with feature distance to design anomaly scores. Instead of directly using the patch point clouds we introduce adversarial perturbations to the input patch xyz data before feeding into the 3D normalcy classifier for the classification-based anomaly score. We show that 3DzAL outperforms the state-of-the-art anomaly detection and localization performance.

Recent efforts to understand intermediate representations in deep neural networks have commonly attempted to label individual neurons and combinations of neurons that make up linear directions in the latent space by examining extremal neuron activations and the highest direction projections. In this paper we show that this approach although yielding a good approximation for many purposes fails to capture valuable information about the behaviour of a representation. Neural network activations are generally dense and so a more complex but realistic scenario is that linear directions encode information at various levels of stimulation. We hypothesise that non-extremal level activations contain complex information worth investigating such as statistical associations and thus may be used to locate confounding human interpretable concepts. We explore the value of studying a range of neuron activations by taking the case of mid-level output neuron activations and demonstrate on a synthetic dataset how they can inform us about aspects of representations in the penultimate layer not evident through analysing maximal activations alone. We use our findings to develop a method to curate data from mid-range logit samples for retraining to mitigate spurious correlations or confounding concepts in the penultimate layer on real benchmark datasets. The success of our method exemplifies the utility of inspecting non-maximal activations to extract complex relationships learned by models.

Blind face restoration methods have shown remarkable performance particularly when trained on large-scale synthetic datasets with supervised learning. These datasets are often generated by simulating low-quality face images with a handcrafted image degradation pipeline. The models trained on such synthetic degradations however cannot deal with inputs of unseen degradations. In this paper we address this issue by using only a set of input images with unknown degradations and without ground truth targets to fine-tune a restoration model that learns to map them to clean and contextually consistent outputs. We utilize a pre-trained diffusion model as a generative prior through which we generate high quality images from the natural image distribution while maintaining the input image content through consistency constraints. These generated images are then used as pseudo targets to fine-tune a pre-trained restoration model. Unlike many recent approaches that employ diffusion models at test time we only do so during training and thus maintain an efficient inference-time performance. Extensive experiments show that the proposed approach can consistently improve the perceptual quality of pre-trained blind face restoration models while maintaining great consistency with the input contents. Our best model also achieves the state-of-the-art results on both synthetic and real-world datasets.