WACV 2025 Papers

This paper addresses the challenge of artwork colorization by proposing a benchmark for manga colorization using real black-and-white and colorized image pairs. Color images are widely recognized for their ability to capture attention and improve memory retention yet the manual process of colorization is labor-intensive. Deep learning methods for supervised image-to-image translation offer a promising solution relying on aligned pairs of black-and-white and color images for training. However these pairs are often generated synthetically introducing a domain gap that limits model performance. To address this we explore the use of real data proposing a method for creating such datasets. Our benchmarks reveal that models trained on real data significantly outperform those trained on synthetic pairs. Furthermore we present a pipeline for text removal and panel segmentation streamlining the comic colorization process. These contributions aim to enhance the generalization and applicability of deep learning models for artwork colorization.

The fusion of multiple modalities such as vision and language has led to significant progress in grounding and tracking tasks. However this success has not yet translated to aerial single-object tracking (SOT) due to the lack of text annotations in existing aerial SOT datasets. To overcome this limitation we provide text annotations for five existing aerial datasets designed to support and promote multi-modal research in the aerial tracking domain. Furthermore to address challenges such as small object dimensions similar-looking objects and target size fluctuations we introduce a third input modality: click (or point prompt). To offer a user-friendly and interactive alternative to precise bounding box annotations we seamlessly integrate click and language information in the model's input. This enables approximate target specification with reduced effort and time. We introduce CLaVi a novel multimodal framework that redefines input interaction by incorporating multiple modalities. This integration improves target localization and tracking efficiency providing a significant advancement in the way input is provided to the model. Furthermore we conduct experiments on the five datasets to provide AerTrack-460 benchmark to validate the effectiveness of our approach. AerTrack-460 benchmark shows competitive performance and in some cases outperforms previous language-based grounding and tracking techniques setting a strong baseline for future research. Code and data will be made available soon.

Robust classification methods predominantly concentrate on algorithms that address a specific threat model resulting in ineffective defenses against other threat models. Real-world applications are exposed to this vulnerability as malicious attackers might exploit alternative threat models. In this work we propose a novel test-time threat model agnostic algorithm that enhances Adversarial-Trained (AT) models. Our method operates through COnditional image transformation and DIstance-based Prediction (CODIP) and includes two main steps: First we transform the input image into each dataset class where the input image might be either clean or attacked. Next we make a prediction based on the shortest transformed distance. The conditional transformation utilizes the perceptually aligned gradients property possessed by AT models and as a result eliminates the need for additional models or additional training. Moreover it allows users to choose the desired balance between clean and robust accuracy without training. The proposed method achieves state-of-the-art results demonstrated through extensive experiments on various models AT methods datasets and attack types. Notably applying CODIP leads to substantial robust accuracy improvement of up to +23% +20% +26% and +22% on CIFAR10 CIFAR100 ImageNet and Flowers datasets respectively.

Scene sketch semantic segmentation is a crucial task for various applications including sketch-to-image retrieval and scene understanding. Existing sketch segmentation methods treat sketches as bitmap images leading to the loss of temporal order among strokes due to the shift from vector to image format. Moreover these methods struggle to segment objects from categories absent in the training data. In this paper we propose a Class-Agnostic Visio-Temporal Network (CAVT) for scene sketch semantic segmentation. CAVT employs a class-agnostic object detector to detect individual objects in a scene and groups the strokes of instances through its post-processing module. This is the first approach that performs segmentation at both the instance and stroke levels within scene sketches. Furthermore there is a lack of free-hand scene sketch datasets with both instance and stroke-level class annotations. To fill this gap we collected the largest Free-hand Instance- and Stroke-level Scene Sketch Dataset (FrISS) that contains 1K scene sketches and covers 403 object classes with dense annotations. Extensive experiments on FrISS and other datasets demonstrate the superior performance of our method over state-of-the-art scene sketch segmentation models. Our code and dataset can be accessed from https://github.com/aleynakutuk6/CAVT.

In computer vision image clarity is crucial particularly when challenges like water droplets on camera lenses can significantly impair the accurate analysis of visual data. While existing methods mainly focus on small droplets the impact of larger droplets has been largely overlooked. This paper introduces a novel approach that models water droplets using randomly generated points and Bezier curves to simulate their shape on the lens. Based on this geometric framework we developed a classifier to distinguish between two visual scenarios within the droplet. For larger droplets we use a heuristic method to simulate various lens blockages. We evaluate this simulation framework using a real stereo dataset from [19] with clear and soiled images. Our method evaluated using Mean Squared Error (MSE) and Structural Similarity Index Measure (SSIM) demonstrates superior performance in MSE while also achieving competitive results in SSIM compared to existing techniques for generating realistic water droplets. Additionally we applied this technique for data augmentation in object detection tasks using the YOLOv7 [25] model. The results show improved robustness especially in challenging conditions where large droplets obstruct the lens.

Colorization is a well-explored problem in the domains of image and video processing. However extending colorization to 3D scenes presents significant challenges. Recent Neural Radiance Field (NeRF) and Gaussian-Splatting(3DGS) methods enable high-quality novel-view synthesis for multi-view images. However the question arises: How can we colorize these 3D representations? This work presents a method for synthesizing colorized novel views from input grayscale multi-view images. Using image or video colorization methods to colorize novel views from these 3D representations naively will yield output with severe inconsistencies. We introduce a novel method to use powerful image colorization models for colorizing 3D representations. We propose a distillation-based method that transfers color from these networks trained on natural images to the target 3D representation. Notably this strategy does not add any additional weights or computational overhead to the original representation during inference. Extensive experiments demonstrate that our method produces high-quality colorized views for indoor and outdoor scenes showcasing significant cross-view consistency advantages over baseline approaches. Our method is agnostic to the underlying 3D representation and easily generalizable to NeRF and 3DGS methods. Further we validate the efficacy of our approach in several diverse applications: 1.) Infra-Red (IR) multi-view images and 2.) Legacy grayscale multi-view image sequences. Project Webpage: https://val.cds.iisc.ac.in/chroma-distill.github.io/

Diffusion models have demonstrated extensive capabilities for generative modelling in both conditional and conditional image synthesis tasks. The reverse sampling has been the center of interest to improve the overall image quality without retraining the model from scratch. In this work we propose a plug-and-play module by utilizing the characteristic function of the distributions to minimize sampling drift. We experiment with existing diffusion solvers with our module in-place during denoising step to provide additional performance gain in image synthesis linear inverse problem tasks and text-conditioned image synthesis. Moreover We theoretically establish the effectiveness of the method in terms of improved Frechet Inception Distance (FID) and second order Tweedie moment for reduced trajectory deviation.

In this paper we introduce the Channel Propagation method which aims to increase the channels of the Vision Transformer systematically. Skip connections are commonly acknowledged as a propagation approach that improves the stability of the performance in Vision Transformers. Nevertheless it is important to note that these skip connections may give rise to the problem of over-smoothing wherein similar features are represented in multiple layers. To tackle this matter our proposed approach for Channel Propagation in Vision Transformers retains the present signal information while concurrently propagating location-specific signals in a newly introduced channel dimension. On the other hand the proposed Channel Propagation method effectively maintains the integrity of identity representation while simultaneously including patch-wise location-specific supervision by introducing a new channel dimension. The inclusion of this approach in Vision Transformers mitigates the issue of over-smoothing while also improving the performance of visual recognition tasks. In our experiments we confirm that the proposed method is effective for various visual recognition tasks. Specifically our method demonstrates enhanced performance when implemented on Vision Transformer models; the classification accuracy is increased considerably for plain and hierarchical architectures on the ImageNet dataset.

Facial landmark detection is a fundamental problem in computer vision for many downstream applications. This paper introduces a new facial landmark detector based on vision transformers which consists of two unique designs: Dual Vision Transformer (D-ViT) and Long Skip Connections (LSC). Based on the observation that the channel dimension of feature maps essentially represents the linear bases of the heatmap space we propose learning the interconnections between these linear bases to model the inherent geometric relations among landmarks via Channel-split ViT. We integrate such channel-split ViT into the standard vision transformer (i.e. spatial-split ViT) forming our Dual Vision Transformer to constitute the prediction blocks. We also suggest using long skip connections to deliver low-level image features to all prediction blocks thereby preventing useful information from being discarded by intermediate supervision. Extensive experiments are conducted to evaluate the performance of our proposal on the widely used benchmarks i.e. WFLW COFW and 300W demonstrating that our model outperforms the previous SOTAs across all three benchmarks.

The SYNTAX score has become a widely used measure of coronary disease severity crucial in selecting the optimal mode of the revascularization procedure. This paper introduces a new medical regression and classification problem -- automatically estimating SYNTAX score from coronary angiography. Our study presents a comprehensive CardioSYNTAX dataset of 3018 patients for the SYNTAX score estimation and coronary dominance classification. The dataset features a balanced distribution of individuals with zero and non-zero scores. This dataset includes a first-of-its-kind complete coronary angiography samples captured through a multi-view X-ray video allowing one to observe coronary arteries from multiple perspectives. Furthermore we present a novel fully automatic end-to-end method for estimating the SYNTAX. For such a difficult task we have achieved a solid coefficient of determination R2 of 0.51 in score value prediction and 77.3% accuracy for zero score classification.

Visual recognition in a low-data regime is challenging and often prone to overfitting. To mitigate this issue several data augmentation strategies have been proposed. However standard transformations e.g. rotation cropping and flipping provide limited semantic variations. To this end we propose Cap2Aug an image-to-image diffusion model-based data augmentation strategy using image captions to condition the image synthesis step. We generate a caption for an image and use this caption as an additional input for an image-to-image diffusion model. This increases the semantic diversity of the augmented images due to caption conditioning compared to the usual data augmentation techniques. We show that Cap2Aug is particularly effective where only a few samples are available for an object class. However naively generating the synthetic images is not adequate due to the domain gap between real and synthetic images. Thus we employ a maximum mean discrepancy loss to align the synthetic images to the real images to minimize the domain gap. We evaluate our method on few-shot classification and image classification with long-tail class distribution tasks. Cap2Aug achieves state-of-the-art performance on both tasks while evaluated on eleven benchmarks. Code: https://github.com/aniket004/Cap_2_Aug.git

Spread of fake news using out-of-context images and captions has become widespread in this era of information overload. Since fake news can belong to different domains like politics sports etc. with their unique characteristics inference on a test image-caption pair is contingent on how well the model has been trained on similar data. Since training individual models for each domain is not practical we propose a novel framework termed DPOD (Domain-specific Prompt tuning using Out-of-Domain data) which can exploit out-of-domain data during training to improve fake news detection of all desired domains simultaneously. First to compute generalizable features we modify the Vision-Language Model CLIP to extract features that help to align the representations of the images and corresponding captions of both the in-domain and out-of-domain data in a label-aware manner. Further we propose a domain-specific prompt learning technique which leverages training samples of all the available domains based on the extent they can be useful to the desired domain. Extensive experiments on the large-scale NewsCLIPpings and VERITE benchmarks demonstrate that DPOD achieves state of-the-art performance for this challenging task. Code: https://github.com/scviab/DPOD.

Large Language Models (LLMs) with in-context learning (ICL) ability can quickly adapt to a specific context given a few demonstrations (demos). Recently Multimodal Large Language Models (MLLMs) built upon LLMs have also shown multimodal ICL ability i.e. responding to queries given a few multimodal demos including images queries and answers. While ICL has been extensively studied on LLMs its research on MLLMs remains limited. One essential question is whether these MLLMs can truly conduct multimodal ICL or if only the textual modality is necessary. We investigate this question by examining two primary factors that influence ICL: 1) Demo content i.e. understanding the influences of demo content in different modalities. 2) Demo selection strategy i.e. how to select better multimodal demos for improved performance. Experiments revealed that multimodal ICL is predominantly driven by the textual content whereas the visual information in the demos has little influence. Interestingly visual content is still necessary and useful for selecting demos to increase performance. Motivated by our analysis we propose a simple yet effective approach termed Mixed Modality In-Context Example Selection (MMICES) which considers both visual and language modalities when selecting demos. Extensive experiments are conducted to support our findings and verify the improvement brought by our method.

Publicly available satellite imagery such as Sentinel-2 often lacks the spatial resolution required for accurate analysis of remote sensing tasks including urban planning and disaster response. Current super-resolution techniques are typically trained on limited datasets leading to poor generalization across diverse geographic regions. In this work we propose a novel super-resolution framework that enhances generalization by incorporating geographic context through location embeddings. Our framework employs GANs and incorporates techniques from diffusion models to enhance image quality. Furthermore we address tiling artifacts by integrating information from neighboring images enabling the generation of seamless high-resolution outputs. We demonstrate the effectiveness of our method on the building segmentation task showing significant improvements over state-of-the-art methods and highlighting its potential for real-world applications.

Numerous adversarial attack methods have been developed to generate imperceptible image perturbations that cause erroneous predictions in state-of-the-art machine learning (ML) models particularly deep neural networks (DNNs). Despite extensive research on adversarial examples limited efforts have been made to explore the hidden characteristics carried by these perturbations. In this study we investigate the feasibility of deducing information about the victim model (VM)--specifically characteristics such as architecture type kernel size activation function and weight sparsity--from adversarial examples. We approach this problem as a supervised learning task where we aim to attribute categories of VM characteristics to individual adversarial examples. To facilitate this we have assembled a dataset of adversarial attacks spanning seven types generated from 135 victim models systematically varied across five architecture types three kernel size configurations three activation functions and three levels of weight sparsity. We demonstrate that a supervised model parsing network (MPN) can effectively extract concealed details of the VM from adversarial examples. We also validate the practicality of this approach by evaluating the effects of various factors on parsing performance such as different input formats and generalization to out-of-distribution cases. Furthermore we highlight the connection between model parsing and attack transferability by showing how the MPN can uncover VM attributes in transfer attacks.

Camouflaged object detection (COD) and camouflaged instance segmentation (CIS) aim to recognize and segment objects that are blended into their surroundings respectively. While several deep neural network models have been proposed to tackle those tasks augmentation methods for COD and CIS have not been thoroughly explored. Augmentation strategies can help improve models' performance by increasing the size and diversity of the training data and exposing the model to a wider range of variations in the data. Besides we aim to automatically learn transformations that help to reveal the underlying structure of camouflaged objects and allow the model to learn to identify better and segment camouflaged objects. To achieve this we propose a learnable augmentation method in the frequency domain for COD and CIS via the Fourier transform approach dubbed CamoFA. Our method leverages a conditional generative adversarial network and cross-attention mechanism to generate a reference image and an adaptive hybrid swapping with parameters to mix the low-frequency component of the reference image and the high-frequency component of the input image. This approach aims to make camouflaged objects more visible for detection and segmentation models. Without bells and whistles our proposed augmentation method boosts the performance of camouflaged object detectors and instance segmenters by large margins.

Safety-critical 3D scene understanding tasks necessitate not only accurate but also confident predictions from 3D perception models. This study introduces Calib3D a pioneering effort to benchmark and scrutinize the reliability of 3D scene understanding models from an uncertainty estimation viewpoint. We comprehensively evaluate 28 state-of-the-art models across 10 diverse 3D datasets uncovering insightful phenomena that cope with both the aleatoric and epistemic uncertainties in 3D scene understanding. We discover that despite achieving impressive levels of accuracy existing models frequently fail to provide reliable uncertainty estimates - a pitfall that critically undermines their applicability in safety-sensitive contexts. Through extensive analysis of key factors such as network capacity LiDAR representations rasterization resolutions and 3D data augmentation techniques we correlate these aspects directly with the model calibration efficacy. Furthermore we introduce DeptS a novel depth-aware scaling approach aimed at enhancing 3D model calibration. Extensive experiments across a wide range of configurations validate the superiority of our method. We hope this work could serve as a cornerstone for fostering reliable 3D scene understanding. Code and benchmark toolkit are publicly available.

Accurate in-cabin depth estimation is critical for advancing automotive safety and occupant comfort. However existing datasets for in-vehicle scene understanding tasks often fall short in providing sufficient information and scale needed to evaluate existing depth estimation methods. In this paper we present a novel benchmark tailored for monocular depth estimation in vehicle interiors containing both near-infrared (NIR) images and corresponding ground truth depth data. Featuring over 41000 frames captured across 36 distinct vehicles and 45 different passengers it offers an unprecedented level of variability for this application domain. Evaluation on our testbench of cutting-edge single-view depth models in different flavors including zero-shot affine-invariant depth estimation or indomain specialization reveals that current depth estimation approaches while promising still have a significant performance gap to overcome before achieving the reliability required for downstream safety-critical applications. In light of its diverse range and complex scenarios we believe this benchmark could serve as a common reference for further research concerning in-cabin monocular depth estimation.

Retinal fundus photography is significant in diagnosing and monitoring retinal diseases. However systemic imperfections and operator/patient-related factors can hinder the acquisition of high-quality retinal images. Previous efforts in retinal image enhancement primarily relied on GANs which are limited by the trade-off between training stability and output diversity. In contrast the Schrodinger Bridge (SB) offers a more stable solution by utilizing Optimal Transport (OT) theory to model a stochastic differential equation (SDE) between two arbitrary distributions. This allows SB to effectively transform low-quality retinal images into their high-quality counterparts. In this work we leverage the SB framework to propose an image-to-image translation pipeline for retinal image enhancement. Additionally previous methods often fail to capture fine structural details such as blood vessels. To address this we enhance our pipeline by introducing Dynamic Snake Convolution whose tortuous receptive field can better preserve tubular structures. We name the resulting retinal fundus image enhancement framework the Context-aware Unpaired Neural Schrodinger Bridge (CUNSB-RFIE). To the best of our knowledge this is the first endeavor to use the SB approach for retinal image enhancement. Experimental results on a large-scale dataset demonstrate the advantage of the proposed method compared to several state-of-the-art supervised and unsupervised methods in terms of image quality and performance on downstream tasks. The code is available at https://github.com/Retinal-Research/CUNSB-RFIE.