WACV 2025 Papers

Developing robust drone detection systems is often constrained by the limited availability of large-scale annotated training data and the high costs associated with real-world data collection. However leveraging synthetic data generated via game engine-based simulations provides a promising and cost-effective solution to overcome this issue. Therefore we present SynDroneVision a synthetic dataset specifically designed for RGB-based drone detection in surveillance applications. Featuring diverse backgrounds lighting conditions and drone models SynDroneVision offers a comprehensive training foundation for deep learning algorithms. To evaluate the dataset's effectiveness we perform a comparative analysis across a selection of recent YOLO detection models. Our findings demonstrate that SynDroneVision is a valuable resource for real-world data enrichment achieving notable enhancements in model performance and robustness while significantly reducing the time and costs of real-world data acquisition. SynDroneVision can be accessed at https://zenodo.org/records/13360116.

The use of deep learning techniques in railway environments faces significant obstacles especially for computer vision tasks. Such obstacles are mainly due to the inherent safety concerns required for installing the proper equipment on a train and the substantial effort required to precisely annotate large datasets especially for segmentation tasks. Public datasets of real-world images are quite scarce and suffer from severe limitations such as coarse manual annotation or narrow range of scenarios. In addition real-world datasets often do not contain scenes that represent critical situations. To address such limitations this paper introduces SynDRA a synthetic dataset of photo-realistic images generated using a railway simulator built on Unreal Engine 5. SynDRA offers precise pixel-level annotations across diverse scenarios thereby facilitating more effective testing and training of deep learning models for semantic segmentation tasks in railway settings. The advantages of the proposed dataset are validated through a series of experiments that highlight the potential of SynDRA to enhance the performance of deep learning models in scenarios where real-world annotated data is scarce. The dataset is publicly available at the following link: https://syndra.retis.santannapisa.it.

Self-supervised image denoising implies restoring the signal from a noisy image without access to the ground truth. State-of-the-art solutions for this task rely on predicting masked pixels with a fully-convolutional neural network. This most often requires multiple forward passes information about the noise model or intricate regularization functions. In this paper we propose a Swin Transformer-based Image Autoencoder (SwinIA) the first fully-transformer architecture for self-supervised denoising. The flexibility of the attention mechanism helps to fulfill the blind-spot property that convolutional counterparts normally approximate. SwinIA can be trained end-to-end with a simple mean squared error loss without masking and does not require any prior knowledge about clean data or noise distribution. Simple to use SwinIA establishes the state of the art on several common benchmarks.

Most deep-learning models for vision tasks rely on RGB images as their primary input layer assuming the model inherently discovers an optimal representation. In this work we challenge this assumption and show that image gradients offer a straightforward yet robust representation for multi-frame image restoration. We demonstrate that clusters naturally emerge within gradient patches indicating improved estimation of the underlying signal. We develop a Video Swin-Transformer model operating in the gradient domain facilitated by the implementation of two differentiable gradient modules. One module computes image gradients using convolutions with gradient filters while the other reconstructs an RGB image from its gradient representation using deconvolution in the frequency domain. Additionally we employ a composite training loss that measures the error both in the color domain and its gradient counterpart. Applied to a multi-frame image restoration task involving the removal of lighting shadows and occlusions our model consistently outperforms RGB-based counterparts without introducing additional parameters thanks to its gradient regularization. We further apply our framework to various restoration tasks discussing its advantages and limitations. Qualitative results highlight the model's improved generalization to real-world video scenarios demonstrating successful adaptation from synthetic image training to real video data deployment.

In person search we detect and rank matches to a query person image within a set of gallery scenes. Most person search models make use of a feature extraction backbone followed by separate heads for detection and re-identification. While pre-training methods for vision backbones are well-established pre-training additional modules for the person search task has not been previously examined. In this work we present the first framework for end-to-end person search pre-training. Our framework splits person search into object-centric and query-centric methodologies and we show that the query-centric framing is robust to label noise and trainable using only weakly-labeled person bounding boxes. Further we provide a novel model dubbed Swap Path Net (SPNet) which implements both query-centric and object-centric training objectives and can swap between the two while using the same weights. Using SPNet we show that query-centric pre-training followed by object-centric fine-tuning achieves state-of-the-art results on the standard PRW and CUHK-SYSU person search benchmarks with 96.4% mAP on CUHK-SYSU and 61.2% mAP on PRW. In addition we show that our method is more effective efficient and robust for person search pre-training than recent backbone-only pre-training alternatives.

Survival prediction is a crucial task associated with cancer diagnosis and treatment planning. This paper presents a novel approach to survival prediction by harnessing comprehensive information from CT and PET scans along with associated Genomic data. Current methods rely on either a single modality or the integration of multiple modalities for prediction without adequately addressing associations across patients or modalities. We aim to develop a robust predictive model for survival outcomes by integrating multi-modal imaging data with genetic information while accounting for associations across patients and modalities. We learn representations for each modality via a self-supervised module and harness the semantic similarities across the patients to ensure the embeddings are aligned closely. However optimizing solely for global relevance is inadequate as many pairs sharing similar high-level semantics such as tumor type are inadvertently pushed apart in the embedding space. To address this issue we use a cross-patient module (CPM) designed to harness inter-subject correspondences. The CPM module aims to bring together embeddings from patients with similar disease characteristics. Our experimental evaluation of the dataset of Non-Small Cell Lung Cancer (NSCLC) patients demonstrates the effectiveness of our approach in predicting survival outcomes outperforming state-of-the-art methods.

Nighttime scenes are hard to semantically perceive with learned models and annotate for humans. Thus realistic synthetic nighttime data become all the more important for learning robust semantic perception at night thanks to their accurate and cheap semantic annotations. However existing data-driven or hand-crafted techniques for generating nighttime images from daytime counterparts suffer from poor realism. The reason is the complex interaction of highly spatially varying nighttime illumination which differs drastically from its daytime counterpart with objects of spatially varying materials in the scene happening in 3D and being very hard to capture with such 2D approaches. The above 3D interaction and illumination shift have proven equally hard to model in the literature as opposed to other conditions such as fog or rain. Our method named Sun Off Lights On (SOLO) is the first to perform nighttime simulation on single images in a photorealistic fashion by operating in 3D. It first explicitly estimates the 3D geometry the materials and the locations of light sources of the scene from the input daytime image and relights the scene by probabilistically instantiating light sources in a way that accounts for their semantics and then running standard ray tracing. Not only is the visual quality and photorealism of our nighttime images superior to competing approaches including diffusion models but the former images are also proven more beneficial for semantic nighttime segmentation in day-to-night adaptation. Code and data are publicly available at https://github.com/ktzevel/SOLO.

Pre-trained Vision-language (VL) models such as CLIP have shown significant generalization ability to downstream tasks even with minimal fine-tuning. While prompt learning has emerged as an effective strategy to adapt pre-trained VL models for downstream tasks current approaches frequently encounter severe overfitting to specific downstream data distributions. This overfitting constrains the original behavior of the VL models to generalize to new domains or unseen classes posing a critical challenge in enhancing the adaptability and generalization of VL models. To address this limitation we propose Style-Pro a novel style-guided prompt learning framework that mitigates overfitting and preserves the zero-shot generalization capabilities of CLIP. Style-Pro employs learnable style bases to synthesize diverse distribution shifts guided by two specialized loss functions that ensure style diversity and content integrity. Then to minimize discrepancies between unseen domains and the source domain Style-Pro maps the unseen styles into the known style representation space as a weighted combination of style bases. Moreover to maintain consistency between the style-shifted prompted model and the original frozen CLIP Style-Pro introduces consistency constraints to preserve alignment in the learned embeddings minimizing deviation during adaptation to downstream tasks. Extensive experiments across 11 benchmark datasets demonstrate the effectiveness of Style-Pro consistently surpassing state-of-the-art methods in various settings including base-to-new generalization cross-dataset transfer and domain generalization.

Following the great progress in text-conditioned image generation there is a dire need for establishing clear comparison benchmarks. Unfortunately assessing performance of such models is highly subjective and notoriously difficult. Current automatic assessment of generated images quality and their alignment to text are approximate at best while human assessment is subjective poorly calibrated and not very well defined. To address these concerns we propose GenomeBench a new framework for assessing quality of text-to-image generative models. It consists of a prompt dataset richly annotated with semantic components based on a formalized grounding of language and images. On top of it we define a procedure to collect human assessment through a carefully guided question answering process. Finally these assessments are summarized into a novel score built around quality and alignment to text. We show the proposal achieves higher inter-annotator agreement with respect to the baseline human assessment and better correlation between quality and alignment compared to automatic assessment. Finally we use this framework to dissect the performance of recent text-to-image models providing insights on strength and weakness of each.

Given a source portrait the automatic human body reshaping task aims at editing it to an aesthetic body shape. As the technology has been widely used in media several methods have been proposed mainly focusing on generating optical flow to warp the body shape. However those previous works only consider the local transformation of different body parts (arms torso and legs) ignoring the global affinity and limiting the capacity to ensure consistency and quality across the entire body. In this paper we propose a novel Adaptive Affinity-Graph Network (AAGN) which extracts the global affinity between different body parts to enhance the quality of the generated optical flow. Specifically our AAGN primarily introduces the following designs: (1) we propose an Adaptive Affinity-Graph (AAG) Block that leverages the characteristic of a fully connected graph. AAG represents different body parts as nodes in an adaptive fully connected graph and captures all the affinities between nodes to obtain a global affinity map. The design could better improve the consistency between body parts. (2) Besides for high-frequency details are crucial for photo aesthetics a Body Shape Discriminator (BSD) is designed to extract information from both high-frequency and spatial domains. Particularly an SRM filter is utilized to extract high-frequency details which are combined with spatial features as input to the BSD. With this design BSD guides the Flow Generator (FG) to pay attention to various fine details rather than rigid pixel-level fitting. Extensive experiments conducted on the BR-5K dataset demonstrate that our framework significantly enhances the aesthetic appeal of reshaped photos surpassing all previous work to achieve state-of-the-art in all evaluation metrics. Our code is available at https://github.com/Randle-Github/AGGN.

Most virtual try-on research is motivated to serve the fashion business by generating images to demonstrate garments on studio models at a lower cost. However virtual try-on should be a broader application that also allows customers to visualize garments on themselves using their own casual photos known as in-the-wild try-on. Unfortunately the existing methods which achieve plausible results for studio try-on settings perform poorly in the in-the-wild context. This is because these methods often require paired images (garment images paired with images of people wearing the same garment) for training. While such paired data is easy to collect from shopping websites for studio settings it is difficult to obtain for in-the-wild scenes. In this work we fill the gap by (1) introducing a StreetTryOn benchmark to support in-the-wild virtual try-on applications and (2) proposing a novel method to learn virtual try-on from a set of in-the-wild person images directly without requiring paired data. We tackle the unique challenges including warping garments to more diverse human poses and rendering more complex backgrounds faithfully by a novel DensePose warping correction method combined with diffusion-based conditional inpainting. Our experiments show competitive performance for standard studio try-on tasks and SOTA performance for street try-on and cross-domain try-on tasks.

Unsupervised domain adaptation (UDA) has become increasingly prevalent in scene text recognition (STR) especially where training and testing data reside in different domains. The efficacy of existing UDA approaches tends to degrade when there is a large gap between the source and target domains. To deal with this problem gradually shifting or progressively learning to shift from domain to domain is the key issue. In this paper we introduce the Stratified Domain Adaptation (StrDA) approach which examines the gradual escalation of the domain gap for the learning process. The objective is to partition the target data into subsets so that the progressively self-trained model can adapt to gradual changes. We stratify the target data by evaluating the proximity of each data sample to both the source and target domains. We propose a novel method for employing domain discriminators to estimate the out-of-distribution and domain discriminative levels of data samples. Extensive experiments on benchmark scene-text datasets show that our approach significantly improves the performance of baseline (source-trained) STR models. The source code is available at https://github.com/KhaLee2307/StrDA.

Few-shot class incremental learning implies the model to learn new classes while retaining knowledge of previously learned classes with a small number of training instances. Existing frameworks typically freeze the parameters of the previously learned classes during the incorporation of new classes. However this approach often results in suboptimal class separation of previously learned classes leading to overlap between old and new classes. Consequently the performance of old classes degrades on new classes. To address these challenges we propose a novel feature augmentation driven contrastive learning framework designed to enhance the separation of previously learned classes to accommodate new classes. Our approach involves augmenting feature vectors and assigning proxy labels to these vectors. This strategy expands the feature space ensuring seamless integration of new classes within the expanded space. Additionally we employ a self-supervised contrastive loss to improve the separation between previous classes. We validate our framework through experiments on three FSCIL benchmark datasets: CIFAR100 miniImageNet and CUB200. The results demonstrate that our Feature Augmentation driven Contrastive Learning framework significantly outperforms other approaches achieving state-of-the-art performance.

Autofocus aims to accurately position the camera lens to bring the desired region of interest into focus. Conventional works search for the sharpest frame within the lens movement. However sharpness measure in many real-world settings is ambiguous and may cause a focus hunting problem where the lens continuously moves back and forth to search for the accurate position. To mitigate this problem we introduce a simple yet powerful loss function specifically designed to produce consistent outputs in autofocus systems. The proposed Focal Consistency Loss (FCL) allows autofocus models to better learn the geometric cues relative to each initial position of the lens significantly reducing distracting lens movement and enhancing the user experience when taking a photo. Furthermore we improve autofocus stability by utilizing multiple consecutive frames in a practical way. Experimental results show the effectiveness of FCL in various practical scenarios including multi-frame autofocus for both conventional and dual-pixel images.

Generating high-resolution images with generative models has recently been made widely accessible by leveraging diffusion models pre-trained on large-scale datasets. Various techniques such as MultiDiffusion and SyncDiffusion have further pushed image generation beyond training resolutions i.e. from square images to panorama by merging multiple overlapping diffusion paths or employing gradient descent to maintain perceptual coherence. However these methods suffer from significant computational inefficiencies due to generating and averaging numerous predictions which is required in practice to produce high-quality and seamless images. This work addresses this limitation and presents a novel approach that eliminates the need to generate and average numerous overlapping denoising predictions. Our method shifts non-overlapping denoising windows over time ensuring that seams in one timestep are corrected in the next. This results in coherent high-resolution images with fewer overall steps. We demonstrate the effectiveness of our approach through qualitative and quantitative evaluations comparing it with MultiDiffusion SyncDiffusion and StitchDiffusion. Our method offers several key benefits including improved computational efficiency and faster inference times while producing comparable or better image quality.

We present SplatFace a novel Gaussian splatting framework designed for 3D human face reconstruction without reliance on accurate pre-determined geometry. Our method is designed to simultaneously deliver both high-quality novel view rendering and accurate 3D mesh reconstructions. We incorporate a generic 3D Morphable Model (3DMM) to provide a surface geometric structure making it possible to reconstruct faces with a limited set of input images. We introduce a joint optimization strategy that refines both the Gaussians and the morphable surface through a synergistic non-rigid alignment process. A novel distance metric splat-to-surface is proposed to improve alignment by considering both the Gaussian position and covariance. The surface information is also utilized to incorporate a world-space densification process resulting in superior reconstruction quality. Our experimental analysis demonstrates that the proposed method is competitive with both other Gaussian splatting techniques in novel view synthesis and other 3D reconstruction methods in producing 3D face meshes with high geometric precision.

As a bio-inspired vision sensor the spiking camera has showcased remarkable capability in high-speed imaging with a sampling rate of 40000 Hz. Reconstructing clear images from continuous spike streams which is obtained by each photosensor continuously detecting photons and firing them asynchronously has garnered significant attention. Despite promising results existing spike-to-image reconstruction methods face challenges in balancing global receptive fields and efficient computation due to the inherent limitations of their backbones. Recently due to powerful long-range modeling and linear complexity the state space model (SSM) has emerged as a competitive alternative to CNNs and Transformers. In this paper we propose a lightweight spike-to-image reconstruction network that harnesses Mamba as the backbone. Our approach sequentially executes three core modules: temporal information integration spatial feature enhancement and progressive image reconstruction. The former accumulates cues across diverse temporal windows to explore both long-term and short-term contexts. Subsequently to model global dependencies while heightening local detail perception we develop a multi-scale SSM block characterized by multi-scale multi-direction scanning which effectively boosts spatial feature representations. Finally intensity images are decoded progressively from the enhanced light-intensity features. Extensive experiments on both synthetic and real-captured data demonstrate that our approach achieves state-of-the-art performance with only 10% of the network parameters and nearly two orders of magnitude less computational effort. The code will be available at https://github.com/interstellarH/Spk2ImgMamba.

We present SpiralMLP a novel architecture introduces a Spiral FC layer as a replacement for the conventional Token Mixing approach. Differing from several existing MLP-based models that primarily emphasize axes our Spiral FC layer is designed as a deformable convolution layer with spiral-like offsets. We further adapt Spiral FC into two variants: Self-Spiral FC and Cross-Spiral FC enabling both local and global feature integration seamlessly eliminating the need for additional processing steps. To thoroughly investigate the effectiveness of the spiral-like offsets and validate our design we conduct ablation studies and explore optimal configurations. In empirical tests SpiralMLP reaches state-of-the-art performance similar to Transformers CNNs and other MLPs benchmarking on ImageNet-1k COCO and ADE20K. SpiralMLP still maintains linear computational complexity O(HW) and is compatible with varying input image resolutions. Our study reveals that targeting the full receptive field is not essential for achieving high performance instead adopting a refined approach offers better results.

Vision transformers have been applied successfully for image recognition tasks. There have been either multiheaded self-attention based (ViT [12] DeIT [54]) similar to the original work in textual models or more recently based on spectral layers (Fnet [29] GFNet [46] AFNO [15]). We hypothesize that spectral layers capture high-frequency information such as lines and edges while attention layers capture token interactions. We investigate this hypothesis through this work and observe that indeed mixing spectral and multi-headed attention layers provides a better transformer architecture. We thus propose the novel Spectformer architecture for vision transformers that has initial spectral and deeper multi-headed attention layers. We believe that the resulting representation allows the transformer to capture the feature representation appropriately and it yields improved performance over other transformer representations. For instance it improves the top-1 accuracy by 2% on ImageNet compared to both GFNet-H and LiT. SpectFormer-H-S reaches 84.25% top-1 accuracy on ImageNet-1K (state of the art for small version). Further Spectformer-H-L achieves 85.7% which is the state of the art for the comparable base version of the transformers. We further validated the SpectFormer performance in other scenarios such as transfer learning on standard datasets such as CIFAR-10 CIFAR-100 Oxford- IIIT-flower and Standford Car datasets. We then investigate its use in downstream tasks such as object detection and instance segmentation on the MS-COCO dataset and observe that Spectformer shows consistent performance that is comparable to the best backbones and can be further optimized and improved. The source code is available on this website https://github.com/badripatro/SpectFormers.