WACV 2025 Papers

Hyperspectral imaging has emerged as a promising technology for medical image classification particularly in skin cancer diagnosis. However current methods face significant challenges in accurately and robustly classifying non-cancerous skin lesions especially when melanoma lesions overlap with pigmented regions. Existing methods also lack sensitivity to spectral variations and accumulate excess redundant data leading to inefficiencies misclassifications and overfitting while struggling to integrate spatial and spectral information effectively. To overcome these challenges we propose a novel method featuring collaborative sparse unmixing and an advanced pixel-wise shuffling approach with inter-similarity hybrid attention aiming to improve the accuracy of skin cancer diagnosis in real-world scenarios. Experiments are conducted on a publicly available histology-verified dataset to evaluate the efficacy of the proposed method. The experimental results demonstrate that the proposed method can accurately classify melanoma lesions even in cases where the lesions overlap with pigmented regions. The findings indicate that the proposed method outperforms state-of-the-art methods by obtaining an overall accuracy of 73.34% even when limited to 20% of the training data. The proposed approach has the potential to be a valuable tool for improving the diagnostic accuracy of skin cancer in clinical practice.

Face swapping is to interchange identity features such as eyes nose and lips between a source and a target face while preserving the target attributes like expression pose skin color and hair. Despite considerable advancements in quality over the years recent studies conducting high-resolution face swapping still encounter challenges in reflecting the source identity and ensuring robust performance. We identify two primary reasons for these challenges: (1) the absence of pixel-level supervision and (2) limitations in the model architecture and pipeline. To address the first problem we construct a pseudo-ground truth paired dataset and provide essential pixel-level supervision for effective attribute and identity handling during face swapping. Models trained on the paired dataset significantly enhance the source identity reflection compared to those without the paired dataset. On the other hand existing StyleGAN-based approaches often underutilize source latent vectors or heavily rely on pre-trained models during inference resulting in incomplete identity representation or reduced robustness. Therefore we introduce a novel face-swapping model that is adept at leveraging spatial information of the target attributes and fully utilizing features of the source identity. Thanks to the effective paired dataset and network our model achieves state-of-the-art performance in both image and video-level face swapping notably improving the source identity reflection while preserving the target attributes. Extensive experiments validate the superior performance of our model over existing baselines.

Extraction of building footprint polygons from remotely sensed data is essential for several urban understanding tasks such as reconstruction navigation & mapping. Despite significant progress in the area extracting accurate polygonal vector building footprints remains an open problem. In this paper we introduce Pix2Poly an attention-based end-to-end trainable & differentiable deep neural network capable of directly generating explicit high-quality building footprints in a ring graph format. Pix2Poly employs a generative encoder-decoder transformer to produce a sequence of graph vertex tokens whose connectivity information is learned by an optimal matching network. Compared to previous graph learning methods ours is a truly end-to-end trainable approach that extracts high-quality building footprints & road networks without requiring complicated computationally intensive raster loss functions & intricate training pipelines. Upon evaluating Pix2Poly on several complex & challenging datasets we report that Pix2Poly outperforms state-of-the-art methods in several vector shape quality metrics while being an entirely explicit method. Our code is available at https://github.com/yeshwanth95/Pix2Poly.

Self-supervised learning plays an important role in current state-of-the-art semi-supervised learning (SSL) methods. These methods learn inter-class heterogeneity among data and generate pseudo-labels based on class level representations. However they often neglect intra-class heterogeneity resulting in the under-exploitation of finer-grained semantic relationships within classes. To address this limitation we introduce PivotAlign a novel SSL approach that aims to 1) learn hierarchical representations to detect both inter-class and intra-class semantic relationships and 2) refine pseudo-labels based on learned representations with a class-debiasing strategy. Specifically we first learn a set of pivots as sub-prototypes of classes. We then train representations so that features align with the assigned pivot and are hierarchically grouped based on both inter-class and intra-class heterogeneity. This allows us to capture both inter-class and intra-class semantic relationships among data and leverage them to better assign and refine pseudo-labels. Additionally since SSL methods are prone to bias toward classes that are easier to learn we further re-balance class predictions to alleviate this class bias. We demonstrate the effectiveness of PivotAlign on various SSL benchmarks where PivotAlign achieves state-of-the-art performances. The source code will be released upon publication of the work.

Coronary artery disease (CAD) is a significant health risk that requires early detection for effective treatment. While recent advances in deep learning have shown promise in automating CAD detection from coronary computed tomography angiography (CCTA) images the accurate segmentation of coronary vessels remains a challenge particularly due to the imbalanced presence of plaque in unhealthy vessels. This paper introduces a physiology-aware approach to coronary vessel segmentation that addresses these challenges. Our proposed pipeline consists of three main components. First a hybrid UNeXt architecture is designed to segment artery boundaries and predict initial boundary contours by leveraging 3D spatial relations among adjacent slices. Second we introduce multi-class circular convolution for iterative contour deformation which generates well-connected contour pairs of the artery wall's inner and outer boundaries through iterative refinement. Finally we propose a focal smooth L1 loss function to handle the implicit class imbalance caused by plaque in unhealthy vessels and to enhance the robustness of the physiology-aware polysnake network by explicitly limiting the accuracy of initial contours. Extensive evaluations demonstrate that our methods significantly improve model performance achieving state-of-the-art results in coronary vessel segmentation.

Quality degradation is observed in underwater images due to the effects of light refraction and absorption by water leading to issues like color cast haziness and limited visibility. This degradation negatively affects the performance of autonomous underwater vehicles used in marine applications. To address these challenges we propose a lightweight phase-based transformer network with 1.77M parameters for underwater image restoration (UIR). Our approach focuses on effectively extracting non-contaminated features using a phase-based self-attention mechanism. We also introduce an optimized phase attention block to restore structural information by propagating prominent attentive features from the input. We evaluate our method on both synthetic (UIEB UFO-120) and real-world (UIEB U45 UCCS SQUID) underwater image datasets. Additionally we demonstrate its effectiveness for low-light image enhancement using the LOL dataset. Through extensive ablation studies and comparative analysis it is clear that the proposed approach outperforms existing state-of-the-art (SOTA) methods. The testing code is included in the supplementary material and will be publicly released upon acceptance.

The sharing of sensitive medical data among institutions presents a significant challenge due to strict privacy regulations the need for robust de-identification processes and the ethical imperative to protect patient confidentiality. Federated Learning (FL) addresses these challenges by enabling institutions to collaboratively train AI models on decentralized data thereby enhancing privacy and security without directly sharing sensitive patient information. However FL requires complex synchronization implementations has costly communication overheads and may fail to capture data heterogeneity across institutions. In this work we propose Personalized Mixture of Local Experts (P-MoLE) a Personalized Federated Learning (PFL) approach that effectively combines predictions from multiple locally trained models in a sample-specific manner. Leveraging both the individuality of each local model and variation across the ensemble P-MoLE learns the profile of each institution's local model and strategically weighs their prediction's contributions to the final segmentation. This approach harnesses the heterogeneity of each institution's unique data to increase the generalization capabilities across all institutions. By each institution sharing only the final models trained locally on the sensitive data no private patient data is exposed and the need for expensive communication infrastructure is removed. Results across two popular multi-institutional medical imaging datasets show P-MoLE achieves state-of-the-art performance without the extensive cooperative effort requirement of previous works. Additionally ablation study results show that P-MoLE is flexible to the number of local models in the ensemble increasing performance over the local models alone in each case.

Video Question Answering (Video QA) is a challenging video understanding task that requires models to comprehend entire videos identify the most relevant information based on contextual cues from a given question and reason accurately to provide answers. Recent advancements in Multimodal Large Language Models (MLLMs) have transformed video QA by leveraging their exceptional commonsense reasoning capabilities. This progress is largely driven by the effective alignment between visual data and the language space of MLLMs. However for video QA an additional space-time alignment poses a considerable challenge for extracting question-relevant information across frames. In this work we investigate diverse temporal modeling techniques to integrate with MLLMs aiming to achieve question-guided temporal modeling that leverages pre-trained visual and textual alignment in MLLMs. We propose T-Former a novel temporal modeling method that creates a question-guided temporal bridge between frame-wise visual perception and the reasoning capabilities of LLMs. Our evaluation across multiple video QA benchmarks demonstrates that T-Former competes favorably with existing temporal modeling approaches and aligns with recent advancements in video QA.

Photometric Stereo (PS) estimates surface normals by analyzing images lit from different angles. Enhancing PS with spectral imaging known as multispectral photometric stereo (MPS) uses varying light source colors for simultaneous image capture. As in traditional PS obtaining a unique solution is challenging in MPS when the reflectance properties of the object are unknown. This paper presents an approach utilizing the spatial arrangement and color of light sources to solve the MPS problem in the condition of spatially varying reflectance from a minimum of seven spectral images without spatial smoothness constraints. A robust optimization technique is introduced to manage real data. Experiments on synthetic and real scenes validate the method's effectiveness including for non-Lambertian surfaces. The method can contribute to advanced digital archiving that simultaneously records surface normal and spectral reflectance.

Despite the significant progress in deep learning for dense visual recognition problems such as semantic segmentation traditional methods are constrained by fixed class sets. Meanwhile vision-language foundation models such as CLIP have showcased remarkable effectiveness in numerous zero-shot image-level tasks owing to their robust generalizability. Recently a body of work has investigated utilizing these models in open-vocabulary semantic segmentation (OVSS). However existing approaches often rely on impractical supervised pre-training or access to additional pre-trained networks. In this work we propose a strong baseline for training-free OVSS termed Neighbour-Aware CLIP (NACLIP) representing a straightforward adaptation of CLIP tailored for this scenario. Our method enforces localization of patches in the self-attention of CLIP's vision transformer which despite being crucial for dense prediction tasks has been overlooked in the OVSS literature. By incorporating design choices favouring segmentation our approach significantly improves performance without requiring additional data auxiliary pre-trained networks or extensive hyperparameter tuning making it highly practical for real-world applications. Experiments are performed on 8 popular semantic segmentation benchmarks yielding state-of-the-art performance on most scenarios. Our code is publicly available at https://github.com/sinahmr/NACLIP.

For decades corporations and governments have relied on scanned documents to record vast amounts of information. However extracting this information is a slow and tedious process due to the sheer volume and complexity of these records. The rise of Vision Language Models (VLMs) presents a way to efficiently and accurately extract the information out of these documents. The current automated workflow often requires a two-step approach involving the extraction of information using optical character recognition software and subsequent usage of large language models for processing this information. Unfortunately these methods encounter significant challenges when dealing with noisy scanned documents often requiring computationally expensive language models to handle high information density effectively. In this study we propose PatchFinder an algorithm that builds upon VLMs to improve information extraction. First we devise a confidence-based score called Patch Confidence based on the Maximum Softmax Probability of the VLMs' output to measure the model's confidence in its predictions. Using this metric PatchFinder determines a suitable patch size partitions the input document into overlapping patches and generates confidence-based predictions for the target information. Our experimental results show that PatchFinder leveraging Phi-3v a 4.2 billion parameter VLM achieves an accuracy of 94% on our dataset of 190 noisy scanned documents outperforming ChatGPT-4o by 18.5 percentage points.

Contrastive image-text pre-trained models such as CLIP have shown remarkable adaptability to downstream tasks. However they face challenges due to the high computational requirements of the Vision Transformer (ViT) backbone. Current strategies to boost ViT efficiency focus on pruning patch tokens but fall short in addressing the multimodal nature of CLIP and identifying the optimal subset of tokens for maximum performance. To address this we propose greedy search methods to establish a "Golden Ranking" and introduce a lightweight predictor specifically trained to approximate this Ranking. To compensate for any performance degradation resulting from token pruning we incorporate learnable visual tokens that aid in restoring and potentially enhancing the model's performance. Our work presents a comprehensive and systematic investigation of pruning tokens within the ViT backbone of CLIP models. Through our framework we successfully reduced 40% of patch tokens in CLIP's ViT while only suffering a minimal average accuracy loss of 0.3% across seven datasets. Our study lays the groundwork for building more computationally efficient multimodal models without sacrificing their performance addressing a key challenge in the application of advanced vision-language models.

We propose Partial Texture VAE (PT-VAE) for rock particle image analysis a variant of the variational autoencoder (VAE) specialized in encoding color and texture properties from arbitrarily sized and shaped unresized images containing invalid background pixels into the vectors with the same size. PT-VAE integrates partial convolution and the mask aware feature aggregation layer in the encoder and is trained by minimizing a simplified style loss in an unsupervised manner. Input texture images do not require resizing or cropping thus preventing deformation and under-representation of the overall texture pattern. This consideration is often overlooked in data-driven texture analysis. Experiments using our textured rock particle images show that the encoded features efficiently capture color and texture information and are invariant to size and shape of the image as well as invalid pixels. Image retrieval tests show that the PT-VAE incorporating the mask aware Texture Encoding Layer (TEL) outperformed other configurations and existing methods.

Numerous deep learning techniques have been developed for Single Image Super-Resolution (SISR) leading to significant performance improvements. However these techniques have also resulted in a substantial increase in parameter size. As a result there is a growing interest in reducing network complexity for more practical usage while still maintaining high SR quality. One such method is parameter-sharing which includes recursive recurrent and multi-scale learning approaches. However sharing identical kernels across layers or up-scaling tasks can reduce the network's representational capacity. To address this we propose Partial filter-Sharing (PS) a new parameter-sharing method that preserves the network's representational power more effectively than previous approaches. Instead of sharing a single filter PS shares segments of filters called partial filters across layers. This approach enables parameter-sharing layers to use diverse filters for each layer or task striking a balance between parameter efficiency and the network's representational ability without imposing excessive computational or parameter overhead. Furthermore the PS framework provides precise control over the network's performance and complexity by adjusting the quantity of partial filters. Extensive experiments demonstrate that our PS framework outperforms traditional parameter-sharing super-resolution (SR) methods without incurring excessive additional parameters or computational cost.

In this paper we introduce a novel task for video understanding that focuses on detecting editing intentions in political advertisement videos. Political advertisement videos are edited with some intentions (e.g. "associating some candidates with negative emotions") of making people unthinkingly believe the messages in the videos potentially ending up with some irrational bias. Detecting such intentions is thus the primary step toward fairer decision-making based on the messages themselves. To this end we classify such editing intentions into 10 categories (referred to as communication techniques) in consultation with a professional editor as well as based on communication techniques presented in the natural language processing community and build a dataset of 12526 political advertisement videos each of which is annotated with several communication technique segments. We also explore the capability of existing video understanding models in detecting editing intentions over the dataset which identifies new dimensions of challenges to be addressed.

3D surface reconstruction from unorganized point clouds is a fundamental task in visual computing with numerous applications in areas such as robotics virtual reality augmented reality and animation. To date many deep learning-based surface reconstruction methods have been proposed demonstrating great performance on many benchmark datasets. Among these neural implicit field learning-based methods have gained popularity for their capability of representing complex structures in a continuous implicit distance field. Existing neural implicit field learning methods either utilize voxelized point cloud then feed them to a deep network or directly take points as input. In this paper we propose an implicit surface reconstruction framework based on point voxel geometric-aware transformer PVT to seamlessly integrate point-based convolution with voxel-based convolution using bidirectional transformers. Experiments show that the proposed PVT framework can better encode local geometry details and provide a significant performance boost over existing state-of-the-art methods.

Recently representations based on polar coordinates have exhibited promising characteristics for 3D perceptual tasks. In addition to Cartesian-based methods representing surrounding spaces through polar grids offers a compelling alternative in these tasks. This approach is advantageous for its ability to represent larger areas while preserving greater detail of nearby spaces. However polar-based methods are inherently challenged by the issue of feature distortion due to the non-uniform division inherent to polar representation. To harness the advantages of polar representation while addressing its challenges we propose Polar Voxel Occupancy Predictor (PVP) a novel 3D multi-modal occupancy predictor operating in polar coordinates. PVP mitigates the issues of feature distortion and misalignment across different modalities with the following two design elements: 1) Global Represent Propagation (GRP) module which incorporates global spatial information into the intermediate 3D volume taking into account the prior spatial structure. It then employs Global Decomposed Attention to accurately propagate features to their correct locations. 2) Plane Decomposed Convolution (PD-Conv) which simplifies 3D distortions in polar coordinates by replacing 3D convolution with a series of 2D convolutions. With these straightforward yet impactful modifications our PVP surpasses state-of-the-art works by significant margins--improving by 1.9% mIoU and 2.9% IoU over LiDAR-only methods and by 7.9% mIoU and 6.8% IoU over multimodal methods on the OpenOccupancy dataset.

Video crime detection is a significant application of computer vision and artificial intelligence. However existing datasets primarily focus on detecting severe crimes by analyzing entire video clips often neglecting the precursor activities (i.e. privacy violations) that could potentially prevent these crimes. To address this limitation we present PV-VTT (Privacy Violation Video To Text) a unique multimodal dataset aimed at identifying privacy violations. PV-VTT provides detailed annotations for both video and text in scenarios. To ensure the privacy of individuals in the videos we only provide video feature vectors avoiding the release of any raw video data. This privacy-focused approach allows researchers to use the dataset while protecting participant confidentiality. Recognizing that privacy violations are often ambiguous and context-dependent we propose a Graph Neural Network (GNN)-based video description model. Our model generates a GNN-based prompt with an image for a Large Language Model (LLM) which delivers cost-effective and high-quality video descriptions. By leveraging a single video frame along with relevant text our method reduces the number of input tokens required maintaining descriptive quality while optimizing LLM API usage. Extensive experiments validate the effectiveness and interpretability of our approach in video description tasks and the flexibility of our PV-VTT dataset.

The non-contact estimation of vital signs particularly heart rate from video data is a promising method for remote health monitoring. 3D convolutional layers are widely used for this task due to their ability to capture both spatial and temporal features. However traditional 3D convolutions while effective in many cases lack the capacity to adjust dynamically to the temporal variability inherent in physiological signals such as remote photoplethysmography (rPPG) which are characterized by subtle frequency changes over time. To address this we propose PULSE (Physiological Understanding with Liquid Signal Extraction) a framework that employs Liquid Time-Constant (LTC) models with 3D convolutional layers to enhance temporal sensitivity and improve the extraction of these fine-grained rPPG signals. In PULSE traditional 3D-conv layers are deployed for initial feature extraction while LTC-based 3D-conv layers dynamically adapt and guide the temporal processing allowing the model to better track and interpret the subtle variations in heart rate signals under different conditions such as motion artifacts and lighting changes. We evaluated the effectiveness of PULSE in an unsupervised training setting demonstrating that our solution performs well even in the absence of labeled datasets a common challenge in rPPG signal extraction. Experimental evaluations on three public datasets confirm that PULSE achieves comparable or superior results to existing methods proving its robustness and efficacy for real-world non-contact health monitoring applications.