WACV 2025 论文列表

This paper targets a new foundation for designing general-purpose learning systems by establishing a consensus method that facilitates self-adaptation and flexibility to deal with different learning tasks and different data distribution. We present the Shapely Consensus Deep Learning (SCDL) as a consensus method for general-purpose intelligence without the help of a domain expert. SCDL is two-level based learning process. In the first level several deep learning models have been trained for each historical observation. The Shapley Value is determined to compute the contribution of each subset of models in the training. The models are pruned according to their contribution in the learning process. In the second level the loss information of each data distribution is saved in the knowledge base. Both levels are explored to prune the models for each new observation. We present the evaluation of the generality of SCDL using different datasets with different shapes and complexities. The results reveal the effectiveness of SCDL for weakly classification. Concretely SCDL achieved 90% of AUC with less than 86% for the baseline solutions.

We introduce ShapeMorph a diffusion-based method specifically designed for generating precise and diverse 3D shape completions. By integrating an irregular discrete representation with a novel blockwise discrete diffusion model ShapeMorph can produce multiple high-quality shape completions while maintaining fidelity to the input. In particular each 3D shape is encoded into a compact sequence of irregularly distributed discrete variables ensuring an accurate capture of the object's topological details. We then propose a blockwise discrete diffusion model to precisely learn the shape completion distribution based on various incompleteness. We also introduce a Flow transformer into our diffusion process serving as a denoising network to enhance the modeling adaptability and flexibility. ShapeMorph addresses common challenges in existing methods such as poor completion limited diversity and misalignment with the input. Results show that ShapeMorph outperforms state-of-the-art methods and effectively processes a variety of input types and levels of incompleteness.

Recent advances in machine learning have greatly benefited object detection and 6D pose estimation. However textureless and metallic objects still pose a significant challenge due to few visual cues and the texture bias of CNNs. To address this issue we propose a strategy for inducing a shape bias to CNN training. In particular by randomizing textures applied to object surfaces during data rendering we create training data without consistent textural cues. This methodology allows for seamless integration into existing data rendering engines and results in negligible computational overhead for data rendering and network training. Our findings demonstrate that the shape bias we induce via randomized texturing improves over existing approaches using style transfer. We evaluate with five detectors and two pose estimators. For three object detectors and for pose estimation in general estimation accuracy improves for textureless and metallic objects. Additionally we show that our approach increases the pose estimation accuracy in the presence of image noise and strong illumination changes. Code available at https://github.com/hoenigpeter/randomized_texturing.

Shadow boundaries can be confused with material boundaries as both exhibit sharp changes in luminance or contrast within a scene. However shadows do not modify the intrinsic color or texture of surfaces. Therefore on both sides of shadow edges traversing regions with the same material the original color and texture should be the same if the shadow is removed properly. These shadow/shadow-free pairs are very useful but difficult-to-collect supervision signals. The crucial contribution of this paper is to learn how to identify those shadow edges that traverse material-consistent regions and how to use them as self-supervision for shadow removal refinement during test time. To achieve this we fine-tune SAM an image segmentation foundation model to produce a shadow-invariant segmentation and then extract material-consistent shadow edges by comparing the SAM segmentation with the shadow mask. Utilizing these shadow edges we introduce color- and texture-consistency losses to enhance the shadow removal process. We demonstrate the effectiveness of our method in improving shadow removal results on more challenging in-the-wild images outperforming the state-of-the-art shadow removal methods. Additionally we propose a new metric and an annotated dataset for evaluating the performance of shadow removal methods without the need for paired shadow/shadow-free data. Our code and dataset are available at: https://github.com/cvlab-stonybrook/ShadowRemovalRefine

Separating an image of a scene illuminated by a light source into direct components such as specular reflection and diffuse reflection and global components such as inter-reflection and subsurface scattering is important as preprocessing for various computer vision and graphics applications. Conventional methods cannot separate direct and global components from novel viewpoints and have difficulties in robustly separating those components from a small number of images even from known viewpoints. In this paper we propose a method for synthesizing the direct and global components of a scene from novel viewpoints by using a relatively small number of images. Specifically our proposed method uses the multi-view images captured by using a coaxial projector-camera system and then recovers the density and radiance values of each component on the basis of neural radiance fields (NeRF). We conduct a number of experiments using real images captured with a projector-camera system and confirm the effectiveness of our method. In addition we demonstrate that our method is useful for two applications: image-based material editing and 3D shape recovery.

We present SensorFlow a novel image and sensor fusion framework for robust high-quality video stabilization. We start with sensor-based pre-stabilization that smooths out large-scale camera motion. A new angular velocity domain optimization has been introduced to achieve frame rate invariance. We then feed the stabilized optical flows into an occlusion-aware 3D CNN that infers dense warp fields to remove residual translation and jitter. To further avoid distortion we propose a novel masking scheme to determine the disoccluded and dynamic regions in optical flow and inpaint them with spatially smooth flow vectors. Our method is appealing as it shares both the dense warping field's flexibility to correct complex motions and the robustness of sensor data for arbitrarily challenging scenes. We have validated its effectiveness and demonstrated our solution outperforms state-of-the-art alternatives via extensive ablation studies and quantitative comparisons.

Pre-trained vision-language models (VLMs) such as CLIP demonstrate impressive zero-shot classification capabilities with free-form prompts and even show some generalization in specialized domains. However their performance on satellite imagery is limited due to the under representation of such data in their training sets which predominantly consist of ground-level images. Existing prompting techniques for satellite imagery are often restricted to generic phrases like "a satellite image of..." limiting their effectiveness for zero-shot land-use/land-cover (LULC) mapping. To address these challenges we introduce SenCLIP which transfers CLIP's representation to Sentinel-2 imagery by leveraging a large dataset of Sentinel-2 images paired with geotagged ground-level photos from across Europe. We evaluate SenCLIP alongside other state-of-the-art remote sensing VLMs on zero-shot LULC mapping tasks using the EuroSAT and BigEarthNet datasets with both aerial and ground-level prompting styles. Our approach which aligns ground-level representations with satellite imagery demonstrates significant improvements in classification accuracy across both prompt styles opening new possibilities for applying free-form textual descriptions in zero-shot LULC mapping. Code dataset and pretrained models are available at https://github.com/pallavijain-pj/SenCLIP

Image Visual Sentiment Analysis (VSA) requires the availability of large annotated datasets whose construction presents many challenges. The necessity of gathering a large amount of labeled images contrasts with the rigorous but lengthy process required for manual annotation based on psychovisual experiments and with the automatic gathering of large amounts of data roughly labeled based on the sentiment analysis of the text accompanying the images like captions tweets and tags. An additional limitation is the scarcity of high-quality datasets with a neutral class which forces the images to be classified into emotions even when the observers show no emotional activation. In this work we present a scalable methodology rooted in semiotics and art theory for the construction of a 3-class (positive negative and neutral) VSA dataset enabling the downloading of a desired quantity of images while maintaining labeling coherence and accuracy. Based on the proposed methodology we introduce and make publicly available a VSA dataset of over 100000 images. To validate the quality of the dataset we used it to train several classifiers and compared their performance with those of classifiers trained on other datasets. The results show that the classifiers trained on the new dataset provide better performance when tested on independent datasets including those commonly used for psychovisual experiments.

This paper tackles the domain of multimodal prompting for visual recognition specifically when dealing with missing modalities through multimodal Transformers. It presents two main contributions: (i) we introduce a novel prompt learning module which is designed to produce sample-specific prompts and (ii) we show that modality-agnostic prompts can effectively adjust to diverse missing modality scenarios. Our model termed SCP exploits the semantic representation of available modalities to query a learnable memory bank which allows the generation of prompts based on the semantics of the input. Notably SCP distinguishes itself from existing methodologies for its capacity of self-adjusting to both the missing modality scenario and the semantic context of the input without prior knowledge about the specific missing modality and the number of modalities. Through extensive experiments we show the effectiveness of the proposed prompt learning framework and demonstrate enhanced performance and robustness across a spectrum of missing modality cases. Our source code is available at https://github.com/vittoriopipoli/SCP_WACV2025.

3D point cloud semantic segmentation is an important step for 3D indoors reconstruction. In recent years many outstanding deep learning models have been proposed for semantic segmentation which can achieve remarkable performance. However it is found the index indicating semantic prediction accuracy in terms of structural components (e.g. columns beams etc.) in buildings is far from satisfying lacking sufficient information for subsequent 3D reconstruction. For better segmenting and identifying structural components this work proposes Architectural-Knowledge-Aware features (AKAFs) i.e. F1 and F2 which are strategically incorporated into a developed two-stage training framework wherein outstanding semantic segmentation models are adopted as backbones. By incorporating F1 which formalizes the position distribution pattern of building structural components semantic information can be explored preliminary in the explicit stage (i.e. the first stage). The second stage is the implicit stage where F2 is derived based on the Semantic and Relative Position Fusing Module (SRPFM) which implicitly introduces more relative position information of building components for semantic segmentation. Extensive experiments have been conducted on the S3DIS dataset adopting three outstanding backbones. Results demonstrate that the proposed AKAFs can increase the accuracy of segmentation for structural components by more than 5%. As a result the overall semantic segmentation performance increases by 2%. Even on the current SOTA model (PT-v3) the proposed AKAFs show promising semantic segmentation promotion. Portable and well-compatible with most point-based semantic segmentation models AKAFs can effectively perceive more architectural characteristics hidden in indoor point clouds especially prominent for points of structural components.

Continual learning aims to refine model parameters for new tasks while retaining knowledge from previous tasks. Recently prompt-based learning has emerged to leverage pre-trained models to be prompted to learn subsequent tasks without the reliance on the rehearsal buffer. Although this approach has demonstrated outstanding results existing methods depend on preceding task-selection process to choose appropriate prompts. However imperfectness in task-selection may lead to negative impacts on the performance particularly in the scenarios where the number of tasks is large or task distributions are imbalanced. To address this issue we introduce a novel task-agnostic approach that focuses on the visual semantic information of image tokens eliminating the preceding task prediction. By leveraging the ability of the pre-trained model to discriminate between similar tokens our method not only subdivides the prompt but also eliminates the need for additional forward pass. Consequently we achieve competitive performance on four benchmarks while significantly reducing training time compared to state-of-the-art methods. The code is available at https://github.com/pilsHan/I-Prompt

Weakly-Supervised Semantic Segmentation (WSSS) aims to train segmentation models using image data with only image-level supervision. Since precise pixel-level annotations are not accessible existing methods typically focus on producing pseudo masks for training segmentation models by refining CAM-like heatmaps. However the produced heatmaps may capture only the discriminative image regions of object categories or the associated co-occurring backgrounds. To address the issues we propose a Semantic Prompt Learning for WSSS (SemPLeS) framework which learns to effectively prompt the CLIP latent space to enhance the semantic alignment between the segmented regions and the target object categories. More specifically we propose Contrastive Prompt Learning and Prompt-guided Semantic Refinement to learn the prompts that adequately describe and suppress the co-occurring backgrounds associated with each object category. In this way SemPLeS can perform better semantic alignment between object regions and class labels resulting in desired pseudo masks for training segmentation models. The proposed SemPLeS framework achieves competitive performance on standard WSSS benchmarks PASCAL VOC 2012 and MS COCO 2014 and shows compatibility with other WSSS methods. Project page: https://projectdisr.github.io/semples/

Accurate self-localization of unmanned aerial systems (UAS) is needed to reduce their dependency on global navigation satellite systems (GNSS). Image retrieval techniques comparing aerial images with a reference database can be used for visual localization (VL). But the search space may be vast and a full search not feasible on a small UAS. In this work we propose a novel solution that divides the reference database into smaller clusters based on the semantic content of images. To this end we generate and make use of a dataset for semantic segmentation of aerial image captures. By characterizing scenes and objects in images semantically retrieval-based systems are able to differentiate images and scenes efficiently. Using a divide-and-conquer approach images with similar semantics are matched within smaller partial databases. This technique leads to reduced search times and approaches VL as a feasible solution for UAS localization in large-scale outdoor environments.

Deep neural networks have been extensively applied in the medical domain for various tasks including image classification segmentation and landmark detection. However their application is often hindered by data scarcity both in terms of available annotations and images. This study introduces a novel application of denoising diffusion probabilistic models (DDPMs) to the landmark detection task specifically addressing the challenge of limited annotated data in x-ray imaging. Our key innovation lies in leveraging DDPMs for self-supervised pre-training in landmark detection a previously unexplored approach in this domain. This method enables accurate landmark detection with minimal annotated training data (as few as 50 images) surpassing both ImageNet supervised pre-training and traditional self-supervised techniques across three popular x-ray benchmark datasets. To our knowledge this work represents the first application of diffusion models for self-supervised learning in landmark detection which may offer a valuable pre-training approach in few-shot regimes for mitigating data scarcity. To bolster further development and reproducibility we provide open access to our code and pre-trained models for a variety of x-ray related applications: https://github.com/Malga-Vision/DiffusionXray-FewShot-LandmarkDetection.

Hyperspectral image (HSI) reconstruction from coded measurement is significant for acquiring images with higher spectral resolution than traditional RGB images. Current advanced neural networks have already shown impressive performance in some datasets like CAVE and KAIST. However these networks rely on a large amount of simulated ground truth measurement pairs. Unfortunately in some scenarios it is hard to obtain a sufficient high-quality HSI training set resulting in low generalization ability. Although iterative algorithms show good generalization ability they are limited by slow speed and low reconstruction quality. To address this challenge in this paper we propose a self-supervised learning framework which can train and fine-tune networks using measurements without ground truth. Besides we propose the spectral low-rank loss function that enables networks to learn the signal model of HSI. Finally we train and fine-tune a representative deep unfolding network GAP-net using our proposed framework. Extensive simulation and real data results show that the proposed self-supervised framework is capable of achieving results competitive with those of supervised networks. Code is available at https://github.com/zjhe02/CASSI-SSL.

Numerical weather prediction (NWP) models are fundamental in meteorology for simulating and forecasting the behavior of various atmospheric variables. The accuracy of precipitation forecasts and the acquisition of sufficient lead time are crucial for preventing hazardous weather events. However the performance of NWP models is limited by the nonlinear and unpredictable patterns of extreme weather phenomena driven by temporal dynamics. In this regard we propose a Self-Supervised Learning with Probabilistic Density Labeling (SSLPDL) for estimating rainfall probability by post-processing NWP forecasts. Our post-processing method uses self-supervised learning (SSL) with masked modeling for reconstructing atmospheric physics variables enabling the model to learn the dependency between variables. The pre-trained encoder is then utilized in transfer learning to a precipitation segmentation task. Furthermore we introduce a straightforward labeling approach based on probability density to address the class imbalance in extreme weather phenomena like heavy rain events. Experimental results show that SSLPDL surpasses other precipitation forecasting models in regional precipitation post-processing and demonstrates competitive performance in extending forecast lead times. Our code is available at https://github.com/joonha425/SSLPDL.

Computing a comprehensive and robust visual representation of an arbitrary object or category of objects is a complex problem. The difficulty increases when one starts from a set of uncalibrated images obtained from different sources. We propose a self-supervised approach Multi-Image Latent Embedding (MILE) which computes a single representation from such an image set. MILE operates incrementally considering one image at a time while processing various depictions of the class through a shared gated cross-attention mechanism. The representations are progressively refined as more available images are incorporated without requiring additional training. Our experiments on Amazon Berkeley Objects (ABO) and iNaturalist demonstrate the effectiveness in two tasks: object or category-specific image retrieval and unsupervised context-conditioned object segmentation. Moreover the proposed multi-image input setup opens new frontiers for the task of object retrieval. Our studies indicate that our models can capture descriptive representations that better encapsulate the intrinsic characteristics of the objects. Our code is available at https://github.com/amazon-science/mile.

A robust anomaly detection mechanism should possess the capability to effectively remediate anomalies restoring them to a healthy state while preserving essential healthy information. Despite the efficacy of existing generative models in learning the underlying distribution of healthy reference data they face primary challenges when it comes to efficiently repair larger anomalies or anomalies situated near high pixel-density regions. In this paper we introduce a self-supervised anomaly detection method based on a diffusion model that samples from multi-frequency four dimensional simplex noise and makes predictions using our proposed Dynamic Transformer UNet (DTUNet). This simplex-based noise function helps address primary problems to some extent and is scalable for three-dimensional and colored images. In the evolution of ViT our developed architecture serving as the backbone for the diffusion model is tailored to treat time and noise image patches as tokens. We incorporate long skip connections bridging the shallow and deep layers along with smaller skip connections within these layers. Furthermore we integrate a partial diffusion Markov process which reduces sampling time thus enhancing scalability. Our method surpasses existing generative-based anomaly detection methods across three diverse datasets which include BrainMRI Brats2021 and the MVtec dataset. It achieves an average improvement of +10.1% in Dice coefficient +10.4% in IOU and +9.6% in AUC. Our source code is made publicly available on Github.

Severity level estimation is a crucial task in medical image diagnosis. However accurately assigning severity class labels to individual images is very costly and challenging. Consequently the attached labels tend to be noisy. In this paper we propose a new framework for training with "ordinal" noisy labels. Since severity levels have an ordinal relationship we can leverage this to train a classifier while mitigating the negative effects of noisy labels. Our framework uses two techniques: clean sample selection and dual-network architecture. A technical highlight of our approach is the use of soft labels derived from noisy hard labels. By appropriately using the soft and hard labels in the two techniques we achieve more accurate sample selection and robust network training. The proposed method outperforms various state-of-the-art methods in experiments using two endoscopic ulcerative colitis (UC) datasets and a retinal Diabetic Retinopathy (DR) dataset. Our codes are available at https://github.com/shumpei-takezaki/Self-Relaxed-Joint-Training.