WACV 2025 论文列表

Picture quality serves as a primary differentiator for prominent display panel manufacturers. AI-based solutions have made remarkable progress in delivering expert-level image color remastering operations. However their demand on intensive computation resources heavily impedes the on-device usage in industries where speed and scale are crucial. In this paper we propose an extremely lightweight image-retouching network (eLIR-Net) that can be deployed on resource-restricted hardware like mobile terminals and embedded devices. The eLIR-Net takes in easily computable and intuitive color distributions to represent the macro view of an image which is encoded by a well-designed condition network to guide the generative base network to converge in the desired direction thus generating visually captivating outputs. Both quantitative and qualitative results show the proposed eLIR-Net can achieve equivalent or superior performance compared to the benchmark models at an affordable cost 6.8K parameters which is 24.1% the size of the smallest state-of-the-art network to the best of our knowledge. This work showcases the possibility that compact models deliver competitive performance compared to large models with affordable cost that enables the benefits of AI to be shared across a wider range of industrial applications.

We present ZeroComp an effective zero-shot 3D object compositing approach that does not require paired composite-scene images during training. Our method leverages ControlNet to condition from intrinsic images and combines it with a Stable Diffusion model to utilize its scene priors together operating as an effective rendering engine. During training ZeroComp uses intrinsic images based on geometry albedo and masked shading all without the need for paired images of scenes with and without composite objects. Once trained it seamlessly integrates virtual 3D objects into scenes adjusting shading to create realistic composites. We develop a high-quality evaluation dataset and demonstrate that ZeroComp outperforms methods using explicit lighting estimations and generative techniques in quantitative and human perception benchmarks. Additionally ZeroComp extends to real and outdoor image compositing even when trained solely on synthetic indoor data showcasing its effectiveness in image compositing.

We address the problem of detecting out-of-context (OOC) objects in a scene. Given an image we aim to detect whether the image has objects that are not present in their usual context and localize such OOC objects. Existing approaches for OOC detection rely on defining the common context in terms of the manually constructed features such as the co-occurrence of objects spatial relations between objects and shape and size of the objects and then learning such context for a given dataset. But context is often nuanced ranging from very common to very surprising. Further learned context from specific datasets may not be generalized as datasets may not truly represent the human notion of what is in context. Motivated by the success of large language models and more generally foundation models (FMs) in common sense reasoning we investigate the FM's ability to capture a more generalized notion of context. We find that a pre-trained FM such as GPT-4 provides a more nuanced notion of OOC and enables zero-shot OOC detection when coupled with other pre-trained FMs for caption generation such as BLIP-2 and image in-painting with Stable Diffusion 2.0. Our approach does not need any dataset-specific training. We demonstrate the efficacy of our approach on two OOC object detection datasets achieving 90.8% zero-shot accuracy on the MIT-OOC dataset and 87.26% on the IJCAI22-COCO-OOC dataset.

Machine unlearning is a crucial area of research. It is driven by the need to remove sensitive information from models to safeguard individuals' right to be forgotten under rigorous regulations such as GDPR. In this work we focus on unlearning within CLIP a dual vision-language encoder model trained on a massive dataset of image-text pairs using contrastive loss. To achieve forgetting we expand the application of Lipschitz regularization to the multimodal context of CLIP. Specifically we smooth both visual and textual embeddings associated with the class intended to be forgotten relative to the perturbation introduced to the samples from that class. Additionally importantly we remove the necessity for real forgetting data by generating synthetic samples via gradient ascent maximizing the target class. Our forgetting procedure is iterative where we track accuracy on a synthetic forget set and stop when accuracy falls below a chosen threshold. We employ a selective layers update strategy based on their average absolute gradient value to mitigate over-forgetting. We validate our approach on several standard datasets and provide thorough ablation analysis and comparisons with previous work.

Facade semantic segmentation is a long-standing challenge in photogrammetry and computer vision. Although the last decades have witnessed the influx of facade segmentation methods there is a lack of comprehensive facade classes and data covering the architectural variability. In ZAHA we introduce Level of Facade Generalization (LoFG) novel hierarchical facade classes designed based on international urban modeling standards ensuring compatibility with real-world challenging classes and uniform methods' comparison. Realizing the LoFG we present to date the largest semantic 3D facade segmentation dataset providing 601 million annotated points at five and 15 classes of LoFG2 and LoFG3 respectively. Moreover we analyze the performance of baseline semantic segmentation methods on our introduced LoFG classes and data complementing it with a discussion on the unresolved challenges for facade segmentation. We firmly believe that ZAHA shall facilitate further development of 3D facade semantic segmentation methods enabling robust segmentation indispensable in creating urban digital twins.

Tracking the full body motions of users in XR (AR/VR) devices is a fundamental challenge to bring a sense of authentic social presence. Due to the absence of dedicated leg sensors currently available body tracking methods adopt a synthesis approach to generate plausible motions given a 3-point signal from the head and controller tracking. In order to enable mixed reality features modern XR devices are capable of estimating depth information of the headset surroundings using available sensors combined with dedicated machine learning models. Such egocentric depth sensing cannot drive the body directly as it is not registered and is incomplete due to limited field-of-view and body self-occlusions. For the first time we propose to leverage the available depth sensing signal combined with self-supervision to learn a multi-modal pose estimation model capable of tracking full body motions in real time on XR devices. We demonstrate how current 3-point motion synthesis models can be extended to point cloud modalities using a semantic point cloud encoder network combined with a residual network for multi-modal pose estimation. These modules are trained jointly in a self-supervised way leveraging a combination of real unregistered point clouds and simulated data obtained from motion capture. We compare our approach against several state-of-the-art systems for XR body tracking and show that our method accurately tracks a diverse range of body motions. XR-MBT tracks legs in XR for the first time whereas traditional synthesis approaches based on partial body tracking are blind.

Recent advances in deep learning have enabled considerable strides in hand pose estimation in well-lit conditions. However to the best of our knowledge there is no existing method for hand pose estimation from RGB images captured in low light conditions. This task is highly challenging due to the overwhelming amount of noise which plague image capture in low-light conditions (<1 lux). In this paper we propose XPose the first method for extreme low light hand pose estimation from RGB images. We also introduce the first dataset for low light hand pose estimation consisting of 120k images along with accurate hand pose labels. Our dataset consists of images captured in low light and well-lit conditions from multiple viewpoints. We propose an innovative deep learning based methodology for monocular low-light hand pose estimation using guidance from well-lit and multi-view images available in our dataset during training time. We show that our method using the proposed LLPose dataset significantly outperforms existing methods for hand pose estimation both qualitatively and quantitatively in low light conditions.

In recent years Graph Neural Networks (GNNs) have demonstrated strong adaptability to various real-world challenges with architectures such as Vision GNN (ViG) achieving state-of-the-art performance in several computer vision tasks. However their practical applicability is hindered by the computational complexity of constructing the graph which scales quadratically with the image size. In this paper we introduce a novel Windowed vision Graph neural Network (WiGNet) model for efficient image processing. WiGNet explores a different strategy from previous works by partitioning the image into windows and constructing a graph within each window. Therefore our model uses graph convolutions instead of the typical 2D convolution or self-attention mechanism. WiGNet effectively manages computational and memory complexity for large image sizes. We evaluate our method in the ImageNet-1k benchmark dataset and test the adaptability of WiGNet using the CelebA-HQ dataset as a downstream task with higher-resolution images. In both of these scenarios our method achieves competitive results compared to previous vision GNNs while keeping memory and computational complexity at bay. WiGNet offers a promising solution toward the deployment of vision GNNs in real-world applications. We publicly released the code and pre-trained models at https://github.com/EIDOSLAB/WiGNet.

Recent advancements in large vision-language models (LVLM) have significantly enhanced their ability to comprehend visual inputs alongside natural language. However a major challenge in their real-world application is hallucination where LVLMs generate non-existent visual elements eroding user trust. The underlying mechanism driving this multimodal hallucination is poorly understood. Minimal research has illuminated whether contexts such as sky tree or grass field involve the LVLM in hallucinating a frisbee. We hypothesize that hidden factors such as objects contexts and semantic foreground-background structures induce hallucination. This study proposes a novel causal approach: a hallucination probing system to identify these hidden factors. By analyzing the causality between images text prompts and network saliency we systematically explore interventions to block these factors. Our experimental findings show that a straightforward technique based on our analysis can significantly reduce hallucinations. Additionally our analyses indicate the potential to edit network internals to minimize hallucinated outputs.

Self-attention in Transformers comes with a high computational cost because of their quadratic computational complexity but their effectiveness in addressing problems in language and vision has sparked extensive research aimed at enhancing their efficiency. However diverse experimental conditions spanning multiple input domains prevent a fair comparison based solely on reported results posing challenges for model selection. To address this gap in comparability we perform a large-scale benchmark of more than 45 models for image classification evaluating key efficiency aspects including accuracy speed and memory usage. Our benchmark provides a standardized baseline for efficiency-oriented transformers. We analyze the results based on the Pareto front - the boundary of optimal models. Surprisingly despite claims of other models being more efficient ViT remains Pareto optimal across multiple metrics. We observe that hybrid attention-CNN models exhibit remarkable inference memory- and parameter-efficiency. Moreover our benchmark shows that using a larger model in general is more efficient than using higher resolution images. Thanks to our holistic evaluation we provide a centralized resource for practitioners and researchers facilitating informed decisions when selecting or developing efficient transformers.

The recently proposed Visual State Space Model (VMamba) operating on the principle of state space mechanisms (SSM) processes images as a sequence of patches and outperforms Vision Transformers (ViT) in several computer vision tasks. Given their substantial design differences from CNNs and ViT it is crucial to investigate their vulnerability to backdoor attacks and the impact of various advanced backdoor attacks on their robustness. Backdoor attacks involve embedding a specific trigger into a small subset of training images which remains dormant until activated later. While the model performs well on clean test images an attacker can manipulate its decisions by presenting the trigger in one of the test images. This work examines state-of-the-art (SOTA) vision architectures (ResNet ViT MLP-mixer VMamba) focusing on their susceptibility to backdoor attacks and the effect of different backdoor attacks on their robustness. The well-known Visual State Space Model (VMamba) is the least susceptible to backdoor attacks among these architectures. To address this in this paper we propose two novel QR decomposition-based backdoor attacks that are visually imperceptible and achieve a high attack success rate (ASR) against the VMamba. We also present a qualitative analysis of the proposed backdoor attacks explaining the reasons behind their success or failure against the VMamba model. Experiments and results conducted on two popular image datasets (CIFAR-10 and ImageNet-1K) demonstrate that the proposed backdoor attacks exceed the performance of SOTA backdoor attacks and effectively fool the recently proposed VMamba model.

In Federated Learning (FL) multiple clients collaboratively train a global model without sharing private data. In semantic segmentation the Federated source Free Domain Adaptation (FFREEDA) setting is of particular interest where clients undergo unsupervised training after supervised pretraining at the server side. While few recent works address FL for autonomous vehicles intrinsic real-world challenges such as the presence of adverse weather conditions and the existence of different autonomous agents are still unexplored. To bridge this gap we address both problems and introduce a new federated semantic segmentation setting where both car and drone clients co-exist and collaborate. Specifically we propose a novel approach for this setting which exploits a batch-norm weather-aware strategy to dynamically adapt the model to the different weather conditions while hyperbolic space prototypes are used to align the heterogeneous client representations. Finally we introduce FLYAWARE the first semantic segmentation dataset with adverse weather data for aerial vehicles.

Few-shot knowledge distillation recently emerged as a viable approach to harness the knowledge of large-scale pre-trained models using limited data and computational resources. In this paper we propose a novel few-shot feature distillation approach for vision transformers. Our approach is based on two key steps. Leveraging the fact that vision transformers have a consistent depth-wise structure we first copy the weights from intermittent layers of existing pre-trained vision transformers (teachers) into shallower architectures (students) where the intermittence factor controls the complexity of the student transformer with respect to its teacher. Next we employ an enhanced version of Low-Rank Adaptation (LoRA) to distill knowledge into the student in a few-shot scenario aiming to recover the information processing carried out by the skipped teacher layers. We present comprehensive experiments with supervised and self-supervised transformers as teachers on six data sets from various domains (natural medical and satellite images) and tasks (classification and segmentation). The empirical results confirm the superiority of our approach over state-of-the-art competitors. Moreover the ablation results demonstrate the usefulness of each component of the proposed pipeline. We release our code at https://github.com/dianagrigore/WeCoLoRA.

Weeds are one of the major reasons for crop yield loss but current weeding practices fail to manage weeds in an efficient and targeted manner. Effective weed management is especially important for crops with high worldwide production such as maize to maximize crop yield for meeting increasing global demands. Advances in near-sensing and computer vision enable the development of new tools for weed management. Specifically state-of-the-art segmentation models coupled with novel sensing technologies can facilitate timely and accurate weeding and monitoring systems. However learning-based approaches require annotated data and show a lack of generalization to aerial imaging for different crops. We present a novel dataset for semantic and instance segmentation of crops and weeds in agricultural maize fields. The multispectral UAV-based dataset contains images with RGB red-edge and near-infrared bands a large number of plant instances dense annotations for maize and four weed classes and is multitemporal. We provide extensive baseline results for both tasks including probabilistic methods to quantify prediction uncertainty improve model calibration and demonstrate the approach's applicability to out-of-distribution data. The results show the effectiveness of the two additional bands compared to RGB only and better performance in our target domain than models trained on existing datasets. We hope our dataset advances research on methods and operational systems for fine-grained weed identification enhancing the robustness and applicability of UAV-based weed management. The dataset and code are available at https://github.com/GFZ/weedsgalore.

Hyperspectral imagery (HSI) provides detailed spectral information enabling precise analysis of materials. However HSI imaging suffers from blurring degradation which results in the loss of fine details and hinders subsequent applications. The degree of blurriness is highly related to wavelength and depth existing deblurring methods either lack the utilization of spectral correlation or ignore the depth variation since paired HSI and depth data are difficult to acquire and less discussed leading to degraded performance when encountering wide-range HSIs of non-planar scenes. To address these challenges in both data acquisition and algorithm design we propose a novel approach that simultaneously collects both modalities and integrates depth refinement into a blind HSI deblurring model with wavelength- and depth-aware deep image prior. Specifically we capture blurred HSI and coarse depth map with separate devices followed by registration. Our method performs depth-guided deblurring through depth-variant multi-channel kernel estimation and soft-weight map-based layer composition while simultaneously refining the depth. The proposed approach effectively restores fine details with fewer artifacts showing superior performance for both simulated blurred HSIs and real captured HSIs.

Recent advanced GAN inversion models aim to convey high-fidelity information from original images to generators through methods using generator tuning or high-dimensional feature learning. Despite these efforts accurately reconstructing image-specific details remains as a challenge due to the inherent limitations both in terms of training and structural aspects leading to a bias towards low-frequency information. In this paper we look into the widely used pixel loss in GAN inversion revealing its predominant focus on the reconstruction of low-frequency features. We then propose WINE a Wavelet-guided GAN inversion and editing model which transfers the high-frequency information through wavelet coefficients via newly proposed wavelet loss and wavelet fusion scheme. Notably WINE is the first attempt to interpret GAN inversion in the frequency domain. Our experimental results showcase the precision of WINE in preserving high-frequency details and enhancing image quality. Even in editing scenarios WINE outperforms existing state-of-the-art GAN inversion models with a fine balance between editability and reconstruction quality. Pre-trained model and codes will be publicized after the review process.

This paper introduces WARLearn a novel framework designed for adaptive representation learning in challenging and adversarial weather conditions. Leveraging the in-variance principal used in Barlow Twins we demonstrate the capability to port the existing models initially trained on clear weather data to effectively handle adverse weather conditions. With minimal additional training our method exhibits remarkable performance gains in scenarios characterized by fog and low-light conditions. This adaptive framework extends its applicability beyond adverse weather settings offering a versatile solution for domains exhibiting variations in data distributions. Furthermore WARLearn is invaluable in scenarios where data distributions undergo significant shifts over time enabling models to remain updated and accurate. Our experimental findings reveal a remarkable performance with a mean average precision (mAP) of 52.6% on unseen real-world foggy dataset (RTTS). Similarly in low light conditions our framework achieves a mAP of 55.7% on unseen real-world low light dataset (ExDark). Notably WARLearn surpasses the performance of state-of-the-art frameworks including FeatEnHancer Image Adaptive YOLO DENet C2PNet PairLIE and ZeroDCE by a substantial margin in adverse weather improving the baseline performance in both foggy and low light conditions. The WARLearn code is available at https://github.com/ShubhamAgarwal12/WARLearn

Buildings are a central feature of human culture and are increasingly being analyzed with computational methods. However recent works on computational building understanding have largely focused on natural imagery of buildings neglecting the fundamental element defining a building's structure - its floorplan. Conversely existing works on floorplan understanding are extremely limited in scope often focusing on floorplans of a single semantic category and region (e.g. floorplans of apartments from a single country). In this work we introduce WAFFLE a novel multimodal floorplan understanding dataset of nearly 20K floorplan images and metadata curated from Internet data spanning diverse building types locations and data formats. By using a large language model and multimodal foundation models we curate and extract semantic information from these images and their accompanying noisy metadata. We show that WAFFLE enables progress on new building understanding tasks both discriminative and generative which were not feasible using prior datasets. We will publicly release WAFFLE along with our code and trained models providing the research community with a new foundation for learning the semantics of buildings.

Volumetric shape representations have become ubiquitous in multi-view reconstruction tasks. They often build on regular voxel grids as discrete representations of 3D shape functions such as SDF or radiance fields either as the full shape model or as sampled instantiations of continuous representations as with neural networks. Despite their proven efficiency voxel representations come with the precision versus complexity trade-off. This inherent limitation can significantly impact performance when moving away from simple and uncluttered scenes. In this paper we investigate an alternative discretization strategy with the Centroidal Voronoi Tessellation (CVT). CVTs allow to better partition the observation space with respect to shape occupancy and to focus the discretization around shape surfaces. To leverage this discretization strategy for multi-view reconstruction we introduce a volumetric optimization framework that combines explicit SDF fields with a shallow color network in order to estimate 3D shape properties over tetrahedral grids. Experimental results with Chamfer statistics validate this approach with unprecedented reconstruction quality on various scenarios such as objects open scenes or human.