WACV 2025 论文列表

Deep learning algorithms have depicted commendable performance in a variety of computer vision applications. However training a robust deep neural network necessitates a large amount of labeled training data which is time-consuming and labor-intensive to acquire. This problem is even more serious for an application like image segmentation as the human oracle has to hand-annotate each and every pixel in a given training image which is extremely laborious. Active learning algorithms automatically identify the salient and exemplar samples from large amounts of unlabeled data and tremendously reduce human annotation effort in inducing a machine learning model. In this paper we propose a novel active learning algorithm for image segmentation with the goal of further reducing the labeling burden on the human oracles. Our framework identifies a batch of informative images together with a list of semantic classes for each and the human annotator merely needs to answer whether a given semantic class is present or absent in a given image. To the best of our knowledge this is the first research effort to develop an active learning framework for image segmentation which poses only binary (yes/no) queries to the users. We pose the image and class selection as a constrained optimization problem and derive a linear programming relaxation to select a batch of (image-class) pairs which are maximally informative to the underlying deep neural network. Our extensive empirical studies on three challenging datasets corroborate the potential of our method in substantially reducing human annotation effort for real-world image segmentation applications.

Event cameras provide a natural and data efficient representation of visual information motivating novel computational strategies towards extracting visual information. Inspired by the biological vision system we propose a behavior driven approach for object-wise distance estimation from event camera data. This behavior-driven method mimics how biological systems like the human eye stabilize their view based on object distance: distant objects require minimal compensatory rotation to stay in focus while nearby objects demand greater adjustments to maintain alignment. This adaptive strategy leverages natural stabilization behaviors to estimate relative distances effectively. Unlike traditional vision algorithms that estimate depth across the entire image our approach targets local depth estimation within a specific region of interest. By aligning events within a small region we estimate the angular velocity required to stabilize the image motion. We demonstrate that under certain assumptions the compensatory rotational flow is inversely proportional to the object's distance. The proposed approach achieves new state-of-the-art accuracy in distance estimation - a performance gain of 16% on EVIMO2. EVIMO2 event sequences comprise complex camera motion and substantial variance in depth of static real world scenes. Code: https://github.com/pbideau/AAEDepth.

We present ActionDiffusion - a novel diffusion model for procedure planning in instructional videos that is the first to take temporal inter-dependencies between actions into account. Our approach is in stark contrast to existing methods that fail to exploit the rich information content available in the particular order in which actions are performed. Our method unifies the learning of temporal dependencies between actions and denoising of the action plan in the diffusion process by projecting the action information into the noise space. This is achieved 1) by adding action embeddings in the noise masks in the noise-adding phase and 2) by introducing an attention mechanism in the noise prediction network to learn the correlations between different action steps. We report extensive experiments on three instructional video benchmark datasets (CrossTask Coin and NIV) and show that our method outperforms previous state-of-the-art methods on all metrics on CrossTask and NIV and all metrics except accuracy on Coin dataset. We show that by adding action embeddings into the noise mask the diffusion model can better learn action temporal dependencies and increase the performances on procedure planning. Codes are available at https://www.collaborative-ai.org/publications/shi25_wacv/

Federated Learning (FL) enables multiple clients to train a model collaboratively without sharing their local data. Yet the FL system is vulnerable to well-designed Byzantine attacks which aim to disrupt the model training process by uploading malicious model updates. Existing robust aggregation rule-based defense methods overlook the diversity of magnitude and direction across different layers of the model updates resulting in limited robustness performance particularly in non-IID settings. To address these challenges we propose the Layer-Adaptive Sparsified Model Aggregation (LASA) approach which combines pre-aggregation sparsification with layer-wise adaptive aggregation to improve robustness. Specifically LASA includes a pre-aggregation sparsification module that sparsifies updates from each client before aggregation reducing the impact of malicious parameters and minimizing the interference from less important parameters for the subsequent filtering process. Based on sparsified updates a layer-wise adaptive filter then adaptively selects benign layers using both magnitude and direction metrics across all clients for aggregation. We provide a detailed theoretical robustness analysis of LASA and a resilience analysis of the FL integrated with LASA. Extensive experiments are conducted on various IID and non-IID datasets. The numerical results demonstrate the effectiveness of LASA. Code is available at https://github.com/JiiahaoXU/LASA.

Accurately detecting and classifying damage in analogue media such as paintings photographs textiles mosaics and frescoes is essential for cultural heritage preservation. While machine learning models excel in correcting degradation if the damage operator is known a priori we show that they fail to robustly predict where the damage is even after supervised training; thus reliable damage detection remains a challenge. Motivated by this we introduce ARTeFACT a dataset for damage detection in diverse types analogue media with over 11000 annotations covering 15 kinds of damage across various subjects media and historical provenance. Furthermore we contribute human-verified text prompts describing the semantic contents of the images and derive additional textual descriptions of the annotated damage. We evaluate CNN Transformer diffusion-based segmentation models and foundation vision models in zero-shot supervised unsupervised and text-guided settings revealing their limitations in generalising across media types. Our dataset is available at https://daniela997.github.io/ARTeFACT/ as the first-of-its kind benchmark for analogue media damage detection and restoration.

Diffusion models have demonstrated exceptional capabilities in generating a broad spectrum of visual content yet their proficiency in rendering text is still limited: they often generate inaccurate characters or words that fail to blend well with the underlying image. To address these shortcomings we introduce a novel framework named ARTIST which incorporates a dedicated textual diffusion model to focus on the learning of text structures specifically. Initially we pretrain this textual model to capture the intricacies of text representation. Subsequently we finetune a visual diffusion model enabling it to assimilate textual structure information from the pretrained textual model. This disentangled architecture design and training strategy significantly enhance the text rendering ability of the diffusion models for text-rich image generation. Additionally we leverage the capabilities of pretrained large language models to interpret user intentions better contributing to improved generation quality. Empirical results on the MARIO-Eval benchmark underscore the effectiveness of the proposed method showing an improvement of up to 15% in various metrics.

The radiance fields style transfer is an emerging field that has recently gained popularity as a means of 3D scene stylization thanks to the outstanding performance of neural radiance fields in 3D reconstruction and view synthesis. We highlight a research gap in radiance fields style transfer the need for sufficient perceptual controllability motivated by the existing concept in the 2D image style transfer. In this paper we present ARF-Plus a unique 3D neural style transfer framework offering manageable control over perceptual factors to systematically explore the perceptual controllability in 3D scene stylization. Four distinct types of controls - color preservation control (style pattern) scale control spatial (selective stylization area) control and depth enhancement control - come with our proposed novel loss functions and strategies seamlessly integrated into this framework. This unlocks a realm of limitless possibilities allowing customized modifications of stylization effects and flexible merging of the strengths of different styles ultimately enabling the creation of novel and eye-catching stylistic effects on 3D scenes.

The variational autoencoder (VAE) is a popular deep latent-variable model (DLVM) due to its simple yet effective formulation for modeling the data distribution. Moreover optimizing the VAE objective function is more manageable than other DLVMs. The bottleneck dimension of the VAE is a crucial design choice and it has strong ramifications for the model's performance such as finding the hidden explanatory factors of a dataset using the representations learned by the VAE. However the size of the latent dimension of the VAE is often treated as a hyperparameter estimated empirically through trial and error. To this end we propose a statistical formulation to discover the relevant latent factors required for modeling a dataset. In this work we use a hierarchical prior in the latent space that estimates the variance of the latent axes using the encoded data which identifies the relevant latent dimensions. For this we replace the fixed prior in the VAE objective function with a hierarchical prior keeping the remainder of the formulation unchanged. We call the proposed method the automatic relevancy detection in the variational autoencoder (ARD-VAE). We demonstrate the efficacy of the ARD-VAE on multiple benchmark datasets in finding the relevant latent dimensions and their effect on different evaluation metrics such as FID score and disentanglement analysis.

We introduce the novel task of Crowd Volume Estimation (CVE) defined as the process of estimating the collective body volume of crowds using only RGB images. Besides event management and public safety CVE can be instrumental in approximating body weight unlocking weight-sensitive applications such as infrastructure stress assessment and assuring even weight balance. We propose the first benchmark for CVE comprising ANTHROPOS-V a synthetic photorealistic video dataset featuring crowds in diverse urban environments. Its annotations include each person's volume SMPL shape parameters and keypoints. Also we explore metrics pertinent to CVE define baseline models adapted from Human Mesh Recovery and Crowd Counting domains and propose a CVE-specific methodology that surpasses baselines. Although synthetic the weights and heights of individuals are aligned with the real-world population distribution across genders and they transfer to the downstream task of CVE from real images. Benchmark and code are available at github.com/colloroneluca/Crowd-Volume-Estimation.

Vision transformers (ViTs) have revolutionized computer vision but face significant challenges due to their high computational and memory demands. Existing post-training quantization methods struggle to maintain performance at low bit-widths due to activation asymmetry and reliance on manual configurations. To overcome these challenges we introduce SymAlign to address activation asymmetry and reduce clamping loss. Additionally we propose AutoScale an automatic and data-driven mechanism that adapts to variant activations. We incorporate the above-mentioned techniques and propose an adaptive mixed-precision post-training quantization framework for vision transformers (AMP-ViT). Our comprehensive approach addresses asymmetry variant distribution and uneven sensitivities making it the first to tackle these challenges thoroughly. Our experiments on ViT DeiT and Swin demonstrate significant accuracy improvements compared with SOTA on the ImageNet dataset. Specifically our proposed methods achieve accuracy improvements ranging from 0.90% to 23.35% on 4-bit ViTs with single-precision and from 3.82% to 78.14% on 5-bit fully quantized ViTs with mixed-precision.

Surgical instrument segmentation (SIS) is pivotal for robotic-assisted minimally invasive surgery assisting surgeons by identifying surgical instruments in endoscopic video frames. Recent unsupervised surgical instrument segmentation (USIS) methods primarily rely on pseudo-labels derived from low-level features such as color and optical flow but these methods show limited effectiveness and generalizability in complex and unseen endoscopic scenarios. In this work we propose a label-free unsupervised model featuring a novel module named Multi-View Normalized Cutter (m-NCutter). Different from previous USIS works our model is trained using a graph-cutting loss function that leverages patch affinities for supervision eliminating the need for pseudo-labels. The framework adaptively determines which affinities from which levels should be prioritized. Therefore the low- and high-level features and their affinities are effectively integrated to train a label-free unsupervised model showing superior effectiveness and generalization ability. We conduct comprehensive experiments across multiple SIS datasets to validate our approach's state-of-the-art (SOTA) performance robustness and exceptional potential as a pre-trained model. Our code is released at https://github.com/MingyuShengSMY/AMNCutter.

We propose an online 3D semantic segmentation method that incrementally reconstructs a 3D semantic map from a stream of RGB-D frames. Unlike offline methods ours is directly applicable to scenarios with real-time constraints such as robotics or mixed reality. To overcome the inherent challenges of online methods we make two main contributions. First to effectively extract information from the input RGB-D video stream we jointly estimate geometry and semantic labels per frame in 3D. A key focus of our approach is to reason about semantic entities both in the 2D input and the local 3D domain to leverage differences in spatial context and network architectures. Our method predicts 2D features using an off-the-shelf segmentation network. The extracted 2D features are refined by a lightweight 3D network to enable reasoning about the local 3D structure. Second to efficiently deal with an infinite stream of input RGBD frames a subsequent network serves as a temporal expert predicting the incremental scene updates by leveraging 2D 3D and past information in a learned manner. These updates are then integrated into a global scene representation. Using these main contributions our method can enable scenarios with real-time constraints and can scale to arbitrary scene sizes by processing and updating the scene only in a local region defined by the new measurement. Our experiments demonstrate improved results compared to existing online methods that purely operate in local regions and show that complementary sources of information can boost the performance. We provide a thorough ablation study on the benefits of different architectural as well as algorithmic design decisions. Our method yields competitive results on the popular ScanNet benchmark and SceneNN dataset.

3D object detection plays a crucial role in various applications such as autonomous vehicles robotics and augmented reality. However training 3D detectors requires a costly precise annotation which is a hindrance to scaling annotation to large datasets. To address this challenge we propose a weakly supervised 3D annotator that relies solely on 2D bounding box annotations from images along with size priors. One major problem is that supervising a 3D detection model using only 2D boxes is not reliable due to ambiguities between different 3D poses and their identical 2D projection. We introduce a simple yet effective and generic solution: we build 3D proxy objects with annotations by construction and add them to the training dataset. Our method requires only size priors to adapt to new classes. To better align 2D supervision with 3D detection our method ensures depth invariance with a novel expression of the 2D losses. Finally to detect more challenging instances our annotator follows an offline pseudo-labelling scheme which gradually improves its 3D pseudo-labels. Extensive experiments on the KITTI dataset demonstrate that our method not only performs on-par or above previous works on the Car category but also achieves performance close to fully supervised methods on more challenging classes. We further demonstrate the effectiveness and robustness of our method by being the first to experiment on the more challenging nuScenes dataset. We additionally propose a setting where weak labels are obtained from a 2D detector pre-trained on MS-COCO instead of human annotations. The code is available at https://github.com/CEA-LIST/ALPI

3D open-vocabulary semantic segmentation aims at recognizing countless categories beyond the limited set of annotations used in traditional settings. Due to the lack of large-scale 3D-vision-language segmentation data instead of training models from scratch the current solutions distill knowledge from pre-trained 2D vision-language models (VLMs) into 3D models. However this distillation is supervised by misaligned 3D-scene-image-to-text data pairs consequently leading to suboptimal performance. Moreover as 2D VLMs are trained on 2D datasets text encoders of VLMs which serve as the bridge between 3D models and an unbounded set of categories lack 3D semantics. In this paper to address these issues and improve generalization performance we propose an AlIgned 3D Open-Vocabulary SEmantic Segmentation framework called AIDE with two novel modules. To collect high-quality and well-aligned 3D-scene-image-to-text pairs our CLIP-rewarded alignment module (i) generates diverse captions of multi-view images of 3D scenes to capture details by varying the temperatures and then (ii) samples captions based on their similarity to corresponding images for rich and accurate associations. Next to adapt 2D VLMs to 3D contexts our adaptive segmentation module introduces (iii) trainable tokens within the input space and each layer of the text encoder while freezing the text encoder to avoid catastrophic forgetting. Extensive experiments show that AIDE outperforms previous methods by a large margin on three representative benchmarks demonstrating its effectiveness.

Over the recent years there has been a growing interest in class-incremental 3D object detection based on point clouds. However the current state-of-the-art (SOTA) methods still fall short of practical adoption mainly due to two key observations. Firstly existing SOTA methods are limited by the capability of feature representation from the object detection model. Secondly these methods overlook the importance of incorporating prior information or geometry constraints which are crucial elements for 3D point cloud tasks. In this study we strive to enhance the performance of class-incremental 3D object detection for indoor scenes by proposing AIC3DOD - Advancing Indoor Class-incremental 3D Object Detection using the point transformer architecture with room layout constraints. Our approach employs a transformer architecture in our detection model and optimizes the class incremental step in the transformer architecture. Besides AIC3DOD incorporates additional prior information namely room layout to impose physical constraints on detected objects thereby enhancing overall object detection performance. Extensive experimental results on the ScanNet dataset demonstrate the effectiveness of our approach showcasing our superior performance compared to other SOTA methods in the class-incremental 3D object detection task.

Open compound domain adaptation (OCDA) is a practical domain adaptation problem that consists of a source domain target compound domain and unseen open domain. In this problem the absence of domain labels and pixel-level segmentation labels for both compound and open domains poses challenges to the direct application of existing domain adaptation and generalization methods. To address this issue we propose Amplitude-based curriculum learning and a Hopfield segmentation model for Open Compound Domain Adaptation (AH-OCDA). Our method comprises two complementary components: 1) amplitude-based curriculum learning and 2) Hopfield segmentation model. Without prior knowledge of target domains within the compound domains amplitude-based curriculum learning gradually induces the semantic segmentation model to adapt from the near-source compound domain to the far-source compound domain by ranking unlabeled compound domain images through Fast Fourier Transform (FFT). Additionally the Hopfield segmentation model maps segmentation feature distributions from arbitrary domains to the feature distributions of the source domain. AH-OCDA achieves state-of-the-art performance on two OCDA benchmarks and extended open domains demonstrating its adaptability to continuously changing compound domains and unseen open domains.

Medical images acquired from standardized protocols show consistent macroscopic or microscopic anatomical structures and these structures consist of composable/decomposable organs and tissues but existing self-supervised learning (SSL) methods do not appreciate such composable/decomposable structure attributes inherent to medical images. To overcome this limitation this paper introduces a novel SSL approach called ACE to learn anatomically consistent embedding via composition and decomposition with two key branches: (1) global consistency capturing discriminative macro-structures via extracting global features; (2) local consistency learning fine-grained anatomical details from composable/decomposable patch features via corresponding matrix matching. Experimental results across 6 datasets and 2 backbones evaluated in few-shot learning fine-tuning and property analysis show ACE's superior robustness transferability and clinical potential. The innovations of our ACE lie in grid-wise image cropping leveraging the intrinsic properties of compositionality and decompositionality of medical images bridging the semantic gap from high-level pathologies to low-level tissue anomalies and providing a new SSL method for medical imaging.

Vision-language models (VLMs) are capable of recognizing unseen actions. However existing VLMs lack intrinsic understanding of procedural action concepts. Hence they overfit to fixed labels and are not invariant to unseen action synonyms. To address this we propose a simple fine-tuning technique Action Concept Enhancement (ACE) to improve the robustness and concept understanding of VLMs in procedural action classification. ACE continually incorporates augmented action synonyms and negatives in an auxiliary classification loss by stochastically replacing fixed labels during training. This creates new combinations of action labels over the course of fine-tuning and prevents overfitting to fixed action representations. We show the enhanced concept understanding of our VLM by visualizing the alignment of encoded embeddings of unseen action synonyms in the embedding space. Our experiments on the ATA IKEA and GTEA datasets demonstrate the efficacy of ACE in domains of cooking and assembly leading to significant improvements in zero-shot action classification while maintaining competitive performance on seen actions.

Computed tomography (CT) reconstruction plays a crucial role in industrial nondestructive testing and medical diagnosis. Sparse view CT reconstruction aims to reconstruct high-quality CT images while only using a small number of projections which helps to improve the detection speed of industrial assembly lines and is also meaningful for reducing radiation in medical scenarios. Sparse CT reconstruction methods based on implicit neural representations (INRs) have recently shown promising performance but still produce artifacts because of the difficulty of obtaining useful prior information. In this work we incorporate a powerful prior: the total number of material categories of objects. To utilize the prior we design AC-IND a self-supervised method based on Attenuation Coefficient Estimation and Implicit Neural Distribution. Specifically our method first transforms the traditional INR from scalar mapping to probability distribution mapping. Then we design a compact attenuation coefficient estimator initialized with values from a rough reconstruction and fast segmentation. Finally our algorithm finishes the CT reconstruction by jointly optimizing the estimator and the generated distribution. Through experiments we find that our method not only outperforms the comparative methods in sparse CT reconstruction but also can automatically generate semantic segmentation maps. Code is available at https://github.com/AIIAAI/AC-IND.