WACV 2025 论文列表

Recent advancements in text-guided 3D object generation using diffusion priors struggle with illumination inconsistencies when applied to scene editing tasks like object replacement or insertion. To address this we propose an illumination-aware 3D scene editing pipeline for 3D Gaussian Splatting (3DGS). Our method leverages state-of-the-art 2D diffusion inpainting to handle global illumination context effectively. Specifically we identify representative anchor views that capture scene-wide illumination inpaint them using 2D diffusion models and integrate the results into a coarse-to-fine 3DGS optimization process. In the fine step we introduce Depth-guided Inpainting Score Distillation Sampling (DI-SDS) to refine geometry and texture details capitalizing on the diversity of 2D priors. Our approach achieves locally precise edits with globally consistent illumination demonstrating robustness in real scenes with highlights and shadows. Comparisons show superior results over state-of-the-art text-to-3D editing methods. Project page: https://corneliushsiao.github.io/GSLE.html.

Self-supervised learning for computer vision has progressed tremendously and improved many downstream vision tasks such as image classification semantic segmentation and object detection. Among these generative self-supervised vision learning approaches such as MAE and BEiT show promising performance. However their global reconstruction mechanism is computationally demanding especially for high-resolution images. The computational cost increases extensively when scaled to a large-scale dataset. To address this issue we propose local masked reconstruction (LoMaR) a simple yet effective approach that reconstructs image patches from small neighboring regions. The strategy can be easily integrated into any generative self-supervised learning techniques and improves the trade-off between efficiency and accuracy compared to reconstruction over the entire image. LoMaR is 2.5x faster than MAE and 5.0x faster than BEiT on 384x384 ImageNet pretraining and surpasses them by 0.2% and 0.8% in accuracy respectively. It is 2.1x faster than MAE on iNaturalist pretraining and gains 0.2% in accuracy. On MS COCO LoMaR outperforms MAE by 0.5 APbox on object detection and 0.5 APmask on instance segmentation. It also outperforms MAE by 0.2% on semantic segmentation. Our code and pretrained models are available at: https://github.com/junchen14/LoMaR.

Temporal Action Localization (TAL) involves localizing and classifying action snippets in an untrimmed video. The emergence of large video foundation models has led RGB-only video backbones to outperform previous methods needing both RGB and optical flow modalities. Leveraging these large models is often limited to training only the TAL head due to the prohibitively large GPU memory required to adapt the video backbone for TAL. To overcome this limitation we introduce LoSA the first memory-and-parameter-efficient backbone adapter designed specifically for TAL to handle untrimmed videos. LoSA specializes for TAL by introducing Long-Short-range Adapters that adapt the intermediate layers of the video backbone over different temporal ranges. These adapters run parallel to the video backbone to significantly reduce memory footprint. LoSA also includes Long-Short-range Gated Fusion that strategically combines the output of these adapters from the video backbone layers to enhance the video features provided to the TAL head. Experiments show that LoSA significantly outperforms all existing methods on standard TAL benchmarks THUMOS-14 and ActivityNet-v1.3 by scaling end-to-end backbone adaptation to billion-parameter-plus models like VideoMAEv2 (ViT-g) and leveraging them beyond head-only transfer learning.

We introduce Lifting By Gaussians (LBG) a novel approach for open-world instance segmentation of 3D Gaussian Splatted Radiance Fields (3DGS). Recently 3DGS Fields have emerged as a highly efficient and explicit alternative to Neural Field-based methods for high-quality Novel View Synthesis. Our 3D instance segmentation method directly lifts 2D segmentation masks from SAM (alternately FastSAM etc.) together with CLIP and DINOv2 features directly fusing them onto 3DGS (or similar Gaussian radiance fields such as 2DGS). Unlike previous approaches LBG requires no per-scene training allowing it to operate seamlessly on any existing 3DGS reconstruction. Our approach is not only an order of magnitude faster and simpler than existing approaches; it is also highly modular enabling 3D semantic segmentation of existing 3DGS fields without requiring a specific parametrization of the 3D Gaussians. Furthermore our technique achieves superior semantic segmentation for 2D semantic novel view synthesis and 3D asset extraction results while maintaining flexibility and efficiency. We further introduce a novel approach to evaluate individually segmented 3D assets from 3D radiance field segmentation methods.

One of the current trends in robotics is to employ large language models (LLMs) to provide non-predefined commands execution and natural human-robot interaction. It is useful to have an environment map together with its language representation which can be further utilized by LLMs. Such a comprehensive scene representation enables numerous ways of interaction with the map for autonomously operating robots. In this work we present an approach that enhances incremental implicit mapping through the integration of visual-language features. Specifically we (i) propose a decoder optimization technique for implicit language maps which can be used when new objects appear on the scene and (ii) address the problem of inconsistent visual-language predictions between different viewing positions. Our experiments demonstrate the effectiveness of LiLMaps and solid improvements in performance.

Group Activity Recognition (GAR) remains challenging in computer vision due to the complex nature of multi-agent interactions. This paper introduces LiGAR a LIDAR-Guided Hierarchical Transformer for Multi-Modal Group Activity Recognition. LiGAR leverages LiDAR data as a structural backbone to guide the processing of visual and textual information enabling robust handling of occlusions and complex spatial arrangements. Our framework incorporates a Multi-Scale LIDAR Transformer Cross-Modal Guided Attention and an Adaptive Fusion Module to integrate multi-modal data at different semantic levels effectively. LiGAR's hierarchical architecture captures group activities at various granularities from individual actions to scene-level dynamics. Extensive experiments on the JRDB-PAR Volleyball and NBA datasets demonstrate LiGAR's superior performance achieving state-of-the-art results with improvements of up to 10.6% in F1-score on JRDB-PAR and 5.9% in Mean Per Class Accuracy on the NBA dataset. Notably LiGAR maintains high performance even when LiDAR data is unavailable during inference showcasing its adaptability. Our ablation studies highlight the significant contributions of each component and the effectiveness of our multi-modal multi-scale approach in advancing the field of group activity recognition.

Several methods have been proposed to estimate 3D human pose from multi-view images achieving satisfactory performance on public datasets collected under relatively simple conditions. However there are limited approaches studying extracting 3D human skeletons from multimodal inputs such as RGB and point cloud data. To address this gap we introduce LiCamPose a pipeline that integrates multi-view RGB and sparse point cloud information to estimate robust 3D human poses via single timestamp. We demonstrate the effectiveness of the volumetric architecture in combining these modalities. Furthermore to circumvent the need for manually labeled 3D human pose annotations we develop a synthetic dataset generator for pretraining and design an unsupervised domain adaptation strategy to train a 3D human pose estimator without manual annotations. To validate the generalization capability of our method LiCamPose is evaluated on four datasets including two public datasets one synthetic dataset and one challenging self-collected dataset named BasketBall covering diverse scenarios. The results demonstrate that LiCamPose exhibits great generalization performance and significant application potential. The code generator and datasets are available at https://github.com/Yu-Yy/LiCamPose.

As Vision Language Models (VLMs) become increasingly accessible to farmers and agricultural experts there is a growing need to evaluate their potential in specialized tasks. We present AgEval a comprehensive benchmark for assessing VLMs' capabilities in plant stress phenotyping offering a solution to the challenge of limited annotated data in agriculture. Our study explores how general-purpose VLMs can be leveraged for domain-specific tasks with only a few annotated examples providing insights into their behavior and adaptability. AgEval encompasses 12 diverse plant stress phenotyping tasks evaluating zero-shot and few-shot in-context learning performance of state-of-the-art models including Claude GPT Gemini and LLaVA. Our results demonstrate VLMs' rapid adaptability to specialized tasks with the best-performing model showing an increase in F1 scores from 46.24% to 73.37% in 8-shot identification. To quantify performance disparities across classes we introduce metrics such as the coefficient of variation (CV) revealing that VLMs' training impacts classes differently with CV ranging from 26.02% to 58.03%. We also find that strategic example selection enhances model reliability with exact category examples improving F1 scores by 15.38% on average. AgEval establishes a framework for assessing VLMs in agricultural applications offering valuable benchmarks for future evaluations. Our findings suggest that VLMs with minimal few-shot examples show promise as a viable alternative to traditional specialized models in plant stress phenotyping while also highlighting areas for further refinement. Results and benchmark details are available at: https://github.com/arbab-ml/AgEval

Multimodal emotion recognition (MER) aims to identify human emotions by combining data from various modalities such as language audio and vision. Despite the recent advances of MER approaches the limitations in obtaining extensive datasets impede the improvement of performance. To mitigate this issue we leverage a Contrastive Language-Image Pre-training (CLIP)-based architecture and its semantic knowledge from massive datasets that aims to enhance the discriminative multimodal representation. We propose a label encoder-guided MER framework based on CLIP (MER-CLIP) to learn emotion-related representations across modalities. Our approach introduces a label encoder that treats labels as text embeddings to incorporate their semantic information leading to the learning of more representative emotional features. To further exploit label semantics we devise a cross-modal decoder that aligns each modality to a shared embedding space by sequentially fusing modality features based on emotion-related input from the label encoder. Finally the label encoder-guided prediction enables generalization across diverse labels by embedding their semantic information as well as word labels. Experimental results show that our method outperforms the state-of-the-art MER methods on the benchmark datasets CMU-MOSI and CMU-MOSEI.

Research in 3D scene understanding particularly in autonomous driving and indoor segmentation has made significant strides. However most available datasets focus on urban settings. We introduce TS40K a 3D point cloud dataset spanning 40000 km of electrical transmission systems in rural terrain addressing power-grid inspections to prevent outages damages and fires. TS40K offers high point density and no occlusion presenting challenges like noisy labels diverse structures and sensor noise causing spurious points. We evaluate state-of-the-art methods on 3D semantic segmentation and object detection revealing limitations in power grid inspection. TS40K invites further research to tackle these challenges. Resources available in: https://github.com/dlavado/TS40K

We introduce the novel concept of visually Connecting Actions and Their Effects (CATE) in video understanding. CATE can have applications in areas like task planning and learning from demonstration. We identify and explore two different aspects of the concept of CATE: Action Selection (AS) and Effect-Affinity Assessment (EAA) where video understanding models connect actions and effects at semantic and fine-grained levels respectively. We design various baseline models for AS and EAA. Despite the intuitive nature of the task we observe that models struggle and humans outperform them by a large margin. Our experiments show that in solving AS and EAA models learn intuitive properties like object tracking and encoding pose-related features without explicit supervision. We demonstrate that CATE can be an effective self-supervised task for learning video representations from unlabeled videos. The study aims to showcase the fundamental nature and versatility of CATE with the hope of inspiring advanced formulations and models.

As vision-language models advance addressing the Zero-Shot Learning (ZSL) problem in the open world becomes increasingly crucial. Specifically a robust model must handle three types of samples during inference: seen classes with visual and semantic information provided in training unseen classes with only the semantic information in training and unknown samples with no prior information from training. Existing methods either handle seen and unseen classes together (ZSL) or seen and unknown classes (known as Open-Set Recognition OSR). However none addresses the simultaneous handling of all three which we term Open-Set Zero-Shot Learning (OZSL). To address this problem we propose a two-stage approach for OZSL that recognizes seen unseen and unknown samples. The first stage classifies samples as either seen or not while the second stage distinguishes unseen from unknown. Furthermore we introduce a cross-stage knowledge transfer mechanism that leverages semantic relationships between seen and unseen classes to enhance learning in the second stage. Extensive experiments demonstrate the efficacy of the proposed approach compared to naively combining existing ZSL and OSR methods. The code is available at https://github.com/smufang/OZSL.

Class-agnostic counting (CAC) has numerous potential applications across various domains. The goal is to count objects of an arbitrary category during testing based on only a few annotated exemplars. However existing methods often count all objects in the image including those from different categories than the exemplars. To address this issue we propose localizing the area containing the objects of interest via an exemplar-based segmentation model before counting them. To train this model we propose a novel method to obtain pseudo-labeled segmentation masks. Specifically we use an unsupervised image clustering method to generate a set of candidate pseudo object masks from which we select the optimal one using a pre-trained CAC model. We show that the trained segmentation model can effectively localize objects of interest based on the exemplars and prevent the model from counting everything. To properly evaluate the performance of CAC methods in real-world scenarios we introduce two new benchmarks: a synthetic test set and a new test set of real images containing countable objects from multiple classes. Our proposed method shows a significant advantage over previous CAC methods on these two benchmarks.

Negation is a fundamental aspect of natural language reasoning yet foundational vision-language models (VLMs) like CLIP face significant challenges in accurately interpreting it. These models often process text prompts holistically making it difficult to isolate and understand the role of negated terms. To overcome this limitation we present CC-Neg: a novel dataset consisting of 228246 images each paired with both true captions and their corresponding negated versions. CC-Neg provides a critical benchmark to assess and improve foundational VLMs' ability to process negations focusing specifically on how the presence of terms like 'not' alters the semantic relationship between images and their textual descriptions. To illustrate the effectiveness of the CC-Neg dataset in enhancing negation understanding we introduce the CoN-CLIP framework which incorporates targeted modifications to CLIP's contrastive loss function. When trained with CC-Neg CoN-CLIP achieves a 3.85% average improvement in top-1 accuracy for zero-shot image classification across eight datasets and a 4.4% performance boost on challenging compositionality benchmarks such as SugarCREPE. These results highlight CoN-CLIP's enhanced understanding of the nuanced semantic relationships involving negation. Our code and the CC-Neg benchmark are available at: https://github.com/jaisidhsingh/CoN-CLIP.

In the field of open-set recognition conventional models often focus on addressing challenges within a single hierarchical category and these methods frequently lack interpretability. In this paper we propose a novel solution that utilizes attributes and hierarchical relationships to achieve interpretable open-set recognition. Our method is centered around the visual-semantic attribute space. By leveraging hierarchy division we can decompose the attributes into more granular components thereby yielding additional performance improvements. When confronted with an unfamiliar object our method not only classifies it as an unknown category but also provides insights into the broader category and its associated attributes. This capability enhances interpretability by offering valuable information regarding the potential category and characteristics of the object. Experimental results demonstrate great performance improvements compared to existing methods.

Visual grounding tasks aim to localize image regions based on natural language references. In this work we explore whether generative VLMs predominantly trained on image-text data could be leveraged to scale up the text annotation of visual grounding data. We find that grounding knowledge already exists in generative VLM and can be elicited by proper prompting. We thus prompt a VLM to generate object-level descriptions by feeding it object regions from existing object detection datasets. We further propose attribute modeling to explicitly capture the important object attributes and spatial relation modeling to capture inter-object relationship both of which are common linguistic pattern in referring expression. Our constructed dataset (500K images 1M objects 16M referring expressions) is one of the largest grounding datasets to date and the first grounding dataset with purely model-generated queries and human-annotated objects. To verify the quality of this data we conduct zero-shot transfer experiments to the popular RefCOCO benchmarks for both referring expression comprehension (REC) and segmentation (RES) tasks. On both tasks our model significantly outperform the state-of-the-art approaches without using human annotated visual grounding data. Our results demonstrate the promise of generative VLM to scale up visual grounding in the real world.

A common practice in metric learning is to train and test an embedding model for each dataset. This dataset-specific approach fails to simulate real-world scenarios that involve multiple heterogeneous distributions of data. In this regard we explore a new metric learning paradigm called Unified Metric Learning (UML) which learns a unified distance metric capable of capturing relations across multiple data distributions. UML presents new challenges such as imbalanced data distribution and bias towards dominant distributions. These issues cause standard metric learning methods to fail in learning a unified metric. To address these challenges we propose Parameter-efficient Unified Metric leArning (PUMA) which consists of a pre-trained frozen model and two additional modules stochastic adapter and prompt pool. These modules enable to capture dataset-specific knowledge while avoiding bias towards dominant distributions. Additionally we compile a new unified metric learning benchmark with a total of 8 different datasets. PUMA outperforms the state-of-the-art dataset-specific models while using about 69 times fewer trainable parameters.

Semi-supervised medical image segmentation has shown promise in training models with limited labeled data and abundant unlabeled data. However state-of-the-art methods ignore a potentially valuable source of unsupervised semantic information--spatial registration transforms between image volumes. To address this we propose CCT-R a contrastive cross-teaching framework incorporating registration information. To leverage the semantic information available in registrations between volume pairs CCT-R incorporates two proposed modules: Registration Supervision Loss (RSL) and Registration-Enhanced Positive Sampling (REPS). The RSL leverages segmentation knowledge derived from transforms between labeled and unlabeled volume pairs providing an additional source of pseudo-labels. REPS enhances contrastive learning by identifying anatomically-corresponding positives across volumes using registration transforms. Experimental results on two challenging medical segmentation benchmarks demonstrate the effectiveness and superiority of CCT-R across various semi-supervised settings with as few as one labeled case. Our code is available at https://github.com/kathyliu579/ContrastiveCrossteachingWithRegistration.

In 3D data analysis point clouds provide detailed geometric insights for applications like computer vision and geospatial analysis. However their irregularity and diversity make classification challenging especially in domain generalization where models must generalize to new data distributions. Our research introduces a novel 3D Domain Generalization (3DDG) method using Unsupervised Part Decomposition (UPD) and Graph Structure Induction (GSI). The UPD module employs spectral clustering and a modified Shannon entropy method to segment point clouds into meaningful parts. The GSI module constructs a graph of these parts' spatial relationships processed by a Graph Neural Network (GNN) to understand complex geometries. Our approach enhances part-based analysis improving classification accuracy on the PointDA-10 and GraspNetPC-10 datasets by 1.25% and 2.6% respectively. These results highlight our advancements in 3D domain generalization enabling more robust classification models for diverse point cloud data.