WACV 2025 论文列表

Foundation models have marked a significant stride toward addressing generalization challenges in deep learning. While the Segment Anything Model (SAM) has established a strong foothold in image segmentation existing video segmentation methods still require extensive mask labeling for fine-tuning or face performance drops on unseen data domains otherwise. In this paper we show how foundation models for image segmentation make a step toward enhancing domain generalizability in video segmentation. We discover that combined with long-term point tracking image segmentation models yield state-of-the-art results in zero-shot video segmentation across multiple benchmarks. Surprisingly point trackers exhibit generalization to domains beyond their synthetic pre-training sequences which we attribute to the trackers' ability to harness the rich local information in the vicinity of each tracked point. Thus we introduce SAM-PT an innovative method for point-centric video segmentation leveraging the capabilities of SAM alongside long-term point tracking. SAM-PT extends SAM's capability to tracking and segmenting anything in dynamic videos. Unlike traditional video segmentation methods that focus on object-centric mask propagation our approach uniquely exploits point propagation to utilize local structure information independent of object semantics. The effectiveness of point-based tracking is underscored by direct evaluation on the zero-shot open-world UVO benchmark. Our experiments on popular video object segmentation and multi-object segmentation tracking benchmarks including DAVIS YouTube-VOS and BDD100K suggest that a point-based segmentation tracker yields better zero-shot performance and efficient interactions. We release our code at https://github.com/SysCV/sam-pt.

Semantic segmentation is essential for analyzing high-definition remote sensing images (HRSIs) because it allows the precise classification of objects and regions at the pixel level. However remote sensing data present challenges owing to geographical location weather and environmental variations making it difficult for semantic segmentation models to generalize across diverse scenarios. Existing methods are often limited to specific data domains and require expert annotators and specialized equipment for semantic labeling. In this study we propose a novel unsupervised domain adaptation technique for remote sensing semantic segmentation by utilizing geographical coordinates that are readily accessible in remote sensing setups as metadata in a dataset. To bridge the domain gap we propose a novel approach that considers the combination of an image's location-encoding trait and the spherical nature of Earth's surface. Our proposed SegDesicNet module regresses the GRID positional encoding of the geocoordinates projected over the unit sphere to obtain the domain loss. Our experimental results demonstrate that the proposed SegDesicNet outperforms state-of-the-art domain adaptation methods in remote sensing image segmentation achieving an improvement of approximately 6% in the mean intersection over union (MIoU) with a 27% drop in parameter count on benchmarked subsets of the publicly available FLAIR #1 dataset. We also benchmarked our method performance on the custom split of the ISPRS Potsdam dataset. Our algorithm seeks to reduce the modeling disparity between artificial neural networks and human comprehension of the physical world making the technology more human-centric and scalable.

This paper addresses the problem of image annotation for segmentation tasks. Semantic segmentation involves labeling each pixel in an image with predefined categories such as sky cars roads and humans. Deep learning models require numerous annotated images for effective training but manual annotation is slow and time-consuming. To mitigate this challenge we leverage the Segment Anything Model (SAM)- a vision foundation model. We introduce SegBuilder a framework that incorporates SAM to automatically generate segments which are then tagged by human annotators using a quick selection list. To demonstrate SegBuilder's effectiveness we introduced a novel dataset for image segmentation in underwater environments featuring animals such as sea lions beavers and jellyfish. Experiments on this dataset showed that SegBuilder significantly speeds up the annotation process compared to the publicly available tool Label Studio. SegBuilder also includes a free-form drawing tool allowing users to create correct segments missed by SAM. This feature is particularly useful for scenes with shadows camouflaged objects and part-based segmentation tasks where SAM falls short. Experimentally we demonstrated SegBuilder's efficacy in these scenarios showcasing its potential for generating pixel-wise annotations crucial for training robust deep learning models for semantic segmentation.

Understanding what makes a video memorable has important applications in advertising and education technology. Towards this goal we investigate spatio-temporal attention mechanisms underlying video memorability. Different from previous works that fuse multiple features we adopt a simple CNN+Transformer architecture that enables analysis of spatio-temporal attention while matching state-of-the-art (SoTA) performance on video memorability prediction. We compare model attention against human gaze fixations collected through a small-scale eye-tracking study where humans perform the video memory task. We uncover the following insights: (i) Quantitative saliency metrics show that our model trained only to predict a memorability score exhibits similar spatial attention patterns to human gaze especially for more memorable videos. (ii) The model assigns greater importance to initial frames in a video mimicking human attention patterns. (iii) Panoptic segmentation reveals that both (model and humans) assign a greater share of attention to things and less attention to stuff as compared to their occurrence probability.

Event cameras capture the motion of intensity gradients (edges) in the image plane in the form of rapid asynchronous events. When accumulated in 2D histograms these events depict overlays of the edges in motion consequently obscuring the spatial structure of the generating edges. Contrast maximization (CM) is an optimization framework that can reverse this effect and produce sharp spatial structures that resemble the moving intensity gradients by estimating the motion trajectories of the events. Nonetheless CM is still an underexplored area of research with avenues for improvement. In this paper we propose a novel hybrid approach that extends CM from uni-modal (events only) to bi-modal (events and edges). We leverage the underpinning concept that given a reference time optimally warped events produce sharp gradients consistent with the moving edge at that time. Specifically we formalize a correlation-based objective to aid CM and provide key insights into the incorporation of multiscale and multireference techniques. Moreover our edge-informed CM method yields superior sharpness scores and establishes new state-of-the-art event optical flow benchmarks on the MVSEC DSEC and ECD datasets.

Implicit Neural Representations (INRs) have recently advanced the field of deep learning due to their ability to learn continuous representations of signals without the need for large training datasets. Although INR methods have been studied for medical image super-resolution their adaptability to localized priors in medical images has not been extensively explored. Medical images contain rich anatomical divisions that could provide valuable local prior information to enhance the accuracy and robustness of INRs. In this work we propose a novel framework referred to as the Semantically Conditioned INR (SeCo-INR) that conditions an INR using local priors from a medical image enabling accurate model fitting and interpolation capabilities to achieve super-resolution. Our framework learns a continuous representation of the semantic segmentation features of a medical image and utilizes it to derive the optimal INR for each semantic region of the image. We tested our framework using several medical imaging modalities and achieved higher quantitative scores and more realistic super-resolution outputs compared to state-of-the-art methods.

This paper introduces Scene-LLM a 3D-visual-language model that enhances embodied agents' abilities in interactive 3D indoor environments by integrating the reasoning strengths of Large Language Models (LLMs). Scene-LLM adopts a unified 3D visual feature representation that incorporates dense spatial information and supports scene state updates. The model employs a projection layer to efficiently project these features in the pre-trained textual embedding space enabling effective interpretation of 3D visual information. Unique to our approach is the integration of both scene-level and egocentric 3D information with a compact hybrid representation. This combination is pivotal for interactive planning where scene-level data supports global planning and egocentric data is important for localization. Notably we use egocentric 3D frame features for feature alignment an efficient technique that incorporates the model with fine-grained concepts. Our experiments with Scene-LLM demonstrate its strong capabilities in scene captioning question answering and interactive planning. We believe Scene-LLM advances the field of 3D visual understanding and reasoning offering new possibilities for sophisticated agent interactions in indoor settings.

Recent advancements in action recognition leverage both skeleton and video modalities to achieve state-of-the-art performance. However due to the challenges of early fusion which tends to underutilize the strengths of each modality existing methods often resort to late fusion consequently leading to more complex designs. Additionally self-supervised learning approaches utilizing both modalities remain underexplored. In this paper we introduce SV-data2vec a novel self-supervised framework for learning from skeleton and video data. Our approach employs a student-teacher architecture where the teacher network generates contextualized targets based on skeleton data. The student network performs a masked prediction task using both skeleton and visual data. Remarkably after pretraining with both modalities our method allows for fine-tuning with RGB data alone achieving results on par with multimodal approaches by effectively learning video representations through skeleton data guidance. Extensive experiments on benchmark datasets NTU RGB+D 60 NTU RGB+D 120 and Toyota Smarthome confirm that our method outperforms existing RGB based state-of-the-art techniques. The code is available at github.com/zoranadozdor/SVdata2vec.

Visual attention modeling important for interpreting and prioritizing visual stimuli plays a significant role in applications such as marketing multimedia and robotics. Traditional saliency prediction models especially those based on Convolutional Neural Networks (CNNs) or Transformers achieve notable success by leveraging large-scale annotated datasets. However the current state-of-the-art (SOTA) models that use Transformers are computationally expensive. Additionally separate models are often required for each image type lacking a unified approach. In this paper we propose Saliency Unification through Mamba (SUM) a novel approach that integrates the efficient long-range dependency modeling of Mamba with U-Net to provide a unified model for diverse image types. Using a novel Conditional Visual State Space (C-VSS) block SUM dynamically adapts to various image types including natural scenes web pages and commercial imagery ensuring universal applicability across different data types. Our comprehensive evaluations across five benchmarks demonstrate that SUM seamlessly adapts to different visual characteristics and consistently outperforms existing models. These results position SUM as a versatile and powerful tool for advancing visual attention modeling offering a robust solution universally applicable across different types of visual content. Our codebase and pretrained models are publicly accessible on the https://arhosseini77.github.io/sum_page/.

The capability to accurately estimate 3D human poses is crucial for diverse fields such as action recognition gait recognition and virtual/augmented reality. However a persistent and significant challenge within this field is the accurate prediction of human poses under conditions of severe occlusion. Traditional image-based estimators struggle with heavy occlusions due to a lack of temporal context resulting in inconsistent predictions. While video-based models benefit from processing temporal data they encounter limitations when faced with prolonged occlusions that extend over multiple frames. This challenge arises because these models struggle to generalize beyond their training datasets and the variety of occlusions is hard to capture in the training data. Addressing these challenges we propose STRIDE (Single-video based TempoRally contInuous occlusion Robust 3D Pose Estimation) a novel Test-Time Training (TTT) approach to fit a human motion prior for each video. This approach specifically handles occlusions that were not encountered during the model's training. By employing STRIDE we can refine a sequence of noisy initial pose estimates into accurate temporally coherent poses during test time effectively overcoming the limitations of prior methods. Our framework demonstrates flexibility by being model-agnostic allowing us to use any off-the-shelf 3D pose estimation method for improving robustness and temporal consistency. We validate STRIDE's efficacy through comprehensive experiments on challenging datasets like Occluded Human3.6M Human3.6M and OCMotion where it not only outperforms existing single-image and video-based pose estimation models but also showcases superior handling of substantial occlusions achieving fast robust accurate and temporally consistent 3D pose estimates. Code is made publicly available at https://github.com/take2rohit/stride

Spatio-Temporal predictive Learning is a self-supervised learning paradigm that enables models to identify spatial and temporal patterns by predicting future frames based on past frames. Traditional methods which use recurrent neural networks to capture temporal patterns have proven their effectiveness but come with high system complexity and computational demand. Convolutions could offer a more efficient alternative but are limited by their characteristic of treating all previous frames equally resulting in poor temporal characterization and by their local receptive field limiting the capacity to capture distant correlations among frames. In this paper we propose STLight a novel method for spatio-temporal learning that relies solely on channel-wise and depth-wise convolutions as learnable layers. STLight overcomes the limitations of traditional convolutional approaches by rearranging spatial and temporal dimensions together using a single convolution to mix both types of features into a comprehensive spatio-temporal patch representation. This representation is then processed in a purely convolutional framework capable of focusing simultaneously on the interaction among near and distant patches and subsequently allowing for efficient reconstruction of the predicted frames. Our architecture achieves state-of-the-art performance on STL benchmarks across different datasets and settings while significantly improving computational efficiency in terms of parameters and computational FLOPs. The code is publicly available

In layout-to-image (L2I) synthesis controlled complex scenes are generated from coarse information like bounding boxes. Such a task is exciting to many downstream applications because the input layouts offer strong guidance to the generation process while remaining easily reconfigurable by humans. In this paper we proposed STyled LAYout Diffusion (STAY Diffusion) a diffusion-based model that produces photo-realistic images and provides fine-grained control of stylized objects in scenes. Our approach learns a global condition for each layout and a self-supervised semantic map for weight modulation using a novel Edge-Aware Normalization (EA Norm). A new Styled-Mask Attention (SM Attention) is also introduced to cross-condition the global condition and image feature for capturing the objects' relationships. These measures provide consistent guidance through the model enabling more accurate and controllable image generation. Extensive benchmarking demonstrates that our STAY Diffusion presents high-quality images while surpassing previous state-of-the-art methods in generation diversity accuracy and controllability.

In this paper we propose SPACE a novel anomaly detection methodology that integrates a Feature Encoder (FE) into the structure of the Student-Teacher method. The proposed method has two key elements: Spatial Consistency regularization Loss (SCL) and Feature converter Module (FM). SCL prevents overfitting in student models by avoiding excessive imitation of the teacher model. Simultaneously it facilitates the expansion of normal data features by steering clear of abnormal areas generated through data augmentation. This dual functionality ensures a robust boundary between normal and abnormal data. The FM prevents the learning of ambiguous information from the FE. This protects the learned features and enables more effective detection of structural and logical anomalies. Through these elements SPACE is available to minimize the influence of the FE while integrating various data augmentations. In this study we evaluated the proposed method on the MVTec LOCO MVTec AD and VisA datasets. Experimental results through qualitative evaluation demonstrate the superiority of detection and efficiency of each module compared to state-of-the-art methods.

Adapting large-scale pre-trained generative models in a parameter-efficient manner is gaining traction. Traditional methods like low rank adaptation achieve parameter efficiency by imposing constraints but may not be optimal for tasks requiring high representation capacity. We propose a novel spectrum-aware adaptation framework for generative models. Our method adjusts both singular values and their basis vectors of pretrained weights. Using the Kronecker product and efficient Stiefel optimizers we achieve parameter-efficient adaptation of orthogonal matrices. Specifically we introduce Spectral Orthogonal Decomposition Adaptation (SODA) which balances computational efficiency and representation capacity. Extensive evaluations on text-to-image diffusion models demonstrate SODA's effectiveness offering a spectrum-aware alternative to existing fine-tuning methods.

Ensuring safe and independent navigation poses considerable difficulties for individuals who are blind or have low vision (BLV) as it requires detailed knowledge of their immediate environment. Our research highlights the critical need for accessible data on sidewalk materials and objects which is currently lacking in existing map services. To bridge this gap we present the Sidewalk Material Data Acquisition Framework (SMDAF) designed for large-scale data collection. This framework includes (1) a lightweight data collection system embedded in a white cane which captures audio data through the interaction of the cane tip with the sidewalk surface and a mobile app that facilitates data storage and management resulting in a novel multimodal dataset comprising both image and audio data; and (2) a unique Cross-Modal Knowledge Distillation (CMKD) technique for an enhanced audio material classifier. Our CMKD approach employs an image-based model as the teacher to improve the audio model incorporating an Enhanced Bidirectional learning method with an intuitive filtering technique: Bidirectional Correct Sample Filtering (BCSF). BCSF filters correct samples to prevent the distillation of incorrect knowledge addressing the issue of inaccurate cross-modal learning. This novel approach has resulted in a 1.84% improvement in Macro Accuracy achieving an overall accuracy of 87.62% surpassing all state-of-the-art KD and CMKD methods. This study underscores the efficacy of SMDAF and provides a practical CMKD technique for future cross-modal learning tasks. Code and dataset are available https://github.com/FgSurewin/SMDAF-CMKD.

As a solution for the lost-in-space star identification problem we present Star Identification using Graph Neural Network (SIGNN) a novel approach using Graph Attention Networks. By representing the celestial sphere as a graph data structure created from the ESA's Hipparcos catalogue we are able to accurately capture the rich information and relationships within local star fields. Graph learning techniques allow our model to aggregate information and learn the relative importance of the nodes and structure within each stars local neighbourhood to it's identification. This approach combined with our parametric data-generation and noise simulation allows us to train a highly robust model capable of accurate star identification even under intensive noise outperforming existing methods. Code and generation techniques will be available on https://github.com/FloydHepburn/SIGNN.

Partial label learning (PLL) aims to train classification models under conditions where each training sample is associated with a candidate set of labels. This set contains multiple labels among which only one is correct. This work addresses instance-dependent noise in PLL by leveraging hierarchical structures within the label space. We introduce a method to derive label hierarchies from instance-dependent partial labels. Subsequently we propose Structural Hierarchies for Instance-dependent Partial label (SHIP). SHIP is a modular component that integrates into deep learning architectures with applications that have intrinsic hierarchies. SHIP harnesses label hierarchy to enhance instance-dependent PLL performance across various deep-learning algorithms with hierarchy in the dataset. We conduct experiments on five publicly available benchmark datasets with four recent PLL algorithms. Experimental results show that incorporating SHIP into state-of-the-art architectures yields up to a 2.6% improvement in accuracy when hierarchies are present in the data. Moreover when the number of classes is high SHIP achieves up to a 2.5% reduction in mean mistake severity highlighting its effectiveness in mitigating error severity.

Novel View Synthesis (NVS) for street scenes plays a critical role in the autonomous driving simulation. Current mainstream methods such as Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS) struggle to maintain rendering quality at the viewpoint that deviates significantly from the training viewpoints. This issue stems from the sparse training views captured by a fixed camera on a moving vehicle. To tackle this problem we propose a novel approach that enhances the capacity of 3DGS by leveraging prior from a Diffusion Model along with complementary multi-modal data. Specifically we first fine-tune a Diffusion Model by adding images from adjacent frames as condition meanwhile exploiting depth data from LiDAR point clouds to supply additional spatial information. Then we apply the fine-tuned Diffusion Model to regularize the 3DGS at unseen views during training. Experimental results validate the effectiveness of our method compared with current state-of-the-art models and demonstrate its advances in rendering images from broader views.

Integrated silicon photonic devices which manipulate light to transmit and process information on a silicon-on-insulator chip are highly sensitive to structural variations. Minor deviations during nanofabrication--the precise process of building structures at the nanometer scale--such as over- or under-etching corner rounding and unintended defects can significantly impact performance. To address these challenges we introduce SEMU-Net a comprehensive set of methods that automatically segments scanning electron microscope (SEM) images and uses them to train two deep neural network models based on U-Net and its variants. The predictor model anticipates fabrication-induced variations while the corrector model adjusts the design to address these issues ensuring that the final fabricated structures closely align with the intended specifications. Experimental results show that the segmentation U-Net reaches an average IoU score of 99.30% while the corrector attention U-Net in a tandem architecture achieves an average IoU score of 98.67%.