WACV 2025 论文列表

The field of human motion prediction in computer vision faces challenges especially in 3D Skeleton-based Human Motion. Deep learning models albeit successful in most vision tasks were designed for data characterized by an underlying Euclidean structure which is not always fulfilled as pre-processed data may often reside in a non-linear space. Conventional RNNs struggle with capturing long-term dependencies in motion contexts. Our novel approach focuses on geometry-aware deep learning to predict the motion. We use a compact manifold-valued representation of 3D human skeleton motion integrating self-attention in transformer networks. This representation maps motions to points on a manifold ensuring smooth and coherent long-term predictions. Combining Kendall's shape space for non-rigid deformation and Lie group for rigid deformation provides a complete transformation. Experiments on various datasets demonstrate superiority over state-of-the-art methods in both short and long-term horizons.

The enduring inability of image generative models to recreate intricate geometric features such as those present in human hands and fingers has been an ongoing problem in image generation for nearly a decade. While strides have been made by increasing model sizes and diversifying training datasets this issue remains prevalent across all models from denoising diffusion models to Generative Adversarial Networks (GAN) pointing to a fundamental shortcoming in the underlying architectures. In this paper we demonstrate how this problem can be mitigated by augmenting convolution layers geometric capabilities through providing them with a single input channel incorporating the relative n-dimensional Cartesian coordinate system. We show this drastically improves quality of hand and face images generated by GANs and Variational AutoEncoders (VAE).

Diffusion models are currently one of the most effective tools in image generation. This is in particular due to the fact that contrary to GANs during training they can be easily conditioned. However given a pretrained diffusion guiding it to obtain desired result is typically a more delicate task. A typical technique based on the probabilistic derivation lies in adding the rescaled gradient of the classifier during backward propagation. In this paper we switch the perspective from probabilistic to metric. By studying the distance of the trajectory of the diffusion model from the data manifold we introduce a new guideance model GeoGuide. GeoGuide is not only easy to apply as it is based on classifier gradient normalization but it outperforms the probabilistic approach both with respect to FID and the quality of generated images.

The success of image generative models has enabled us to build methods that can edit images based on text or other user input. However these methods are bespoke imprecise require additional information or are limited to only 2D image edits. We present GeoDiffuser a zero-shot optimization-based method that unifies common 2D and 3D image-based object editing capabilities into a single method. Our key insight is to view image editing operations as geometric transformations. We show that these transformations can be directly incorporated into the attention layers in diffusion models to implicitly perform editing operations. Our training-free optimization method uses an objective function that seeks to preserve object style but generate plausible images for instance with accurate lighting and shadows. It also inpaints disoccluded parts of the image where the object was originally located. Given a natural image and user input we segment the foreground object using SAM and estimate a corresponding transform which is used by our optimization approach for editing. GeoDiffuser can perform common 2D and 3D edits like object translation 3D rotation and removal. We present quantitative results including a perceptual study that shows how our approach is better than existing methods. The success of image generative models has enabled us to build methods that can edit images based on text or other user input. However these methods are bespoke imprecise require additional information or are limited to only 2D image edits. We present GeoDiffuser a zero-shot optimization-based method that unifies common 2D and 3D image-based object editing capabilities into a single method. Our key insight is to view image editing operations as geometric transformations. We show that these transformations can be directly incorporated into the attention layers in diffusion models to implicitly perform editing operations. Our training-free optimization method uses an objective function that seeks to preserve object style but generate plausible images for instance with accurate lighting and shadows. It also inpaints disoccluded parts of the image where the object was originally located. Given a natural image and user input we segment the foreground object using SAM and estimate a corresponding transform which is used by our optimization approach for editing. GeoDiffuser can perform common 2D and 3D edits like object translation 3D rotation and removal. We present quantitative results including a perceptual study that shows how our approach is better than existing methods.

Infrared (IR) imaging is commonly used in various scenarios including autonomous driving fire safety and defense applications. Thus semantic segmentation of such images is of great interest. However this task faces several challenges including data scarcity differing contrast and input channel number compared to natural images and emergence of classes not represented in databases in certain scenarios such as defense applications. Few-shot segmentation (FSS) provides a framework to overcome these issues by segmenting query images using a few labeled support samples. However existing FSS models for IR images require paired visible RGB images which is a major limitation since acquiring such paired data is difficult or impossible in some applications. In this work we develop new strategies for FSS of IR images by using generative modeling and fusion techniques. To this end we propose to synthesize auxiliary data to provide additional channel information to complement the limited contrast in the IR images as well as IR data synthesis for data augmentation. Here the former helps the FSS model to better capture the relationship between the support and query sets while the latter addresses the issue of data scarcity. Finally to further improve the former aspect we propose a novel fusion ensemble module for integrating the two different modalities. Our methods are evaluated on different IR datasets and improve upon the state-of-the-art (SOTA) FSS models.

In this paper we address the challenge of generating realistic 3D human motions for action classes that were never seen during the training phase. Our approach involves decomposing complex actions into simpler movements specifically those observed during training by leveraging the knowledge of human motion contained in GPTs models. These simpler movements are then combined into a single realistic animation using the properties of diffusion models. Our claim is that this decomposition and subsequent recombination of simple movements can synthesize an animation that accurately represents the complex input action. This method operates during the inference phase and can be integrated with any pre-trained diffusion model enabling the synthesis of motion classes not present in the training data. We evaluate our method by dividing two benchmark human motion datasets into basic and complex actions and then compare its performance against the state-of-the-art. Our code and models are publicly available at our https://github.com/divanoLetto/MotionCompositionDiffusion.

Explaining deep learning models in a way that humans can easily understand is essential for responsible artificial intelligence applications. Attribution methods constitute an important area of explainable deep learning. The attribution problem involves finding parts of the network's input that are the most responsible for the model's output. In this work we demonstrate that implicit neural representations (INRs) constitute a good framework for generating visual explanations. Firstly we utilize coordinate-based implicit networks to reformulate and extend the extremal perturbations technique and generate attribution masks. Experimental results confirm the usefulness of our method. For instance by proper conditioning of the implicit network we obtain attribution masks that are well-behaved with respect to the imposed area constraints. Secondly we present an iterative INR-based method that can be used to generate multiple non-overlapping attribution masks for the same image. We depict that a deep learning model may associate the image label with both the appearance of the object of interest as well as with areas and textures usually accompanying the object. Our study demonstrates that implicit networks are well-suited for the generation of attribution masks and can provide interesting insights about the performance of deep learning models.

We tackle the challenge of generating long-take videos encompassing multiple non-repetitive yet coherent events. Existing approaches generate long videos conditioned on single input guidance often leading to repetitive content. To address this problem we develop a framework that uses multiple guidance sources to enhance long video generation. The main idea of our approach is to decouple video generation into keyframe generation and frame interpolation. In this process keyframe generation focuses on creating multiple coherent events while the frame interpolation stage generates smooth intermediate frames between keyframes using existing video generation models. A novel mask attention module is further introduced to improve coherence and efficiency. Experiments on challenging real-world videos demonstrate that the proposed method outperforms prior methods by up to 9.5% in objective metrics.

We consider the problem of test-time domain generalization where a model is trained on several source domains and adjusted on target domains never seen during training. Different from the common methods that fine-tune the model or adjust the classifier parameters online we propose to generate multiple layer parameters on the fly during inference by a lightweight meta-learned transformer which we call GeneralizeFormer. The layer-wise parameters are generated per target batch without fine-tuning or online adjustment. By doing so our method is more effective in dynamic scenarios with multiple target distributions and also avoids forgetting valuable source distribution characteristics. Moreover by considering layer-wise gradients the proposed method adapts itself to various distribution shifts. To reduce the computational and time cost we fix the convolutional parameters while only generating parameters of the Batch Normalization layers and the linear classifier. Experiments on six widely used domain generalization datasets demonstrate the benefits and abilities of the proposed method to efficiently handle various distribution shifts generalize in dynamic scenarios and avoid forgetting.

In this paper we present a novel generalizable object pose estimation method to determine the object pose using only one RGB image. Unlike traditional approaches that rely on instance-level object pose estimation and necessitate extensive training data our method offers generalization to unseen objects without extensive training operates with a single reference image of the object and eliminates the need for 3D object models or multiple views of the object. These characteristics are achieved by utilizing a diffusion model to generate novel-view images and conducting a two-sided matching on these generated images. Quantitative experiments demonstrate the superiority of our method over existing pose estimation techniques across both synthetic and real-world datasets. Remarkably our approach maintains strong performance even in scenarios with significant viewpoint changes highlighting its robustness and versatility in challenging conditions. The code will be released at https://github.com/scy639/Gen2SM

Single-source domain generalization (SDG) aims to learn a model from a single source domain that can generalize well on unseen target domains. This is an important task in computer vision particularly relevant to medical imaging where domain shifts are common. In this work we consider a challenging yet practical setting: SDG for cross-modality medical image segmentation. We combine causality-inspired theoretical insights on learning domain-invariant representations with recent advancements in diffusion-based augmentation to improve generalization across diverse imaging modalities. Guided by the "intervention-augmentation equivariant" principle we use controlled diffusion models (DMs) to simulate diverse imaging styles while preserving the content leveraging rich generative priors in large-scale pretrained DMs to comprehensively perturb the multidimensional style variable. Extensive experiments on challenging cross-modality segmentation tasks demonstrate that our approach consistently outperforms state-of-the-art SDG methods across three distinct anatomies and imaging modalities. The source code is available at https://github.com/ratschlab/ICMSeg.

Generalist models capable of handling multiple modalities and tasks simultaneously are currently one of the hottest research topics. However due to interference between different tasks during the training process existing generalist models require a very large decoder to achieve good results in various tasks which makes real-time prediction difficult for current generalist models. This paper introduces Generalist YOLO which takes a significant step towards real-time prediction systems for visual language generalist models. The proposed Generalist YOLO uses a unified encoder to reduce conflicts between different tasks thereby decreasing the complexity required by the decoder. It also introduces a primary-secondary co-attention mechanism that allows different tasks to learn together more effectively achieving high efficiency and high accuracy. We propose a semantically consistent asymmetric training strategy allowing various tasks to benefit from performance improvements brought by the latest research results in various fields. The proposed Generalist YOLO achieves excellent results on various vision and language tasks based on MS COCO. While maintaining high accuracy across all tasks it is 135 times faster than existing generalist models. The source code is released on GitHub at https://github.com/WongKinYiu/GeneralistYOLO.

Medical eye-tracking data is an important information source for understanding how radiologists visually interpret medical images. This information not only improves the accuracy of deep learning models for X-ray analysis but also their interpretability enhancing transparency in decision-making. However the current eye-tracking data is dispersed unprocessed and ambiguous making it difficult to derive meaningful insights. Therefore there is a need to create a new dataset with more focus and purposeful eyetracking data improving its utility for diagnostic applications. In this work we propose a refinement method inspired by the target-present visual search challenge: there is a specific finding and fixations are guided to locate it. After refining the existing eye-tracking datasets we transform them into a curated visual search dataset called GazeSearch specifically for radiology findings where each fixation sequence is purposefully aligned to the task of locating a particular finding. Subsequently we introduce a scan path prediction baseline called ChestSearch specifically tailored to GazeSearch. Finally we employ the newly introduced GazeSearch as a benchmark to evaluate the performance of current state-of-the-art methods offering a comprehensive assessment for visual search in the medical imaging domain. Code is available at https://github.com/ UARK-AICV/GazeSearch.

The Bird's-eye View (BeV) representation is widely used for 3D perception from multi-view camera images. It allows to merge features from different cameras into a common space providing a unified representation of the 3D scene. The key component is the view transformer which transforms image views into the BeV. However actual view transformer methods based on geometry or cross-attention do not provide a sufficiently detailed representation of the scene as they use a sub-sampling of the 3D space that is non-optimal for modeling the fine structures of the environment. In this paper we propose GaussianBeV a novel method for transforming image features to BeV by finely representing the scene using a set of 3D gaussians located and oriented in 3D space. This representation is then splattered to produce the BeV feature map by adapting recent advances in 3D representation rendering based on gaussian splatting. GaussianBeV is the first approach to use this 3D gaussian modeling and 3D scene rendering process in an optimization free manner i.e. without optimizing it on a specific scene and directly integrated into a single stage model for BeV scene understanding. Experiments show that the proposed representation is highly effective and place GaussianBeV as the new state-of-the-art on the BeV semantic segmentation task on the nuScenes dataset.

Recent advancements in 3D Gaussian Splatting (3DGS) have unlocked significant potential for modeling 3D head avatars providing greater flexibility than mesh-based methods and more efficient rendering compared to NeRF-based approaches. Despite these advancements the creation of controllable 3DGS-based head avatars remains time-intensive often requiring tens of minutes to hours. To expedite this process we here introduce the "Gaussian Deja-vu" framework which first obtains a generalized model of the head avatar and then personalizes the result. The generalized model is trained on large 2D (synthetic and real) image datasets. This model provides a well-initialized 3D Gaussian head that is further refined using a monocular video to achieve the personalized head avatar. For personalizing we propose learnable expression-aware rectification blendmaps to correct the initial 3D Gaussians ensuring rapid convergence without the reliance on neural networks. Experiments demonstrate that the proposed method meets its objectives. It outperforms state-of-the-art 3D Gaussian head avatars in terms of photorealistic quality as well as reduces training time consumption to at least a quarter of the existing methods producing the avatar in minutes. Project homepage: https://peizhiyan.github.io/docs/dejavu

We propose a method that achieves state-of-the-art rendering quality and efficiency on monocular dynamic scene reconstruction using deformable 3D Gaussians. Implicit deformable representations commonly model motion with a canonical space and time-dependent backward-warping deformation field. Our method GauFRe uses a forward-warping deformation to explicitly model non-rigid transformations of scene geometry. Specifically we propose a template set of 3D Gaussians residing in a canonical space and a time-dependent forward-warping deformation field to model dynamic objects. Additionally we tailor a 3D Gaussian-specific static component supported by an inductive bias-aware initialization approach which allows the deformation field to focus on moving scene regions improving the rendering of complex real-world motion. The differentiable pipeline is optimized end-to-end with a self-supervised rendering loss. Experiments show our method achieves competitive results and higher efficiency than both previous state-of-the-art NeRF and Gaussian-based methods. For real-world scenes GauFRe can train in 20 mins and offer 96 FPS real-time rendering on an RTX 3090 GPU.

Gait recognition holds the promise to robustly identify subjects based on walking patterns instead of appearance information. In recent years this field has been dominated by learning methods based on two input formats: silhouette images and sparse keypoints. Compared to image-based approaches keypoint-based methods can achieve significantly higher efficiency due to their sparsity. However sparsity also results in information loss thereby reducing performance. In this work we propose a novel keypoint-based Contour-Pose representation which compactly encodes both body shape and parts information. We further propose a local-to-global architecture called GaitContour to leverage this novel representation and efficiently compute subject embedding in two stages. The first stage consists of a local transformer that extracts features from five different body regions. The second stage then aggregates the regional features to estimate a global human gait representation. Such a design significantly reduces the complexity of the attention operation and improves both efficiency and performance. Through large scale experiments GaitContour is shown to perform significantly better than previous keypoint-based methods. Furthermore the Contour-Pose representation also achieves new SoTA performances on fusion-based gait recognition methods.

Gait recognition using point clouds captured by LiDAR (Light Detection And Ranging) sensors offers better adaptability to variations in walking conditions compared to camera-based methods due to the precise spatial information captured. However existing methods typically project the point clouds into a sequence of 2D depth images extended along the time dimension and adopt gait recognition networks optimized for camera-based approaches. This planar projection compromises the integrity of the 3D coordinates (length width and depth) and results in severe silhouette deformations with varied observation viewpoints similar to the camera-based methods. To better utilize the spatial information in gait point clouds we propose a true 3D gait representation using eff icient point cloud voxelization termed GaitCloud. Additionally we explore the unique nature of LiDAR-captured point clouds and present two improved modules adapted to our method called Layer Encoder (LE) and Horizontal Convolutional Pooling (HCP). Evaluation results using the open-access gait dataset SUSTech1K show that our method outperforms the state-of-the-art achieving recognition accuracies of 93.1% and 89.2% in cross-view and variance experiments respectively. These results demonstrate that 3D gait representation based on point cloud voxelization more effectively utilizes spatial information than depth images offering new possibilities for high-performance LiDAR-based gait recognition. The source code is available at https://github.com/seagrgz/GaitCloud-master.git.

High Dynamic Range (HDR) content (i.e. images and videos) has a broad range of applications. However capturing HDR content from real-world scenes is expensive and time-consuming. Therefore the challenging task of reconstructing visually accurate HDR images from their Low Dynamic Range (LDR) counterparts is gaining attention in the vision research community. A major challenge is the lack of datasets which capture diverse scene conditions (e.g. lighting weather locations) and various image features (e.g. color contrast saturation). To address this gap we introduce GTA-HDR a large-scale synthetic dataset of photo-realistic HDR images sampled from the GTA-V video game. We perform thorough evaluation of the proposed dataset which enables significant qualitative and quantitative improvements of the state-of-the-art HDR image reconstruction methods. Furthermore we demonstrate the effectiveness of the proposed dataset and its impact on additional computer vision tasks including 3D human pose estimation human body part segmentation and holistic scene segmentation. The dataset data collection pipeline and evaluation code are available at: https://github.com/HrishavBakulBarua/GTA-HDR.