WACV 2025 Papers

3D human pose estimation (3D HPE) is an important computer vision task with various practical applications. However 3D pose estimation for multi-person from a monocular video (3DMPPE) is particularly challenging. Recent transformer-based approaches focus on capturing the spatial-temporal information from sequential 2D poses which unfortunately loses the visual feature relevant for 3D pose estimation. In this paper we propose an end-to-end framework called Event Guided Video Transformer (EVT) which predicts 3D poses directly from video frames by learning spatial-temporal contextual information from visual features effectively. In addition our design is the first that incorporates event features to help guide 3D pose estimation. EVT first decouples persons into different instance-aware feature maps from video frames. These features containing specific clues of body structure information are then fed together with event features into an attention based Event-Aware Embedding Module. Next the fused features for each instance are then fed into an intra-human relation extraction module and subsequently to a temporal transformer to extract inter-frame relationship. Finally the extracted features are fed into a decoder for 3D pose estimation. Experiments using three widely used 3D pose estimation benchmarks show that our proposed EVT achieves better performance than state-of-the-art models.

Recent video semantic segmentation (VSS) methods have demonstrated promising results in well-lit environments. However their performance significantly drops in low-light scenarios due to limited visibility and reduced contextual details. In addition unfavorable low-light conditions make it harder to incorporate temporal consistency across video frames and thus lead to video flickering effects. Compared with conventional cameras event cameras can capture motion dynamics filter out temporal-redundant information and are robust to lighting conditions. To this end we propose EVSNet a lightweight framework that leverages event modality to guide the learning of a unified illumination-invariant representation. Specifically we leverage a Motion Extraction Module to extract short-term and long-term temporal motions from event modality and a Motion Fusion Module to integrate image features and motion features adaptively. Furthermore we use a Temporal Decoder to exploit video contexts and generate segmentation predictions. Such designs in EVSNet result in a lightweight architecture while achieving SOTA performance. Experimental results on 3 large-scale datasets demonstrate our proposed EVSNet outperforms SOTA methods with up to 11x higher parameter efficiency.

3D human pose estimation (3D HPE) is an important computer vision task with various practical applications. Researchers have proposed various deep learning-based methods for 3D HPE. However the majority of such methods rely on lifting 2D pose sequence to 3D which do not perform well in challenging scenarios and are often computationally expensive. Such methods typically rely on 2D joint coordinates which do not provide much spatial context to solve ambiguity problem. In addition merely relying on information extracted from RGB frames may miss temporal information and structural context. Thus in this paper we propose a framework that incorporates event stream as an additional input since event features provide such useful information. Moreover instead of using 2D joint coordinates in pose sequence our framework uses intermediate visual representations produced by off-the-shelf 2D pose detectors to implicitly encode joint-centric spatial context. Our new framework is a novel state space model (SSM)-based solution called Event-Guided Context Aware MambaPose (CA-MambaPose). In CA-MambaPose framework we design a novel cross modality fusion mamba module to skillfully fuse the RGB and Event features. CA-MambaPose has lower computational cost due to the efficiency of Mamba blocks. We conduct extensive experiments to evaluate CA-MambaPose using two existing datasets. Our experimental results show that CA-MambaPose achieves better performance than SOTA methods.

While the performance of deep neural networks is rapidly developing their reliability is increasingly receiving more attention. Explainability methods are one of the most relevant tools to enhance reliability mainly by highlighting important input features for the explanation purpose. Although numerous explainability methods have been proposed their assessment remains challenging due to the absence of ground truth. Several existing studies propose evaluation methods from a certain aspect e.g. fidelity robustness etc. However they typically address only one property of explanations and thus more assessing perspectives contribute to a better explanation evaluating system. This work proposes an evaluation method from a novel perspective called sensitivity consistency where the intuition behind is that features and parameters that strongly impact the predictions and explanations should be highly consistent and vise versa. Extensive experiments on different datasets and models evaluate popular explainability methods while providing qualitative and quantitative results. Our approach further complements the existing evaluation systems and aims to facilitate the proposal of an acknowledged explanation evaluation methodology.

Spatial understanding of the semantics of the surroundings is a key capability needed by autonomous cars to enable safe driving decisions. Recently purely vision-based solutions have gained increasing research interest. In particular approaches extracting a bird's eye view (BEV) from multiple cameras have demonstrated great performance for spatial understanding. This paper addresses the dependency on learned positional encodings to correlate image and BEV feature map elements for transformer-based methods. We propose leveraging epipolar geometric constraints to model the relationship between cameras and the BEV by Epipolar Attention Fields. They are incorporated into the attention mechanism as a novel attribution term serving as an alternative to learned positional encodings. Experiments show that our method EAFormer outperforms previous BEV approaches by 2% mIoU for map semantic segmentation and exhibits superior generalization capabilities compared to implicitly learning the camera configuration.

In neural networks recognizing visual patterns is challenging because global average pooling disregards local patterns and solely relies on over-concentrated activation. Global average pooling enforces the network to learn objects regardless of their location so features tend to be activated only in specific regions. To support this claim we provide a novel analysis of the problems that over-concentration brings about in networks with extensive experiments. We analyze the over-concentration through problems arising from feature variance and dead neurons that are not activated. Based on our analysis we introduce a multi-token attention pooling layer to alleviate the over-concentration problem. Our attention-pooling layer captures broad-sight local patterns by learning multiple tokens with the proposed distillation algorithm. It resolves the high bias and high variance errors of learned multi-tokens which is crucial when aggregating local patterns with multi-tokens. Our method applies to various vision tasks and network architectures such as CNN ViT and MLP-Mixer. The proposed method improves baselines with few extra resources and a network employing our pooling method works favorably against state-of-the-art networks. We open-source the code at https://github.com/Lab-LVM/imagenet-models.

Current facial expression recognition (FER) models are often designed in a supervised learning manner and thus are constrained by the lack of large-scale facial expression images with high-quality annotations. Consequently these models often fail to generalize well performing poorly on unseen images in inference. Vision-language-based zero-shot models demonstrate a promising potential for addressing such challenges. However these models lack task-specific knowledge and therefore are not optimized for the nuances of recognizing facial expressions. To bridge this gap this work proposes a novel method Exp-CLIP to enhance zero-shot FER by transferring the task knowledge from large language models (LLMs). Specifically based on the pre-trained vision-language encoders we incorporate a projection head designed to map the initial joint vision-language space into a space that captures representations of facial actions. To train this projection head for subsequent zero-shot predictions we propose to align the projected visual representations with task-specific semantic meanings derived from the LLM encoder and the text instruction-based strategy is employed to customize the LLM knowledge. Given unlabelled facial data and efficient training of the projection head Exp-CLIP achieves superior zero-shot results to the CLIP models and several other large vision-language models (LVLMs) on seven in-the-wild FER datasets.

The performance of vision-language models (VLMs) such as CLIP in visual classification tasks has been enhanced by leveraging semantic knowledge from large language models (LLMs) including GPT. Recent studies have shown that in zero-shot classification tasks descriptors incorporating additional cues high-level concepts or even random characters often outperform those using only category names. In many classification tasks while the top-1 accuracy may be relatively low the top-5 accuracy is often significantly higher. This gap implies that most misclassifications occur among a few similar classes highlighting the model's difficulty in distinguishing between classes with subtle differences. To address this challenge we introduce a novel concept of comparative descriptors. These descriptors emphasize the unique features of a target class against its most similar classes enhancing differentiation. By generating and integrating these comparative descriptors into the classification framework we refine the semantic focus and improve classification accuracy. An additional filtering process ensures that these descriptors are closer to the image embeddings in the CLIP space further enhancing performance. Our approach demonstrates improved accuracy and robustness in visual classification tasks by addressing the specific challenge of subtle inter-class differences. Code is available at https://github.com/hk1ee/Comparative-CLIP

Large-scale pre-trained Vision-Language Models (VLMs) have exhibited impressive zero-shot performance and transferability allowing them to adapt to downstream tasks in a data-efficient manner. However when only a few labeled samples are available adapting VLMs to distinguish subtle differences between similar classes in specific downstream tasks remains challenging. In this work we propose a Simple yet effective Negative Learning approach SimNL to more efficiently exploit the task-specific knowledge from few-shot labeled samples. Unlike previous methods that focus on identifying a set of representative positive features defining "what is a CLASS " SimNL discovers a complementary set of negative features that define "what is not a CLASS " providing additional insights that supplement the positive features to enhance task-specific recognition capability. Further we identify that current adaptation approaches are particularly vulnerable to potential noise in the few-shot sample set. To mitigate this issue we introduce a plug-and-play few-shot instance reweighting technique to suppress noisy outliers and amplify clean samples for more stable adaptation. Our extensive experimental results across 15 datasets validate that the proposed SimNL outperforms existing state-of-the-art methods on both few-shot learning and domain generalization tasks while achieving competitive computational efficiency. Code is available at https://github.com/zhangce01/SimNL.

Current AI-assisted skin image diagnosis has achieved dermatologist-level performance in classifying skin cancer driven by rapid advancements in deep learning architectures. However unlike traditional vision tasks skin images in general present unique challenges due to the limited availability of well-annotated datasets complex variations in conditions and the necessity for detailed interpretations to ensure patient safety. Previous segmentation methods have sought to reduce image noise and enhance diagnostic performance but these techniques require fine-grained pixel-level ground truth masks for training. In contrast with the rise of foundation models the Segment Anything Model (SAM) has been introduced to facilitate promptable segmentation enabling the automation of the segmentation process with simple yet effective prompts. Efforts applying SAM predominantly focus on dermatoscopy images which present more easily identifiable lesion boundaries than clinical photos taken with smartphones. This limitation constrains the practicality of these approaches to real-world applications. To overcome the challenges posed by noisy clinical photos acquired via non-standardized protocols and to improve diagnostic accessibility we propose a novel Cross-Attentive Fusion framework for interpretable skin lesion diagnosis. Our method leverages SAM to generate visual concepts for skin diseases using prompts integrating local visual concepts with global image features to enhance model performance. Extensive evaluation on two skin disease datasets demonstrates our proposed method's effectiveness on lesion diagnosis and interpretability.

This work introduces an enhanced approach to generating scene graphs by incorporating both a relationship hierarchy and commonsense knowledge. Specifically we begin by proposing a hierarchical relation head that exploits an informative hierarchical structure. It jointly predicts the relation super-category between object pairs in an image along with detailed relations under each super-category. Following this we implement a robust commonsense validation pipeline that harnesses foundation models to critique the results from the scene graph prediction system removing nonsensical predicates even with a small language-only model. Extensive experiments on Visual Genome and OpenImage V6 datasets demonstrate that the proposed modules can be seamlessly integrated as plug-and-play enhancements to existing scene graph generation algorithms. The results show significant improvements with an extensive set of reasonable predictions beyond dataset annotations. Codes are available at https://github.com/bowenupenn/scene graph_commonsense.

Identifying predictive covariates which forecast individual treatment effectiveness is crucial for decision-making across different disciplines such as personalized medicine. These covariates referred to as biomarkers are extracted from pre-treatment data often within randomized controlled trials and should be distinguished from prognostic biomarkers which are independent of treatment assignment. Our study focuses on discovering predictive imaging biomarkers specific image features by leveraging pre-treatment images to uncover new causal relationships. Unlike labor-intensive approaches relying on handcrafted features prone to bias we present a novel task of directly learning predictive features from images. We propose an evaluation protocol to assess a model's ability to identify predictive imaging biomarkers and differentiate them from purely prognostic ones by employing statistical testing and a comprehensive analysis of image feature attribution. We explore the suitability of deep learning models originally developed for estimating the conditional average treatment effect (CATE) for this task which have been assessed primarily for their precision of CATE estimation while overlooking the evaluation of imaging biomarker discovery. Our proof-of-concept analysis demonstrates the feasibility and potential of our approach in discovering and validating predictive imaging biomarkers from synthetic outcomes and real-world image datasets. Our code is available at https://github.com/MIC-DKFZ/predictive_image_biomarker_analysis.

In this work we address the challenging and emergent problem of novel object detection (NOD) focusing on the accurate detection of both known and novel object categories during inference. Traditional object detection algorithms are inherently closed-set limiting their capability to handle NOD. We present a novel approach to transform existing closed-set detectors into open-set detectors. This transformation is achieved by leveraging the complementary strengths of pre-trained foundational models specifically CLIP and SAM through our cooperative mechanism. Furthermore by integrating this mechanism with state-of-the-art open-set detectors such as GDINO we establish new benchmarks in object detection performance. Our method achieves 17.42 mAP in novel object detection and 42.08 mAP for known objects on the challenging LVIS dataset. Adapting our approach to the COCO OVD split we obtain an impressive result of 49.6 Novel AP50 which outperforms existing SOTA methods with similar backbone. Our code is available at: https://rohit901.github.io/coop-foundation-models/

Monocular depth estimation (MDE) is a challenging task in computer vision often hindered by the cost and scarcity of high-quality labeled datasets. We tackle this challenge using auxiliary datasets from related vision tasks for an alternating training scheme with a shared decoder built on top of a pre-trained vision foundation model while giving a higher weight to MDE. Through extensive experiments we demonstrate the benefits of incorporating various in-domain auxiliary datasets and tasks to improve MDE quality on average by 11%. Our experimental analysis shows that auxiliary tasks have different impacts confirming the importance of task selection highlighting that quality gains are not achieved by merely adding data. Remarkably our study reveals that using semantic segmentation datasets as Multi-Label Dense Classification (MLDC) often results in additional quality gains. Lastly our method significantly improves the data efficiency for the considered MDE datasets enhancing their quality while reducing their size by at least 80%. This paves the way for using auxiliary data from related tasks to improve MDE quality despite limited availability of high-quality labeled data. Code is available at https://jugit.fz-juelich.de/ias-8/mdeaux.

Diffusion models are a powerful class of generative models capable of producing high-quality images from pure noise using a simple text prompt. While most methods which introduce additional spatial constraints into the generated images (e.g. bounding boxes) require fine-tuning a smaller and more recent subset of these methods take advantage of the models' attention mechanism and are training-free. These methods generally fall into one of two categories. The first entails modifying the cross-attention maps of specific tokens directly to enhance the signal in certain regions of the image. The second works by defining a loss function over the cross-attention maps and using the gradient of this loss to guide the latent. While previous work explores these as alternative strategies we provide an interpretation for these methods which highlights their complimentary features and demonstrate that it is possible to obtain superior performance when both methods are used in concert.

Deep-learning and large scale language-image training have produced image object detectors that generalise well to diverse environments and semantic classes. However existing object detection paradigms are not optimally tailored for the embodied conditions inherent in robotics where the same objects are repeatedly observed over time. In this setting detectors that operate on single images or short sequences are likely to produce inconsistent predictions. Motivated by this we explore if the embodiment of the detector can be utilised to generate more consistent and reliable detections during repeat observation of a scene. We propose a novel framework that incrementally updates a spatial feature memory while using it as a prior to perform image object detection. By leveraging the embodiment of the robot in this way raw object detection performance is enhanced by up to 4.12 mAP and downstream robotic tasks such as semantic mapping and object recall are improved. We also investigate the structure this spatial memory should take leading to an implementation that aggregates features from the shared language-image embedding space. This approach allows the detector to effectively balance the use of memory and image features while ensuring that the benefits of language-image pre-training can be enjoyed alongside our spatial memory.

Endoscopic assessment using the Mayo clinic score (or Mayo score 4 categories) is currently the standard for diagnosing and evaluating mucosal disease activities. However annotating Mayo scores is time-consuming and often relies on weakly labeled evaluations from central and local readers (doctors) leading to a large number of unlabeled or mislabeled video clips. Additionally such labels also suffer from data imbalance due to patient distributions and varying disease severity levels. This gap underscores the need for more customizable and refined methods for endoscopic scoring and localization. To address these challenges we introduce an end-to-end pipeline with a new dataset for endoscopic scoring and localization in unconstrained clinical trial videos. Specifically we propose an automated scoring system that includes an active learning-based pre-processing stage for cleaning the raw videos which are then weakly labeled by doctors. Therefore we obtain a comprehensive dataset for endoscopic Mayo scoring comprising approximately 49.9 hours of video and 86423 clips providing a solid foundation for developing endoscopic scoring and localization models. Then we propose a video classification model for Mayo score classifications. For personalized disease quantification in localization we introduce a novel 1D trajectory model with a novel cumulative disease score that addresses the limitations of previous 3D trajectory projection methods. Our dataset and end-to-end pipeline offer a valuable foundation for advancing endoscopic clinical trial research.

A notable challenge in 3D talking head generation consists in blending speech-related motions with expression dynamics. This is primarily caused by the lack of comprehensive 3D datasets that combine diversity in spoken sentences with a variety of facial expressions. Some literature works attempted to overcome such lack of data by fitting parametric 3D models (3DMMs) to 2D videos and using the reconstructed 3D faces as replacement. However their underlying parametric space limits the precision required to accurately reproduce convincing lip motions and synching which is crucial for the application at hand. In this work we look at the problem from a different perspective and developed a data-driven technique to combine inexpressive 3D talking heads with a set of 3D expressive sequences which we used for creating a synthetic dataset called EmoVOCA. We then designed and trained an emotional 3D talking head generator that accepts a 3D face an audio file an emotion label and an intensity value as inputs and learns to animate the audio-synchronized lip movements with expressive traits of the face. Comprehensive experiments both quantitative and qualitative using our data and generator evidence superior ability in synthesizing convincing animations when compared with the best performing methods in the literature. Our code and pre-trained models are available at https://github.com/miccunifi/EmoVOCA.

Sampling from Diffusion Models can alternatively be seen as solving differential equations where there is a challenge in balancing speed and image visual quality. ODE-based samplers offer rapid sampling time but reach a performance limit whereas SDE-based samplers achieve superior quality albeit with longer iterations. In this work we formulate the sampling process as an Extended Reverse-Time SDE (ER SDE) unifying prior explorations into ODEs and SDEs. Theoretically leveraging the semi-linear structure of ER SDE solutions we offer exact solutions and approximate solutions for VP SDE and VE SDE respectively. Based on the approximate solution space of the ER SDE referred to as one-step prediction errors we yield mathematical insights elucidating the rapid sampling capability of ODE solvers and the high-quality sampling ability of SDE solvers. Additionally we unveil that VP SDE solvers stand on par with their VE SDE counterparts. Based on these findings leveraging the dual advantages of ODE solvers and SDE solvers we devise efficient high-quality samplers namely ER-SDE-Solvers. Experimental results demonstrate that ER-SDE-Solvers achieve state-of-the-art performance across all stochastic samplers while maintaining efficiency of deterministic samplers. Specifically on the ImageNet 128 x 128 dataset ER-SDE-Solvers obtain 8.33 FID in only 20 function evaluations. Code is available at https://github.com/QinpengCui/ER-SDE-Solver