WACV 2025 Papers

Our research investigates the capability of modern multimodal reasoning models powered by Large Language Models (LLMs) to facilitate vision-powered assistants for multi-step daily activities. Such assistants must be able to 1) encode relevant visual history from the assistant's sensors e.g. camera 2) forecast future actions for accomplishing the activity and 3) replan based on the user in the loop. To evaluate the first two capabilities grounding visual history and forecasting in short and long horizons we conduct benchmarking of two prominent classes of multimodal LLM approaches - Socratic Models and Vision Conditioned Language Models (VCLMs) on video-based action anticipation tasks using offline datasets. These offline benchmarks however do not allow us to close the loop with the user which is essential to evaluate the replanning capabilities and measure successful activity completion in assistive scenarios. To that end we conduct a first-of-its-kind user study with 18 participants performing 3 different multi-step cooking activities while wearing an egocentric observation device called Aria and following assistance from multimodal LLMs. We find that the Socratic approach outperforms VCLMs in both offline and online settings. We further highlight how grounding long visual history common in activity assistance remains challenging in current models especially for VCLMs and demonstrate that offline metrics do not indicate online performance.

With the exponential growth of video content the need for automated video highlight detection to extract key moments or highlights from lengthy videos has become increasingly pressing. This technology has the potential to enhance user experiences by allowing quick access to relevant content across diverse domains. Existing methods typically rely either on expensive manually labeled frame-level annotations or on a large external dataset of videos for weak supervision through category information. To overcome this we focus on unsupervised video highlight detection eliminating the need for manual annotations. We propose a novel unsupervised approach which capitalizes on the premise that significant moments tend to recur across multiple videos of the similar category in both audio and visual modalities. Surprisingly audio remains under-explored especially in unsupervised algorithms despite its potential to detect key moments. Through a clustering technique we identify pseudo-categories of videos and compute audio pseudo-highlight scores for each video by measuring the similarities of audio features among audio clips of all the videos within each pseudo-category. Similarly we also compute visual pseudo-highlight scores for each video using visual features. Then we combine audio and visual pseudo-highlights to create the audio-visual pseudo ground-truth highlight of each video for training an audio-visual highlight detection network. Extensive experiments and ablation studies on three benchmarks showcase the superior performance of our method over prior work.

Unsupervised intrinsic image decomposition (IID) is the task of separating a natural image into albedo and shade without ground truth during training. Although a recent model employing light detection and ranging (LiDAR) intensity demonstrated impressive performance the necessity of LiDAR intensity during inference restricts its practicality. To expand the usage scenario while maintaining the IID quality achieved by using both an image and its corresponding LiDAR intensity we propose a novel approach that utilizes an image without LiDAR intensity during inference while utilizing both an image and LiDAR intensity during training. Specifically our proposed model processes an image and LiDAR intensity individually using distinct encoder paths during training but utilizes only an image-encoder path during inference. Additionally we introduce an albedo-alignment loss aligning the gray-scale albedo from an image to that from its corresponding LiDAR intensity. LiDAR intensity is not affected by illumination effects including cast shadows thus albedo-alignment loss transfers the illumination-invariant property of LiDAR intensity to the image-encoder path. Furthermore we also propose image-LiDAR conversion (ILC) paths that mutually translates the style of an image and LiDAR intensity. IID models translate an image into albedo and shade styles while keeping the image contents thus it is important to separate the image into contents and style. Trained with pairs of an image and its corresponding LiDAR intensity which share contents but differ in style the mutual translation in ILC paths improve the accuracy of the separation. Consequently our model achieves comparable IID quality to the existing model with LiDAR intensity while utilizing only an image without LiDAR intensity during inference.

Unsupervised domain adaptation (UDA) for visual question answering (VQA) has attracted research interest. However with Multi-modal Large Language Models (MLLMs) showing great performance on VQA datasets UDA for VQA based on MLLMs remains unexplored. To fill this gap we propose the first systematic approach to Unsupervised Domain Adaptation VQA based on MLLMs (UDAM). First we introduce semantic context feature alignment and domain query feature alignment which utilize a single token embedding for each modality to capture contextual domain information from unimodal inputs and conduct coarse-grained feature alignment on it thus alleviating domain shifts in the unimodal feature space. Second we propose the novel semantics-guided query feature alignment which differentiates important domain-specific queries from learnable query outputs and conducts fine-grained feature alignment controlled by a semantics-guided weight map to reduce domain shifts in the cross-modal feature space. Third we devise a pair-wise domain-aware prompt strategy which aids UDA by prompting MLLMs to discern the commonality of tasks and the distinctiveness of domains in multi-modal inputs. Extensive experiments demonstrate UDAM's effectiveness in adapting MLLMs to unlabeled new domains.

Accurate analysis of microscopy images is hindered by the presence of noise. This noise is usually signal-dependent and often additionally correlated along rows or columns of pixels. Current self- and unsupervised denoisers can address signal-dependent noise but none can reliably remove noise that is also row- or column-correlated. Here we present the first fully unsupervised deep learning-based denoiser capable of handling imaging noise that is row-correlated as well as signal-dependent. Our approach uses a Variational Autoencoder (VAE) with a specially designed autoregressive decoder. This decoder is capable of modeling row-correlated and signal-dependent noise but is incapable of independently modeling underlying clean signal. The VAE therefore produces latent variables containing only clean signal information and these are mapped back into image space using a proposed second decoder network. Our method does not require a pre-trained noise model and can be trained from scratch using unpaired noisy data. We benchmark our approach on microscopy datatsets from a range of imaging modalities and sensor types each with row- or column-correlated signal-dependent noise and show that it outperforms existing self- and unsupervised denoisers.

Human-Object Interaction (HOI) detection aims to localize human-object pairs and recognize their interactions as <human action object> triplets. Recent advancements in pre-trained vision-language model (VLM) have improved zero-shot HOI detection enabling identification of unseen triplets. However existing methods leverage the VLM as an additional encoder only for interaction prediction not for human/object detection. This limitation hinders their ability to detect unseen objects. Furthermore the additional encoder increases both model size and computational cost. This paper proposes a novel HOI detection framework ECI-HOI which unleashes potentials of the pre-trained VLM for the zero-shot HOI detection by leveraging it for both of the sub-tasks. We first employ CLIP as a single image encoder reducing redundancy in the network architecture. In addition we propose an instance selector and a HO pair decoder to effectively harmonize the human/object detection and the interaction prediction in zero-shot manner. We evaluate our model under various settings on HICO-DET and our two new testsets: out-of-distribution image testset and novel object testset. Our model outperforms the state-of-the-art models while reducing the model size by more than 50% especially achieving a +10.01 mAP improvement under the unseen object setting on HICO-DET. The results on the proposed datasets highlight the zero-shot performance of our model on more challenging settings.

Event cameras deliver asynchronous pixel intensity changes which result in sparse event data that offers the advantages of high temporal resolution. These high temporal characteristics make researchers naturally incorporate event cameras into video frame interpolation (VFI) and video super-resolution (VSR). In this paper we make the first attempt to solve the space-time video super-resolution (STVSR) task effectively addressing both VFI and VSR simultaneously by leveraging temporally dense events. STVSR aims to generate intermediate high-resolution (HR) videos between consecutive low-resolution (LR) frames. To fully exploit the high temporal frequency of events for STVSR we focus on temporal alignment in two stages at low-resolution and after up-sampling in high-resolution. In temporal alignment at low-resolution to upsample spatial dimensions effectively we leverage high temporal features to preserve spatial context. On the other hand for temporal alignment at the high-resolution stage we employ a deformable sampling process from events to achieve accurate alignment with forward and backward directions. In addition we provide the SuperREST dataset which features high-frequency details and complex motion in an RGB-Event setup. Experimental results on several datasets demonstrate that our method achieves a significant performance gain on STVSR tasks with low computational cost. Our codes and datasets are available at https://github.com/Chohoonhee/ESTNet.

Text-guided image generation and editing using diffusion models have achieved remarkable advancements. Among these tuning-free methods have gained attention for their ability to perform edits without extensive model adjustments offering simplicity and efficiency. However existing tuning-free approaches often struggle with balancing fidelity and editing precision. Reconstruction errors in DDIM Inversion are partly attributed to the cross-attention mechanism in U-Net which introduces misalignments during the inversion and reconstruction process. To address this we analyze reconstruction from a structural perspective and propose a novel approach that replaces traditional cross-attention with uniform attention maps significantly enhancing image reconstruction fidelity. Our method effectively minimizes distortions caused by varying text conditions during noise prediction. To complement this improvement we introduce an adaptive mask-guided editing technique that integrates seamlessly with our reconstruction approach ensuring consistency and accuracy in editing tasks. Experimental results demonstrate that our approach not only excels in achieving high-fidelity image reconstruction but also performs robustly in real image composition and editing scenarios. This study underscores the potential of uniform attention maps to enhance the fidelity and versatility of diffusion-based image processing methods. Code is available at https://github.com/Mowenyii/Uniform-Attention-Maps.

Open-World Compositional Zero-Shot Learning (OW-CZSL) addresses the challenge of recognizing novel compositions of known primitives and entities. Even though prior works utilize language knowledge for recognition such approaches exhibit limited interactions between language-image modalities. Our approach primarily focuses on enhancing the inter-modality interactions through fostering richer interactions between image and textual data. Additionally we introduce a novel module aimed at alleviating the computational burden associated with exhaustive exploration of all possible compositions during the inference stage. While previous methods exclusively learn compositions jointly or independently we introduce an advanced hybrid procedure that leverages both learning mechanisms to generate final predictions. Our proposed model achieves state-of-the-art in OW-CZSL in three datasets while surpassing Large Vision Language Models (LLVM) in two datasets. Our code is available at https://github.com/hirunima/OWCZSL

Current methods have shown promising results in applying diffusion models to motion generation given text input. However these methods are limited to unimodal inputs and outputs restricted to motion generation alone and lacking multimodal control capabilities. To address these issues we introduce TMMGE a text-motion multimodal generation and editing framework based on diffusion. TMMGE overcomes single-modality limitations enabling exceptional performance and strong generalization across multiple tasks like text-driven motion generation motion captioning motion completion and multi-modal motion editing. TMMGE comprises three components: UTMV for mapping text and motion into a shared latent space using contrastive learning a controllable diffusion model customized for the UTMV space and MCRE for unifying multimodal conditions into CLIP representations enabling precise multimodal control and flexible motion editing through simple linear operations. We conducted both closed-world experiments and open-world experiments using the Motion-X dataset with detailed text descriptions with results demonstrating our model's effectiveness and generalizability across multiple tasks.

Neural implicit fields have recently emerged as a powerful representation method for multi-view surface reconstruction due to their simplicity and state-of-the-art performance. However reconstructing thin structures of indoor scenes while ensuring real-time performance remains a challenge for dense visual SLAM systems. Previous methods do not consider varying quality of input RGB-D data and employ fixed-frequency mapping process to reconstruct the scene which could result in the loss of valuable information in some frames. In this paper we propose Uni-SLAM a decoupled 3D spatial representation based on hash grids for indoor reconstruction. We introduce a novel defined predictive uncertainty to reweight the loss function along with strategic local-to-global bundle adjustment. Experiments on synthetic and real-world datasets demonstrate that our system achieves state-of-the-art tracking and mapping accuracy while maintaining real-time performance. It significantly improves over current methods with a 25% reduction in depth L1 error and a 66.86% completion rate within 1 cm on the Replica dataset reflecting a more accurate reconstruction of thin structures. Project page: https://shaoxiang777.github.io/project/uni-slam/

We introduce NOVIC an innovative real-time uNconstrained Open Vocabulary Image Classifier that uses an autoregressive transformer to generatively output classification labels as language. Leveraging the extensive knowledge of CLIP models NOVIC harnesses the embedding space to enable zero-shot transfer from pure text to images. Traditional CLIP models despite their ability for open vocabulary classification require an exhaustive prompt of potential class labels restricting their application to images of known content or context. To address this we propose an "object decoder" model that is trained on a large-scale 92M-target dataset of templated object noun sets and LLM-generated captions to always output the object noun in question. This effectively inverts the CLIP text encoder and allows textual object labels from essentially the entire English language to be generated directly from image-derived embedding vectors without requiring any a priori knowledge of the potential content of an image and without any label biases. The trained decoders are tested on a mix of manually and web-curated datasets as well as standard image classification benchmarks and achieve fine-grained prompt-free prediction scores of up to 87.5% a strong result considering the model must work for any conceivable image and without any contextual clues.

Unsupervised metric learning aims to learn the discriminative representations by grouping similar examples in the absence of labels. Many unsupervised metric learning algorithms combine clustering-based pseudo-label generation with embedding fine-tuning. However pseudo-labels can be unreliable and noisy. This could affect metric learning and degrade the quality of the learned representations. In this work we propose an approach to reduce the negative effect of label noise on learning discriminative embeddings by using context and prediction uncertainty. In particular we refine the pseudo-labels by aggregating information from neighbors. We propose a function to weigh the pairs leveraging their prediction confidence and uncertainty. We modify the metric learning loss function to incorporate this weight. Experimental results demonstrate the effectiveness of our proposed method on standard datasets for metric learning.

This work proposes a novel framework Uncertainty-Guided Cross Attention Ensemble Mean Teacher (UG-CEMT) for achieving state-of-the-art performance in semi-supervised medical image segmentation. UG-CEMT leverages the strengths of co-training and knowledge distillation by combining a Cross-attention Ensemble Mean Teacher framework (CEMT) inspired by Vision Transformers (ViT) with uncertainty-guided consistency regularization and Sharpness-Aware Minimization emphasizing uncertainty. UG-CEMT improves semi-supervised performance while maintaining a consistent network architecture and task setting by fostering high disparity between sub-networks. Experiments demonstrate significant advantages over existing methods like Mean Teacher and Cross-pseudo Supervision in terms of disparity domain generalization and medical image segmentation performance. UG-CEMT achieves state-of-the-art results on multi-center prostate MRI and cardiac MRI datasets where object segmentation is particularly challenging. Our results show that using only 10% labeled data UG-CEMT approaches the performance of fully supervised methods demonstrating its effectiveness in exploiting unlabeled data for robust medical image segmentation. The code is publicly available at https://github.com/Meghnak13/UG-CEMT

Deep neural networks (DNNs) have transformed biomedical image analysis particularly in histopathology with Whole Slide Images (WSIs) classification. However training DNNs requires large annotated datasets which is challenging due to the high heterogeneity and high resolution of WSIs. Multiple Instance Learning (MIL) has become a popular method for weakly supervised classification in this context training with only slide-level labels. Despite the advancements ensuring the reliability of model performance is crucial in safety-critical domains including healthcare. Deep learning models in real-world decision-making systems must accurately predict probability estimates to reflect the true likelihood of correctness known as confidence calibration. This study introduces a novel calibration framework UDLS which uses data-wise label smoothing based on predictive uncertainty to improve the calibration of MIL frameworks. This approach involves augmenting WSIs with PatchFeatureDropout computing predictive uncertainty estimates for original data and applying these estimates to each sample for label smoothing during model training. Experimental results on benchmark histopathology datasets show noticeable improvements in both calibration and classification performance highlighting UDLS's potential for enhancing the reliability of predictions from deep learning models in clinical settings.

The importance of quantifying uncertainty in deep networks has become paramount for reliable real-world applications. In this paper we propose a method to improve uncertainty estimation in medical Image-to-Image (I2I) translation. Our model integrates aleatoric uncertainty and employs Uncertainty-Aware Regularization (UAR) inspired by simple priors to refine uncertainty estimates and enhance reconstruction quality. We show that by leveraging simple priors on parameters our approach captures more robust uncertainty maps effectively refining them to indicate precisely where the network encounters difficulties while being less affected by noise. Our experiments demonstrate that UAR not only improves translation performance but also provides better uncertainty estimations particularly in the presence of noise and artifacts. We validate our approach using two medical imaging datasets showcasing its effectiveness in maintaining high confidence in familiar regions while accurately identifying areas of uncertainty in novel/ambiguous scenarios.

Accurate sensor calibration is crucial for autonomous systems yet its uncertainty quantification remains underexplored. We present the first approach to integrate uncertainty awareness into online extrinsic calibration combining Monte Carlo Dropout with Conformal Prediction to generate prediction intervals with a guaranteed level of coverage. Our method proposes a framework to enhance existing calibration models with uncertainty quantification compatible with various network architectures. Validated on KITTI (RGB Camera-LiDAR) and DSEC (Event Camera-LiDAR) datasets we demonstrate effectiveness across different visual sensor types measuring performance with adapted metrics to evaluate the efficiency and reliability of the intervals. By providing calibration parameters with quantifiable confidence measures we offer insights into the reliability of calibration estimates which can greatly improve the robustness of sensor fusion in dynamic environments and usefully serve the Computer Vision community.

Semi-supervised (SS) semantic segmentation exploits both labeled and unlabeled images to overcome tedious and costly pixel-level annotation problems. Pseudolabel supervision is one of the core approaches of training networks with both pseudo labels and ground-truth labels. This work uses aleatoric or data uncertainty and energy based modeling in intersection-union pseudo supervised network. The aleatoric uncertainty is modeling the inherent noise variations of the data in a network with two predictive branches. The per-pixel variance parameter obtained from the network gives a quantitative idea about the data uncertainty. Moreover energy-based loss realizes the potential of generative modeling on the downstream SS segmentation task. The aleatoric and energy loss are applied in conjunction with pseudo-intersection labels pseudo-union labels and ground-truth on the respective network branch. The comparative analysis with state-of-the-art methods has shown improvement in performance metrics. The code is availaible at https://visdomlab.github.io/DUEB/.

Machine-learning-assisted cancer subtyping is a promising avenue in digital pathology. Cancer subtyping models however require careful training using expert annotations so that they can be inferred with a degree of known certainty (or uncertainty). To this end we introduce the concept of uncertainty awareness into a self-supervised contrastive learning model. This is achieved by computing an evidence vector at every epoch which assesses the model's confidence in its predictions. The derived uncertainty score is then utilized as a metric to selectively label the most crucial images that require further annotation thus iteratively refining the training process. With just 1-10% of strategically selected annotations we attain state-of-the-art performance in cancer subtyping on benchmark datasets. Our method not only strategically guides the annotation process to minimize the need for extensive labeled datasets but also improve the precision and efficiency of classifications. This development is particularly beneficial in settings where the availability of labeled data is limited offering a promising direction for future research and application in digital pathology. Our code is available at https://github.com/Nirhoshan/AI-for-histopathology