WACV 2025 Papers

Leaf instance segmentation is a challenging multi-instance segmentation task aiming to separate and delineate each leaf in an image of a plant. Accurate segmentation of each leaf is crucial for plant-related applications such as the fine-grained monitoring of plant growth and crop yield estimation. This task is challenging because of the high similarity (in shape and colour) great size variation and heavy occlusions among leaf instances. Furthermore the typically small size of annotated leaf datasets makes it more difficult to learn the distinctive features needed for precise segmentation. We hypothesise that the key to overcoming the these challenges lies in the specific spatial patterns of leaf distribution. In this paper we propose the Guided Mask Transformer (GMT) which leverages and integrates leaf spatial distribution priors into a Transformer-based segmentor. These spatial priors are embedded in a set of guide functions that map leaves at different positions into a more separable embedding space. Our GMT consistently outperforms the state-of-the-art on three public plant datasets. Our code is available at https://github.com/vios-s/gmt-leaf-ins-seg.

The connection between our 3D surroundings and the descriptive language that characterizes them would be well-suited for localizing and generating human motion in context but for one problem. The complexity introduced by multiple modalities makes capturing this connection challenging with a fixed set of descriptors. Specifically closed vocabulary scene encoders which require learning text-scene associations from scratch have been favored in the literature often resulting in inaccurate motion grounding. In this paper we propose a method that integrates an open vocabulary scene encoder into the architecture establishing a robust connection between text and scene. Our two-step approach starts with pretraining the scene encoder through knowledge distillation from an existing open vocabulary semantic image segmentation model ensuring a shared text-scene feature space. Subsequently the scene encoder is fine-tuned for conditional motion generation incorporating two novel regularization losses that regress the category and size of the goal object. Our methodology achieves up to a 30% reduction in the goal object distance metric compared to the prior state-of-the-art baseline model on the HUMANISE dataset. This improvement is demonstrated through evaluations conducted using three implementations of our framework a perceptual study and an open vocabulary experiment. Additionally our method is designed to accommodate future 2D open vocabulary segmentation methods for distillation in a plug-and-play manner.

In various video-language learning tasks the challenge of achieving cross-modality alignment with multi-grained data persists. We propose a method to tackle this challenge from two crucial perspectives: data and modeling. Given the absence of a multi-grained video-text pretraining dataset we introduce a Granularity EXpansion (GEX) method with Integration and Compression operations to expand the granularity of a single-grained dataset. To better model multi-grained data we introduce an Iterative Approximation Module (IAM) which embeds multi-grained videos and texts into a unified low-dimensional semantic space while preserving essential information for cross-modal alignment. Furthermore GEXIA is highly scalable with no restrictions on the number of video-text granularities for alignment. We evaluate our work on three categories of video tasks across seven benchmark datasets showcasing state-of-the-art or comparable performance. Remarkably our model excels in tasks involving long-form video understanding even though the pretraining dataset only contains short video clips.

Depth estimation plays a pivotal role in autonomous driving facilitating a comprehensive understanding of the vehicle's 3D surroundings. Radar with its robustness to adverse weather conditions and capability to measure distances has drawn significant interest for radar-camera depth estimation. However existing algorithms process the inherently noisy and sparse radar data by projecting 3D points onto the image plane for pixel-level feature extraction overlooking the valuable geometric information contained within the radar point cloud. To address this gap we propose GET-UP leveraging attention-enhanced Graph Neural Networks (GNN) to exchange and aggregate both 2D and 3D information from radar data. This approach effectively enriches the feature representation by incorporating spatial relationships compared to traditional methods that rely only on 2D feature extraction. Furthermore we incorporate a point cloud upsampling task to densify the radar point cloud rectify point positions and derive additional 3D features under the guidance of lidar data. Finally we fuse radar and camera features during the decoding phase for depth estimation. We benchmark our proposed GET-UP on the nuScenes dataset achieving state-of-the-art performance with a 15.3% and 14.7% improvement in MAE and RMSE over the previously best-performing model. Code: https://github.com/ harborsarah/GET-UP

Augmentation by generative modelling yields a promising alternative to the accumulation of surgical data where ethical organisational and regulatory aspects must be considered. Yet the joint synthesis of (image mask) pairs for segmentation a major application in surgery is rather unexplored. We propose to learn semantically comprehensive yet compact latent representations of the (image mask) space which we jointly model with a Latent Diffusion Model. We show that our approach can effectively synthesise unseen high-quality paired segmentation data of remarkable semantic coherence. Generative augmentation is typically applied pre-training by synthesising a fixed number of additional training samples to improve downstream task models. To enhance this approach we further propose Generative Adaptive Uncertainty-guided Diffusion-based Augmentation (GAUDA) leveraging the epistemic uncertainty of a Bayesian downstream model for targeted online synthesis. We condition the generative model on classes with high estimated uncertainty during training to produce additional unseen samples for these classes. By adaptively utilising the generative model online we can minimise the number of additional training samples and centre them around the currently most uncertain parts of the data distribution. GAUDA effectively improves downstream segmentation results over comparable methods by an average absolute IoU of 1.6% on CaDISv2 and 1.5% on CholecSeg8k two prominent surgical datasets for semantic segmentation.

We train a feed-forward text-to-3D diffusion generator for human characters using only single-view 2D data for supervision. Existing 3D generative models cannot yet match the fidelity of image and/or video generative models. State-of-the-art 3D generators are either limited by the volume and diversity of existing 3D data available for supervision or those that can be trained with only 2D data as supervision produce coarser results cannot be text-conditioned and/or must revert to test-time optimization. We observe that GAN- and diffusion-based generators have complementary qualities: GANs can be trained efficiently with 2D supervision to produce high-quality 3D objects but are hard to condition on text. In contrast diffusion models can be conditioned efficiently but tend to be hard to train with only 2D supervision. We introduce GANFusion that starts by generating unconditional triplane features for 3D data using a GAN architecture trained with only single-view 2D data. We then generate random samples from the GAN caption them and train a text-conditioned diffusion model that directly learns to sample from the space of good triplane features that can be decoded into 3D objects. We evaluate the proposed method in the context of text-conditioned full-body human generation and show improvements over possible alternatives.

Lensless imaging offers a significant opportunity to develop ultra-compact cameras by removing the conventional bulky lens system. However without a focusing element the sensor's output is no longer a direct image but a complex multiplexed scene representation. Traditional methods have attempted to address this challenge by employing learnable inversions and refinement models but these methods are primarily designed for 2D reconstruction and do not generalize well to 3D reconstruction. We introduce GANESH a novel framework designed to enable simultaneous refinement and novel view synthesis from multi-view lensless images. Unlike existing methods that require scene-specific training our approach supports on-the-fly inference without retraining on each scene. Moreover our framework allows us to tune our model to specific scenes enhancing the rendering and refinement quality. To facilitate research in this area we also present the first multi-view lens- less dataset LenslessScenes. Extensive experiments demonstrate that our method outperforms current approaches in reconstruction accuracy and refinement quality. Code and video results are available here.

Visual counterfactual explanations are ideal hypothetical images that change the decision-making of the classifier with high confidence toward the desired class while remaining visually plausible and close to the initial image. In this paper we propose a new approach to tackle two key challenges in recent prominent works: i) determining which specific counterfactual features are crucial for distinguishing the "concept" of the target class from the original class and ii) supplying valuable explanations for the non-robust classifier without relying on the support of an adversarially robust model. Our method identifies the essential region for modification through algorithms that provide visual explanations and then our framework generates realistic counterfactual explanations by combining adversarial attacks based on pruning the adversarial gradient of the target classifier and the latent diffusion model. The proposed method outperforms previous state-of-the-art results on various evaluation criteria on ImageNet and CelebA-HQ datasets. In general our method can be applied to arbitrary classifiers highlight the strong association between visual and counterfactual explanations make semantically meaningful changes from the target classifier and provide observers with subtle counterfactual images.

Medical image segmentation is crucial for precise diagnosis treatment planning and disease monitoring in clinical settings. While convolutional neural networks (CNNs) have achieved remarkable success they struggle with modeling long-range dependencies. Vision Transformers (ViTs) address this limitation by leveraging self-attention mechanisms to capture global contextual information. However ViTs often fall short in local feature description which is crucial for precise segmentation. To address this issue we reformulate self-attention in the frequency domain to enhance both local and global feature representation. Our approach the Enhanced Wave Vision Transformer (EW-ViT) incorporates wavelet decomposition within the self-attention block to adaptively refine feature representation in low and high-frequency components. We also introduce the Prompt-Guided High-Frequency Refiner (PGHFR) module to handle image degradation which mainly affects high-frequency components. This module uses implicit prompts to encode degradation-specific information and adjust high-frequency representations accordingly. Additionally we apply a contrastive learning strategy to maintain feature consistency and ensure robustness against noise leading to state-of-the-art (SOTA) performance in medical image segmentation especially under various conditions of degradation. Source code is available at GitHub.

Building on the momentum of image generation diffusion models there is an increasing interest in video-based diffusion models. However video generation poses greater challenges due to its higher-dimensional nature the scarcity of training data and the complex spatiotemporal relationships involved. Image generation models due to their extensive data requirements have already strained computational resources to their limits. There have been instances of these models reproducing elements from the training samples leading to concerns and even legal disputes over sample replication. Video diffusion models which operate with even more constrained datasets and are tasked with generating both spatial and temporal content may be more prone to replicating samples from their training sets. Compounding the issue these models are often evaluated using metrics that inadvertently reward replication. In our paper we present a systematic investigation into the phenomenon of sample replication in video diffusion models. We scrutinize various recent diffusion models for video synthesis assessing their tendency to replicate spatial and temporal content in both unconditional and conditional generation scenarios. Our study identifies strategies that are less likely to lead to replication. Furthermore we propose new evaluation strategies that take replication into account offering a more accurate measure of a model's ability to generate the original content.

Developing robust and versatile deep-learning models is essential for enhancing diagnostic accuracy and guiding clinical interventions in medical imaging but it requires a large amount of annotated data. The advancement of deep learning has facilitated the creation of numerous medical datasets with diverse expert-level annotations. Aggregating these datasets can maximize data utilization and address the inadequacy of labeled data. However the heterogeneity of expert-level annotations across tasks such as classification localization and segmentation presents a significant challenge for learning from these datasets. To this end we introduce Foundation X an end-to-end framework that utilizes diverse expert-level annotations from numerous public datasets to train a foundation model capable of multiple tasks including classification localization and segmentation. To address the challenges of annotation and task heterogeneity we propose a Lock-Release pretraining strategy to enhance the cyclic learning from multiple datasets combined with the student-teacher learning paradigm ensuring the model retains general knowledge for all tasks while preventing overfitting to any single task. To demonstrate the effectiveness of Foundation X we trained a model using 11 chest X-ray datasets covering annotations for classification localization and segmentation tasks. Our experimental results show that Foundation X achieves notable performance gains through extensive annotation utilization excels in cross-dataset and cross-task learning and further enhances performance in organ localization and segmentation tasks. All code and pretrained models are publicly accessible at GitHub.com/JLiangLab/Foundation_X.

This paper tackles the intricate challenge of video question-answering (VideoQA). Despite notable progress current methods fall short of effectively integrating questions with video frames and semantic object-level abstractions to create question-aware video representations. We introduce Local - Global Question Aware Video Embedding (LGQAVE) which incorporates three major innovations to integrate multi-modal knowledge better and emphasize semantic visual concepts relevant to specific questions. LGQAVE moves beyond traditional ad-hoc frame sampling by utilizing a cross-attention mechanism that precisely identifies the most relevant frames concerning the questions. It captures the dynamics of objects within these frames using distinct graphs grounding them in question semantics with the miniGPT model. These graphs are processed by a question-aware dynamic graph transformer (Q-DGT) which refines the outputs to develop nuanced global and local video representations. An additional cross-attention module integrates these local and global embeddings to generate the final video embeddings which a language model uses to generate answers. Extensive evaluations across multiple benchmarks demonstrate that LGQAVE significantly outperforms existing models in delivering accurate multi-choice and open-ended answers.

Post-mortem iris recognition is an emerging application of iris-based human identification in a forensic setup. One factor that may be useful in conditioning iris recognition methods is the tissue decomposition level which is correlated with the post-mortem interval (PMI) i.e. the number of hours that have elapsed since death. PMI however is not always available and its precise estimation remains one of the core challenges in forensic examination. This paper presents the first known to us method of the PMI estimation directly from iris images captured after death. To assess the feasibility of the iris-based PMI estimation we designed models predicting the PMI from (a) near-infrared (NIR) (b) visible (RGB) and (c) multispectral (RGB+NIR) forensic iris images. Models were evaluated following a 10-fold cross-validation in (S1) sample-disjoint (S2) subject-disjoint and (S3) cross-dataset scenarios. We explore two data balancing techniques for S3: resampling-based balancing (S3-real) and synthetic data-supplemented balancing (S3-synthetic). We found that using the multispectral data offers a spectacularly low mean absolute error (MAE) of = 3.5 hours in the scenario (S1) a bit worse MAE = 17.5 hours in the scenario (S2) and MAE = 45.77 hours in the scenario (S3). Additionally supplementing the training set with synthetically-generated forensic iris images (S3-synthetic) significantly enhances the models' ability to generalize to new NIR RGB and multispectral data collected in a different lab. This suggests that if the environmental conditions are favorable (e.g. bodies are kept in low temperatures) forensic iris images provide features that are indicative of the PMI and can be automatically estimated.

Recent transformer-based methods achieve notable gains in the Human-object Interaction Detection (HOID) task by leveraging the detection of DETR and the prior knowledge of Vision-Language Model (VLM). However these methods suffer from extended training times and complex optimization due to the entanglement of object detection and HOI recognition during the decoding process. Especially the query embeddings used to predict both labels and boxes suffer from ambiguous representations and the gap between the prediction of HOI labels and verb labels is not considered. To address these challenges we introduce SOV-STG-VLA with three key components: Subject-Object-Verb (SOV) decoding Specific Target Guided (STG) denoising and a Vision-Language Advisor (VLA). Our SOV decoders disentangle object detection and verb recognition with a novel interaction region representation. The STG denoising strategy learns label embeddings with ground-truth information to guide the training and inference. Our SOV-STG achieves a fast convergence speed and high accuracy and builds a foundation for the VLA to incorporate the prior knowledge of the VLM. We introduce a vision advisor decoder to fuse both the interaction region information and the VLM's vision knowledge and a Verb-HOI prediction bridge to promote interaction representation learning. Our VLA notably improves our SOV-STG and achieves SOTA performance with one-sixth of training epochs compared to recent SOTA. Code and models are available at https://github.com/cjw2021/SOV-STG-VLA.

Synthesizing photo-realistic images of a scene from arbitrary viewpoints and under arbitrary lighting environments is one of the important research topics in computer vision and graphics. In this paper we propose a method for synthesizing photo-realistic images of a scene with fluorescent objects from novel viewpoints and under novel lighting colors. In general fluorescent materials absorb light with certain wavelengths and then emit light with longer wavelengths than the absorbed ones in contrast to reflective materials which preserve wavelengths of light. Therefore we cannot reproduce the colors of fluorescent objects under arbitrary lighting colors by combining conventional view synthesis techniques with the white balance adjustment of the RGB channels. Accordingly we extend the novel view synthesis based on the neural radiance fields by incorporating the superposition principle of light; our proposed method captures a sparse set of images of a scene from varying viewpoints and under varying light source colors by using a display-camera system and then synthesize photo-realistic images of the scene without explicitly modeling the geometric and photometric models of the scene. We conduct a number of experiments using real images and confirm the effectiveness of our method.

Traditional methods utilising data-independent acquisition mass spectrometry (DIA-MS) data for predictions depend on database searches against predefined spectral libraries for characterisation and quantification of the proteomes limiting scalability and adaptability across various applications. However directly applying existing networks on DIA-MS data represented as images for end-to-end predictions struggles to mine a predictive pattern due to non-uniform region importance across the image and divergences exhibited among different regions of the image. To overcome these limitations we propose a new framework with two modules: i) a dynamic sampling module that identifies regions of interest from the DIA-MS image constraining the network to focus on the most informative regions of the image only; ii) a mixture of experts module that sparsely routes the regions of interest to related expert networks facilitating adaptive computation of region features. The region features are then fused for predictions. Experimentally to benchmark our method we collected a large DIA-MS dataset of wheat for flowering time prediction and our approach significantly outperforms previous end-to-end methods i. e. 0.171 R2 improvements.

Deployment of deep neural networks in real-world settings typically requires adaptation to new tasks with few examples. Few-shot classification (FSC) provides a solution to this problem by leveraging pre-trained backbones for fast adaptation to new classes. However approaches for multi-domain FSC typically result in complex pipelines aimed at information fusion and task-specific adaptation without consideration of the importance of backbone training. In this work we introduce an effective strategy for backbone training and selection in multi-domain FSC by utilizing flatness-aware training and fine-tuning. Our work is theoretically grounded and empirically performs on par or better than state-of-the-art methods despite being simpler. Further our results indicate that backbone training is crucial for good generalisation in FSC across different adaptation methods.

Text-guided Video Temporal Grounding (VTG) aims to localize relevant segments in untrimmed videos based on textual descriptions encompassing two subtasks: Moment Retrieval (MR) and Highlight Detection (HD). Although previous typical methods have achieved commendable results it is still challenging to retrieve short video moments. This is primarily due to the reliance on sparse and limited decoder queries which significantly constrain the accuracy of predictions. Furthermore suboptimal outcomes often arise because previous methods rank predictions based on isolated predictions neglecting the broader video context. To tackle these issues we introduce FlashVTG a framework featuring a Temporal Feature Layering (TFL) module and an Adaptive Score Refinement (ASR) module. The TFL module replaces the traditional decoder structure to capture nuanced video content variations across multiple temporal scales while the ASR module improves prediction ranking by integrating context from adjacent moments and multi-temporal-scale features. Extensive experiments demonstrate that FlashVTG achieves state-of-the-art performance on four widely adopted datasets in both MR and HD. Specifically on the QVHighlights dataset it boosts mAP by 5.8% for MR and 3.3% for HD. For short-moment retrieval FlashVTG increases mAP to 125% of previous SOTA performance. All these improvements are made without adding training burdens underscoring its effectiveness.

Map-free LiDAR localization systems accurately localize within known environments by predicting sensor position and orientation directly from raw point clouds eliminating the need for large maps and descriptors. However their long training times hinder rapid adaptation to new environments. To address this we propose FlashMix which uses a frozen scene-agnostic backbone to extract local point descriptors aggregated with an MLP mixer to predict sensor pose. A buffer of local descriptors is used to accelerate training by orders of magnitude combined with metric learning or contrastive loss regularization of aggregated descriptors to improve performance and convergence. We evaluate FlashMix on various LiDAR localization benchmarks examining different regularizations and aggregators and demonstrating its effectiveness for rapid and accurate LiDAR localization in real-world scenarios. The code is available at https://github.com/raktimgg/FlashMix.