WACV 2025 论文列表

Medical image translation is the process of converting from one imaging modality to another in order to reduce the need for multiple image acquisitions from the same patient. This can enhance the efficiency of treatment by reducing the time equipment and labor needed. In this paper we introduce a multi-resolution guided Generative Adversarial Network (GAN)-based framework for 3D medical image translation. Our framework uses a 3D multi-resolution Dense-Attention UNet (3D-mDAUNet) as the generator and a 3D multi-resolution UNet as the discriminator optimized with a unique combination of loss functions including voxel-wise GAN loss and 2.5D perception loss. Our approach yields promising results in volumetric image quality assessment (IQA) across a variety of imaging modalities body regions and age groups demonstrating its robustness. Furthermore we propose a synthetic-to-real applicability assessment as an additional evaluation to assess the effectiveness of synthetic data in downstream applications such as segmentation. This comprehensive evaluation shows that our method produces synthetic medical images not only of high-quality but also potentially useful in clinical applications. Our code is available at github.com/juhha/3D-mADUNet.

Large language models (LLMs) pre-trained on vast amounts of text have shown remarkable abilities in understanding general knowledge and commonsense. Therefore it's desirable to leverage pre-trained LLM to help solve computer vision tasks. Previous works on multi-modal LLM mainly focus on the generation capability. In this work we propose LLM-augmented visual representation learning (LMVR). Our approach involves initially using a vision encoder to extract features which are then projected into the word embedding space of the LLM. The LLM then generates responses based on the visual representation and a text prompt. Finally we aggregate sequence-level features from the hidden layers of the LLM to obtain image-level representations. We conduct extensive experiments on multiple datasets and have the following findings: (a) LMVR outperforms traditional vision encoder on various downstream tasks and effectively learns the correspondence between words and image regions; (b) LMVR improves the generalizability compared to using a vision encoder alone as evidenced by its superior resistance to domain shift; (c) LMVR improves the robustness of models to corrupted and perturbed visual data. Our findings demonstrate LLM-augmented visual representation learning is effective as it learns object-level concepts and commonsense knowledge.

As a globally celebrated sport soccer has seen its appeal greatly amplified by engaging and vivid commentary. Recently Multi-Modal Large Language Models (MLLMs) have attracted attention in generating soccer commentaries due to their remarkable capacities of understanding different modalities of the input videos. Most of these methods have shown that the use of multiple modalities can enhance the commentary quality which includes video audio and structured meta-data. However delivering precise and rich commentary requires the ability to accurately discern subtle differences in similar backgrounds events and players. This presents a significant challenge for existing MLLMs. So we propose SoccerComment a framework for generating soccer commentary that integrates MLLMs with Retrieval-Augmented Generation (RAG) strategies. This framework enhances inference efficiency and reduces the need for continuous training through a multi-modal clustering memory unit and retrieval-augmented in-context learning mechanisms ultimately improving the accuracy and diversity of the commentary. Based on similar retrieved scenarios SoccerComment demonstrates outstanding zero-shot performance offering a new direction and scalable solution for future research in soccer commentary generation.

We propose a novel teacher-student framework to distill knowledge from multiple teachers trained on distinct datasets. Each teacher is first trained from scratch on its own dataset. Then the teachers are combined into a joint architecture which fuses the features of all teachers at multiple representation levels. The joint teacher architecture is fine-tuned on samples from all datasets thus gathering useful generic information from all data samples. Finally we employ a multi-level feature distillation procedure to transfer the knowledge to a student model for each of the considered datasets. We conduct image classification experiments on seven benchmarks and action recognition experiments on three benchmarks. To illustrate the power of our feature distillation procedure the student architectures are chosen to be identical to those of the individual teachers. To demonstrate the flexibility of our approach we combine teachers with distinct architectures. We show that our novel Multi-Level Feature Distillation (MLFD) can significantly surpass equivalent architectures that are either trained on individual datasets or jointly trained on all datasets at once. Furthermore we confirm that each step of the proposed training procedure is well motivated by a comprehensive ablation study. We publicly release our code at https://github.com/AdrianIordache/MLFD.

Despite the critical importance of the medical domain in Deep Learning most of the research in this area solely focuses on training models in static environments. It is only in recent years that research has begun to address dynamic environments and tackle the Catastrophic Forgetting problem through Continual Learning (CL) techniques. Previous studies have primarily focused on scenarios such as Domain Incremental Learning and Class Incremental Learning which do not fully capture the complexity of real-world applications. Therefore in this work we propose a novel benchmark combining the challenges of new class arrivals and domain shifts in a single framework by considering the New Instances and New Classes (NIC) scenario. This benchmark aims to model a realistic CL setting for the multi-label classification problem in medical imaging. Additionally it encompasses a greater number of tasks compared to previously tested scenarios. Specifically our benchmark consists of two datasets (NIH and CXP) nineteen classes and seven tasks a stream longer than the previously tested ones. To solve common challenges (e.g. the task inference problem) found in the CIL and NIC scenarios we propose a novel approach called Replay Consolidation with Label Propagation (RCLP). Our method surpasses existing approaches exhibiting superior performance with minimal forgetting.

Creating maps of the world around us is paramount to many applications including those related to robotics such as navigation and inspection. Given the computational resource limitations typical of robotic platforms there is a pressing need for lightweight 3D representations that capture detailed texture and geometric information with minimal storage. Traditional voxel-based approaches require substantial memory resources. On the other hand neural implicit and 3D Gaussian splatting representations require significant computational power (GPUs) and can hardly run in real-time. In this paper we introduce a novel scene representation Multi-HexPlanes that divides 3D environments into large boxes and utilizes the faces of the boxes to encapsulate texture and geometric information. This representation reduces the memory requirement to store the map making our approach especially suitable for systems with limited memory. Through extensive evaluations on large-scale datasets we find that our method achieves better performance on rendering and more complete 3D reconstruction. We also demonstrate that our map representation can output dense feature points with rich geometric information for downstream tasks such as training 3D Gaussian splats. The proposed technique promises substantial improvements in real-time 3D mapping applications particularly for devices constrained by processing power and storage.

With the advent of large pre-trained vision-language models such as CLIP prompt learning methods aim to enhance the transferability of the CLIP model. They learn the prompt given few samples from the downstream task given the specific class names as prior knowledge which we term as semantic-aware classification. However in many realistic scenarios we only have access to few samples and knowledge of the class names (e.g. when considering instances of classes). This challenging scenario represents the semantic-agnostic discriminative case. Text-to-Image (T2I) personalization methods aim to adapt T2I models to unseen concepts by learning new tokens and endowing these tokens with the capability of generating the learned concepts. These methods do not require knowledge of class names as a semantic-aware prior. Therefore in this paper we first explore Textual Inversion and reveal that the new concept tokens possess both generation and classification capabilities by regarding each category as a single concept. However learning classifiers from single-concept textual inversion is limited since the learned tokens are suboptimal for the discriminative tasks. To mitigate this issue we propose Multi-Class textual inversion which includes a discriminative regularization term for the token updating process. Using this technique our method MC-TI achieves stronger Semantic-Agnostic Classification while preserving the generation capability of these modifier tokens given only few samples per category. In the experiments we extensively evaluate MC-TI on 12 datasets covering various scenarios which demonstrates that MC-TI achieves superior results in terms of both classification and generation outcomes.

3D segmentation of biological structures is critical in biomedical imaging offering significant insights into structures and functions. This paper introduces a novel segmentation of biological images that couples Multi-Aperture representation with Transformers for 3D (MAT3D) segmentation. Our method integrates the global context-awareness of Transformer networks with the local feature extraction capabilities of Convolutional Neural Networks (CNNs) providing a comprehensive solution for accurately delineating complex biological structures. First we evaluated the performance of the proposed technique on two public clinical datasets of ACDC and Synapse multi-organ segmentation rendering superior Dice scores of 93.34+-0.05 and 89.73+-0.04 respectively with fewer parameters compared to the published literature. Next we assessed the performance of our technique on an organoid dataset comprising four breast cancer subtypes. The proposed method achieved a Dice 95.12+-0.02 and a PQ score of 97.01+-0.01 respectively. MAT3D also significantly reduces the parameters to 40 million. The code is available on https://github.com/sohaibcs1/MAT3D.

In the diverse field of medical imaging automatic segmentation has numerous applications and must handle a wide variety of input domains such as different types of Computed Tomography (CT) scans and Magnetic Resonance (MR) images. This heterogeneity challenges automatic segmentation algorithms to maintain consistent performance across different modalities due to the requirement for spatially aligned and paired images. Typically segmentation models are trained using a single modality which limits their ability to generalize to other types of input data without employing transfer learning techniques. Additionally leveraging complementary information from different modalities to enhance segmentation precision often necessitates substantial modifications to popular encoder-decoder designs such as introducing multiple branched encoding or decoding paths for each modality. In this work we propose a simple Multi-Modal Segmentation (MulModSeg) strategy to enhance medical image segmentation across multiple modalities specifically CT and MR. It incorporates two key designs: a modality-conditioned text embedding framework via a frozen text encoder that adds modality awareness to existing segmentation frameworks without significant structural modifications or computational overhead and an alternating training procedure that facilitates the integration of essential features from unpaired CT and MR inputs. Through extensive experiments with both Fully Convolutional Network and Transformer-based backbones MulModSeg consistently outperforms previous methods in segmenting abdominal multi-organ and cardiac substructures for both CT and MR modalities. The code is available in this link.

Self-supervised monocular depth estimation (MDE) has gained popularity for obtaining depth predictions directly from videos. However these methods often produce scale-invariant results unless additional training signals are provided. Addressing this challenge we introduce a novel self-supervised metric-scaled MDE model that requires only monocular video data and the camera's mounting position both of which are readily available in modern vehicles. Our approach leverages planar-parallax geometry to reconstruct scene structure. The full pipeline consists of three main networks a multi-frame network a single-frame network and a pose network. The multi-frame network processes sequential frames to estimate the structure of the static scene using planar-parallax geometry and the camera mounting position. Based on this reconstruction it acts as a teacher distilling knowledge such as scale information masked drivable area metric-scale depth for the static scene and dynamic object mask to the single-frame network. It also aids the pose network in predicting a metric-scaled relative pose between two subsequent images. Our method achieved state-of-the-art results for the driving benchmark KITTI for metric-scaled depth prediction. Notably it is one of the first methods to produce self-supervised metric-scaled depth prediction for the challenging Cityscapes dataset demonstrating its effectiveness and versatility. Project page: https://mono-pp.github.io/

Video Moment Retrieval is a common task to evaluate the performance of visual-language models - it involves localising start and end times of moments in videos from query sentences. The current task formulation assumes that the queried moment is present in the video resulting in false positive moment predictions when irrelevant query sentences are provided. In this paper we propose the task of Negative-Aware Video Moment Retrieval (NA-VMR) which considers both moment retrieval accuracy and negative query rejection accuracy. We make the distinction between In-Domain and Out-of-Domain negative queries and provide new evaluation benchmarks for two popular video moment retrieval datasets: QVHighlights and Charades-STA. We analyse the ability of current SOTA video moment retrieval approaches to adapt to Negative-Aware Video Moment Retrieval and propose UniVTG-NA an adaptation of UniVTG designed to tackle NA-VMR. UniVTG-NA achieves high negative rejection accuracy (avg. 98.4%) scores while retaining moment retrieval scores to within 3.87% Recall@1. Dataset splits are available at https://github.com/keflanagan/MomentofUntruth

In autonomous driving environment perception has significantly advanced with the utilization of deep learning techniques for diverse sensors such as cameras depth sensors or infrared sensors. The diversity in the sensor stack increases the safety and contributes to robustness against adverse weather and lighting conditions. However the variance in data acquired from different sensors poses challenges. In the context of continual learning (CL) incremental learning is especially challenging for considerably large domain shifts e.g. different sensor modalities. This amplifies the problem of catastrophic forgetting. To address this issue we formulate the concept of modality-incremental learning and examine its necessity by contrasting it with existing incremental learning paradigms. We propose the use of a modified Relevance Mapping Network (RMN) to incrementally learn new modalities while preserving performance on previously learned modalities in which relevance maps are disjoint. Experimental results demonstrate that the prevention of shared connections in this approach helps alleviate the problem of forgetting within the constraints of a strict continual learning framework.

We introduce MoRAG a novel multi-part fusion based retrieval-augmented generation strategy for text-based human motion generation. The method enhances motion diffusion models by leveraging additional knowledge obtained through an improved motion retrieval process. By effectively prompting large language models (LLMs) we address spelling errors and rephrasing issues in motion retrieval. Our approach utilizes a multi-part retrieval strategy to improve the generalizability of motion retrieval across the language space. We create diverse samples through the spatial composition of the retrieved motions. Furthermore by utilizing low-level part-specific motion information we can construct motion samples for unseen text descriptions. Our experiments demonstrate that our framework can serve as a plug-and-play module improving the performance of motion diffusion models. Code pre-trained models and sample videos are available at https://motion-rag.github.io.

In real-world scenarios using multiple modalities like visible (RGB) and infrared (IR) can greatly improve the performance of a predictive task such as object detection (OD). Multimodal learning is a common way to leverage these modalities where multiple modality-specific encoders and a fusion module are used to improve performance. In this paper we tackle a different way to employ RGB and IR modalities where only one modality or the other is observed by a single shared vision encoder. This realistic setting requires a lower memory footprint and is more suitable for applications such as autonomous driving and surveillance which commonly rely on RGB and IR data. However when learning a single encoder on multiple modalities one modality can dominate the other producing uneven recognition results. This work investigates how to efficiently leverage RGB and IR modalities to train a common transformer-based OD vision encoder while countering the effects of modality imbalance. For this we introduce a novel training technique to Mix Patches (MiPa) from the two modalities in conjunction with a patch-wise modality agnostic module for learning a common representation of both modalities. Our experiments show that MiPa can learn a representation to reach competitive results on traditional RGB/IR benchmarks while only requiring a single modality during inference. Our code is available at: https://github.com/heitorrapela/MiPa.

This paper introduces MixDiff a new self-supervised learning (SSL) pre-training framework that combines real and synthetic images. Unlike traditional SSL methods that predominantly use real images MixDiff uses a variant of Stable Diffusion to replace an augmented instance of a real image facilitating the learning of cross real-synthetic image representations. Our key insight is that while models trained solely on synthetic images underperform combining real and synthetic data leads to more robust and adaptable representations. Experiments show MixDiff enhances SimCLR BarlowTwins and DINO across various robustness datasets and domain transfer tasks boosting SimCLR's ImageNet-1K accuracy by 4.56%. Our framework also demonstrates comparable performance without needing any augmentations a surprising finding in SSL where augmentations are typically crucial. Furthermore MixDiff achieves similar results to SimCLR while requiring less real data highlighting its efficiency in representation learning

In the context of escalating safety concerns across various domains the tasks of Video Anomaly Detection (VAD) and Video Anomaly Recognition (VAR) have emerged as critically important for applications in intelligent surveillance evidence investigation violence alerting etc. These tasks aimed at identifying and classifying deviations from normal behavior in video data face significant challenges due to the rarity of anomalies which leads to extremely imbalanced data and the impracticality of extensive frame-level data annotation for supervised learning. This paper introduces a novel hierarchical graph neural network (GNN) based model MissionGNN that addresses these challenges by leveraging a state-of-the-art large language model and a comprehensive knowledge graph for efficient weakly supervised learning in VAR. Our approach circumvents the limitations of previous methods by avoiding heavy gradient computations on large multimodal models and enabling fully frame-level training without fixed video segmentation. Utilizing automated mission-specific knowledge graph generation our model provides a practical and efficient solution for real-time video analysis without the constraints of previous segmentation-based or multimodal approaches. Experimental validation on benchmark datasets demonstrates our model's performance in VAD and VAR highlighting its potential to redefine the landscape of anomaly detection and recognition in video surveillance systems.

Recently object counting has shifted towards class-agnostic counting (CAC) which counts instances of arbitrary object classes never seen during model training. With advancements in robust vision-and-language foundation models there is a growing interest in prompt-based CAC where object categories are specified using natural language. However we identify significant limitations in current benchmarks for evaluating this task which hinder both accurate assessment and the development of more effective solutions. Specifically we argue that the current evaluation protocols do not measure the ability of the model to understand which object has to be counted. This is due to two main factors: (i) the shortcomings of CAC datasets which primarily consist of images containing objects from a single class and (ii) the limitations of current counting performance evaluators which are based on traditional class-specific counting and focus solely on counting errors. To fill this gap we introduce the Prompt-Aware Counting (PrACo) benchmark. It comprises two targeted tests coupled with evaluation metrics specifically designed to quantitatively measure the robustness and trustworthiness of existing prompt-based CAC models. We evaluate state-of-the-art methods and demonstrate that although some achieve impressive results on standard class-specific counting metrics they exhibit a significant deficiency in understanding the input prompt indicating the need for more careful training procedures or revised designs. The code for reproducing our results is available at https://github.com/ciampluca/PrACo.

While HDMaps are a crucial component of autonomous driving they are expensive to acquire and maintain. Estimating these maps from sensors therefore promises to significantly lighten costs. These estimations however overlook existing HDMaps with current methods at most geolocalizing low quality maps or considering a general database of known maps. In this paper we propose to account for existing maps of the precise situation studied when estimating HDMaps. To prove this we identify 3 reasonable types of useful existing maps (minimalist noisy and outdated). We then introduce MapEX a novel online HDMap estimation framework that accounts for existing maps. MapEX achieves this by encoding map elements into query tokens and by refining the matching algorithm used to train classic query based map estimation models. We demonstrate that MapEX brings significant improvements on the nuScenes dataset. For instance MapEX - given noisy maps - improves by 38% over the MapTRv2 detector it is based on and by 8% over the current SOTA.

Gait recognition is an important biometric technique over large distances. State-of-the-art gait recognition systems perform very well in controlled environments at close range. Recently there has been an increased interest in gait recognition in the wild prompted by the collection of outdoor more challenging datasets containing variations in terms of illumination pitch angles and distances. An important problem in these environments is that of occlusion where the subject is partially blocked from camera view. While important this problem has received little attention. Thus we propose MimicGait a model-agnostic approach for gait recognition in the presence of occlusions. We train the network using a multi-instance correlational distillation loss to capture both inter-sequence and intra-sequence correlations in the occluded gait patterns of a subject utilizing an auxiliary Visibility Estimation Network to guide the training of the proposed mimic network. We demonstrate the effectiveness of our approach on challenging real-world datasets like GREW Gait3D and BRIAR. The code is available at https://github.com/Ayush-00/mimicgait.