WACV 2025 Papers

Infographics represent a key component of any blog or article facilitating effective communication of ideas while fostering reader engagement. However many content creators possess limited expertise in crafting visually striking infographics. This gap is effectively addressed by our proposed pipeline designed to aid writers in generating compelling infographics tailored to their written content. Our pipeline uses textual content and tabular data from the blog to generate anchor plots. Leveraging LLM for prompt generation the pipeline integrates the generated prompts with these anchor plots through an Image to Image (I2I) generation model. We observe that the majority of the resulting images generated using this approach align with the article's narrative and effectively represent the underlying tabular data. Additionally we introduce our proposed AADaT (Aesthetical Adherence to Data and Text) Score adept at comprehensively assessing aesthetics textual alignment data fidelity and overall image quality concurrently. In comparative evaluations our pipeline has demonstrated around 15% superior performance relative to state-of-the-art models such as DALLE and Stable Diffusion Large by showcasing much better data adherence and aesthetics. While state-of-the-art models excel in some metrics but falter in others our pipeline demonstrates a balanced performance across all metrics.

Can Visual Question Answering (VQA) systems maintain their performance when deployed in the real world? Or are they susceptible to realistic corruption effects e.g. image blur which can be detrimental in sensitive applications such as medical VQA? While linguistic robustness has been thoroughly explored within the VQA literature there has yet to be any significant work on visual robustness. In this work we present the first large-scale benchmark to evaluate the visual robustness of VQA models including multimodal large language models and zero-shot evaluation and assess the strength of the realistic corruption effects. Additionally we have designed several robustness evaluation metrics that quantify an aspect of robustness. These sub-metrics can be aggregated into a unified metric and tailored to fit a variety of use cases. The experiments reveal important insights into the relationships between model size accuracy and robustness against the visual corruptions. Our benchmark highlights the need for a balanced approach in model development that considers model performance without compromising robustness.

Fixed-wing aerial vehicles are equipped with functionalities such as ILS (instrument landing system) PAR (precision approach radar) and DGPS (differential global positioning system) enabling fully automated landings. However these systems impose significant costs on airport operations due to high installation and maintenance requirements. Moreover since these navigation parameters come from ground or satellite signals they are vulnerable to interference. A more cost-effective and independent alternative for guiding landing is a vision-based system that detects the runway and aligns the aircraft reducing the pilot's cognitive load. This paper proposes a novel framework that addresses three key challenges in developing autonomous vision-based landing systems. Firstly to overcome the lack of aerial front-view video data we created high-quality videos simulating landing approaches through the generator code available in the LARD (landing approach runway detection dataset) repository. Secondly in contrast to former studies focusing on object detection for finding the runway we chose the state-of-the-art model LoRAT to track runways within bounding boxes in each video frame. Thirdly to align the aircraft with the designated landing runway we extract runway keypoints from the resulting LoRAT frames and estimate the camera relative pose via the Perspective-n-Point algorithm. Our experimental results over a dataset of generated videos and original images from the LARD dataset consistently demonstrate the proposed framework's highly accurate tracking and alignment capabilities. Our approach source code and the LoRAT model pre-trained with LARD videos are available at https://github.com/jpklock2/vision-based-landing-guidance

Referring image segmentation is a challenging task that involves generating pixel-wise segmentation masks based on natural language descriptions. The complexity of this task increases with the intricacy of the sentences provided. Existing methods have relied mostly on visual features to generate the segmentation masks while treating text features as supporting components. However this under-utilization of text understanding limits the model's capability to fully comprehend the given expressions. In this work we propose a novel framework that specifically emphasizes object and context comprehension inspired by human cognitive processes through Vision-Aware Text Features. Firstly we introduce a CLIP Prior module to localize the main object of interest and embed the object heatmap into the query initialization process. Secondly we propose a combination of two components: Contextual Multimodal Decoder and Meaning Consistency Constraint to further enhance the coherent and consistent interpretation of language cues with the contextual understanding obtained from the image. Our method achieves significant performance improvements on three benchmark datasets RefCOCO RefCOCO+ and G-Ref. Project page: https://vatex.hkustvgd.com.

Recent video inpainting methods have achieved encouraging improvements by leveraging optical flow to guide pixel propagation from reference frames either in the image space or feature space. However they would produce severe artifacts when the masked area is too large and no pixel correspondences could be found. Recently denoising diffusion models have demonstrated impressive performance in generating diverse and high-quality images and have been exploited in a number of works for image inpainting. These methods however cannot be applied directly to videos to produce temporal-coherent inpainting results. In this paper we propose a training-free framework named VipDiff for conditioning diffusion model on the reverse diffusion process to produce temporal-coherent inpainting results without requiring any training data or fine-tuning the pre-trained models. VipDiff takes optical flow as guidance to extract valid pixels from reference frames to serve as constraints in optimizing the randomly sampled Gaussian noise and uses the generated results for further pixel propagation and conditional generation. VipDiff also allows for generating diverse video inpainting results over different sampled noise. Experiments demonstrate that our VipDiff outperforms state-of-the-art methods in terms of both spatial-temporal coherence and fidelity.

Musicians delicately control their bodies to generate music. Sometimes their motions are too subtle to be captured by the human eye. To analyze how they move to produce the music we need to estimate precise 4D human pose (3D pose over time). However current state-of-the-art (SoTA) visual pose estimation algorithms struggle to produce accurate monocular 4D poses because of occlusions partial views and human-object interactions. They are limited by the viewing angle pixel density and sampling rate of the cameras and fail to estimate fast and subtle movements such as in the musical effect of vibrato. We leverage the direct causal relationship between the music produced and the human motions creating them to address these challenges. We propose VioPose: a novel multimodal network that hierarchically estimates dynamics. High-level features are cascaded to low-level features and integrated into Bayesian updates. Our architecture is shown to produce accurate pose sequences facilitating precise motion analysis and outperforms SoTA. As part of this work we collected the largest and the most diverse calibrated violin-playing dataset including video sound and 3D motion capture poses. Code and dataset can be found in our project page https://sj-yoo.info/viopose/.

Large multimodal models known as LMMs hold substantial promise across various domains from personal assistance in daily tasks to sophisticated applications like medical diagnostics. However their capabilities have limitations in the video game domain including challenges with scene understanding hallucinations and inaccurate descriptions of video game content especially in open-source models. This paper describes the development of VideoGameBunny a LLaVA-style model based on Bunny specifically tailored for understanding images from video games. We release intermediate checkpoints training logs and an extensive dataset comprising 185259 video game images from 413 titles along with 389565 image-instruction pairs that include image captions question-answer pairs and a JSON representation of 16 elements of 136974 images. Our experiments show that our high-quality game-related data has the potential to make a relatively small model outperform the much larger state-of-the-art model LLaVa-1.6-34b which has more than four times the number of parameters. Our study paves the way for future research in video game understanding on tasks such as playing commentary and debugging. Code and data are available at videogamebunny.github.io

The demand for privacy in facial image dissemination is gaining ground internationally echoed by the proliferation of regulations such as GDPR DPDPA CCPA PIPL and APPI. While recent advances in anonymization surpass pixelation or blur methods additional constraints to the task pose challenges. Largely unaddressed by current anonymization methods are clinical images and pairs of before-and-after clinical images illustrating facial medical interventions e.g. facial surgeries or dental procedures. We present VerA the first Versatile Anonymization framework that solves two challenges in clinical applications: A) it preserves selected semantic areas (e.g. mouth region) to show medical intervention results that is anonymization is only applied to the areas outside the preserved area; and B) it produces anonymized images with consistent personal identity across multiple photographs which is crucial for anonymizing photographs of the same person taken before and after a clinical intervention. We validate our results on both single and paired anonymization of clinical images through extensive quantitative and qualitative evaluation. We also demonstrate that VerA reaches the state of the art on established anonymization tasks in terms of photorealism and de-identification.

Novel View Synthesis (NVS) which tries to produce a realistic image at the target view given source view images and their corresponding poses is a fundamental problem in 3D Vision. As this task is heavily under-constrained some recent work like Zero123 [18] tries to solve this problem with generative modeling specifically using pre-trained diffusion models. Although this strategy generalizes well to new scenes compared to neural radiance field-based methods it offers low levels of flexibility. For example it can only accept a single-view image as input despite realistic applications often offering multiple input images. This is because the source-view images and corresponding poses are processed separately and injected into the model at different stages. Thus it is not trivial to generalize the model into multi-view source images once they are available. To solve this issue we try to process each pose image pair separately and then fuse them as a unified visual representation which will be injected into the model to guide image synthesis at the target-views. However inconsistency and computation costs increase as the number of input source-view images increases. To solve these issues the Multi-view Cross Former module is proposed which maps variable-length input data to fix-size output data. A two-stage training strategy is introduced to further improve the efficiency during training time. Qualitative and quantitative evaluation over multiple datasets demonstrates the effectiveness of the proposed method against previous approaches. The code will be released according to the acceptance.

We present a framework for learning to generate background music from video inputs. Unlike existing works that rely on symbolic musical annotations which are limited in quantity and diversity our method leverages large-scale web videos accompanied by background music. This enables our model to learn to generate realistic and diverse music. To accomplish this goal we develop a generative video-music Transformer with a novel semantic video-music alignment scheme. Our model uses a joint autoregressive and contrastive learning objective which encourages the generation of music aligned with high-level video content. We also introduce a novel video-beat alignment scheme to match the generated music beats with the low-level motions in the video. Lastly to capture fine-grained visual cues in a video needed for realistic background music generation we introduce a new temporal video encoder architecture allowing us to efficiently process videos consisting of many densely sampled frames. We train our framework on our newly curated DISCO-MV dataset consisting of 2.2M video-music samples which is orders of magnitude larger than any prior datasets used for video music generation. Our method outperforms existing approaches on the DISCO-MV and MusicCaps datasets according to various music generation evaluation metrics including human evaluation. Results are available at https://genjib.github.io/project_page/VMAs/index.html

Gait recognition plays a vital role in biometric applications by analyzing the unique characteristics of an individual's walking pattern. Methods based on 2D representations such as silhouettes and skeletons are increasingly being developed to learn the shape features and joint dynamic movements. Nevertheless the effectiveness of 2D representation-based methods is impeded by factors such as changes in viewpoint partial occlusion and noisy environments. 3D representation-based methods can complement 2D representation-based approaches by providing more precise dynamic body shapes and motion information along with increased robustness against changes in viewpoint and partial occlusion. However the complexity of acquiring accurate 3D representations and the challenges associated with extracting dynamic topological features from sequences of 3D representations hinder the development of 3D representations-based methods. In this paper we present VM-Gait a novel multi-modal gait recognition framework that harnesses the advantages of integrating both 2D and 3D representations. Furthermore we introduce a new 3D representation Virtual Marker into gait recognition to efficiently learn topological features from 3D representations avoiding the computational complexities inherent in directly learning from 3D representations like 3D meshes or 3D point clouds. Extensive experiments demonstrate that the proposed framework effectively learns and fuses discriminative information from different gait modalities enhancing gait recognition performance.

Vision Transformers (ViTs) have emerged as the backbone of many segmentation models consistently achieving state-of-the-art (SOTA) performance. However their success comes at a significant computational cost. Image token pruning is one of the most effective strategies to address this complexity. However previous approaches fall short when applied to more complex task-oriented segmentation (TOS) where the class of each image patch is not predefined but dependent on the specific input task. This work introduces the Vision Language Guided Token Pruning (VLTP) a novel token pruning mechanism that can accelerate ViT-based segmentation models particularly for TOS guided by multi-modal large language model (MLLM). We argue that ViT does not need to process every image token through all of its layers--only the tokens related to reasoning tasks are necessary. We design a new pruning decoder to take both image tokens and vision-language guidance as input to predict the relevance of each image token to the task. Only image tokens with high relevance are passed to deeper layers of the ViT. Experiments show that the VLTP framework reduces the computational costs of ViT by approximately 25% without performance degradation and by around 40% with only a 1% performance drop. The code associated with this study can be found at this URL.

Image denoising mitigates noise from the captured images and thereby enhances the efficacy of high-demand vision applications such as classification and segmentation. Hyperspectral Images (HSIs) with their multiple spectral bands provide valuable information and make them highly applicable in real-world applications. Current Deep Learning methods mainly employ Transformers to denoise HSIs spatially and spectrally through self-attention (SA). However SA focuses on individual samples and overlooks potential correlations within the images indicating room for improvement. Moreover existing Transformer-based denoising methods often fail to appropriately balance the importance of spatial and spectral features. This paper presents a novel method VISIONARY to address these issues by obtaining better HSI feature representation. To this end it introduces the Spatial-Spectral-Cubic Transformer (SSCformer) block to address the shortcomings of Transformers in HSI denoising particularly their inability to capture correlations within images of the same type by introducing Global Feature Attention (GFA). Additionally the SSCformer independently determines the optimal weightage for spatial and spectral features using attention mechanisms leading to more effective denoising. Our method VISIONARY is based on the integration of Transformer U-Net and CNN architecture. Experimental results demonstrate that VISIONARY outperforms well-known methods on publicly available datasets and our SSCformer block can be easily integrated with existing Transformer-based HSI denoising methods to improve their efficacy.

Person Re-identification is a research area with significant real world applications. Despite recent progress existing methods face challenges in robust re-identification in the wild e.g. by focusing only on a particular modality and on unreliable patterns such as clothing. A generalized method is highly desired but remains elusive to achieve due to issues such as the trade-off between spatial and temporal resolution and inaccurate feature extraction. We propose VILLS (Video-Image Learning to Learn Semantics) a self-supervised method that jointly learns spatial and temporal features from images and videos. VILLS first designs a local semantic extraction module that adaptively extracts semantically consistent and robust spatial features. Then VILLS designs a unified feature learning and adaptation module to represent image and video modalities in a consistent feature space. By Leveraging self-supervised large-scale pre-training VILLS establishes a new State-of-The-Art that significantly outperforms existing image and video-based methods.

Images captured in severe low-light circumstances often suffer from significant information absence. Existing singular modality image enhancement methods struggle to restore image regions lacking valid information. By leveraging light-impervious infrared images visible and infrared image fusion methods have the potential to reveal information hidden in darkness. However they primarily emphasize inter-modal complementation but neglect intra-modal enhancement limiting the perceptual quality of output images. To address these limitations we propose a novel task dubbed visible and infrared information synthesis (VIIS) which aims to achieve both information enhancement and fusion of the two modalities. Given the difficulty in obtaining ground truth in the VIIS task we design an information synthesis pretext task (ISPT) based on image augmentation. We employ a diffusion model as the framework and design a sparse attention-based dual-modalities residual (SADMR) conditioning mechanism to enhance information interaction between the two modalities. This mechanism enables features with prior knowledge from both modalities to adaptively and iteratively attend to each modality's information during the denoising process. Our extensive experiments demonstrate that our model qualitatively and quantitatively outperforms not only the state-of-the-art methods in relevant fields but also the newly designed baselines capable of both information enhancement and fusion. The code is available at https://github.com/Chenz418/VIIS.

We present a stereo-matching method for depth estimation from high-resolution images using visual hulls as priors and a memory-efficient technique for the correlation computation. Our method uses object masks extracted from supplementary views of the scene to guide the disparity estimation effectively reducing the search space for matches. This approach is specifically tailored to stereo rigs in volumetric capture systems where an accurate depth plays a key role in the downstream reconstruction task. To enable training and regression at high resolutions targeted by recent systems our approach extends a sparse correlation computation into a hybrid sparse-dense scheme suitable for application in leading recurrent network architectures. We evaluate the performance-efficiency trade-off of our method compared to state-of-the-art approaches and demonstrate the efficacy of the visual hull guidance. In addition we propose a training scheme for a further reduction of memory requirements during optimization facilitating training on high-resolution data.

Visual Geo-localization (VG) is a critical research area for identifying geo-locations from visual inputs particularly in autonomous navigation for robotics and vehicles. Current VG methods often learn feature extractors from geo-labeled images to create dense geographically relevant representations. Recent advances in Self-Supervised Learning (SSL) have demonstrated its capability to achieve performance on par with supervised techniques with unlabeled images. This study presents a novel VG-SSL framework designed for versatile integration and benchmarking of diverse SSL methods for representation learning in VG featuring a unique geo-related pair strategy GeoPair. Through extensive performance analysis we adapt SSL techniques to improve VG on datasets from hand-held and car-mounted cameras used in robotics and autonomous vehicles. Our results show that contrastive learning and information maximization methods yield superior geo-specific representation quality matching or surpassing the performance of state-of-the-art VG techniques. To our knowledge This is the first benchmarking study of SSL in VG highlighting its potential in enhancing geo-specific visual representations for robotics and autonomous vehicles. The code is publicly available at https://github.com/arplaboratory/VG-SSL.

Input aggregation is a simple technique used by state-of-the-art LiDAR 3D object detectors to improve detection. However increasing aggregation is known to have diminishing returns and even performance degradation due to objects responding differently to the number of aggregated frames. To address this limitation we propose an efficient adaptive method which we call Variable Aggregation Detection (VADet). Instead of aggregating the entire scene using a fixed number of frames VADet performs aggregation per object with the number of frames determined by an object's observed properties such as speed and point density. VADet thus reduces the inherent trade-offs of fixed aggregation and is not architecture specific. To demonstrate its benefits we apply VADet to three popular single-stage detectors and achieve state-of-the-art performance on the Waymo dataset.

The field of indoor monocular 3D object detection is gaining significant attention fueled by the increasing demand in VR/AR and robotic applications. However its advancement is impeded by the limited availability and diversity of 3D training data owing to the labor-intensive nature of 3D data collection and annotation processes. In this paper we present V-MIND (Versatile Monocular INdoor Detector) which enhances the performance of indoor 3D detectors across a diverse set of object classes by harnessing publicly available large-scale 2D datasets. By leveraging well-established monocular depth estimation techniques and camera intrinsic predictors we can generate 3D training data by converting large-scale 2D images into 3D point clouds and subsequently deriving pseudo 3D bounding boxes. To mitigate distance errors inherent in the converted point clouds we introduce a novel 3D self-calibration loss for refining the pseudo 3D bounding boxes during training. Additionally we propose a novel ambiguity loss to address the ambiguity that arises when introducing new classes from 2D datasets. Finally through joint training with existing 3D datasets and pseudo 3D bounding boxes derived from 2D datasets V-MIND achieves state-of-the-art object detection performance across a wide range of classes on the Omni3D indoor dataset.