WACV 2025 论文列表

Long-range image understanding is a challenging task for computer vision due to the presence of atmospheric turbulence. Turbulence can degrade image quality (blur and geometric distortion) due to the medium's spatio-temporal varying index of refraction bending light rays. The strength of atmospheric turbulence is quantified by the refractive index structure parameter Cn2 and estimating it is important both as an indicator of image degradation and is useful for downstream tasks including video restoration and estimating true shape and range/depth. However traditional methods for estimating Cn2 involve expensive and complex optical equipment limiting their practicality. In this paper we propose MetaVIn: a Meteorological and Visual Integration system to predict atmospheric turbulence strength. Our method leverages image quality metrics to capture sharpness and blur combined with meteorological information within a Kolmogorov Arnold Network (KAN). We demonstrate that this approach provides a more accurate and generalizable estimation of Cn2 outperforming previous state-of-the-art methods in both blind image quality assessment and passive video-based turbulence strength estimation on a large dataset of 35364 image samples with accompanying ground truth scintillometer measurements for Cn2. Our method enables better prediction and mitigation of atmospheric image degradation while being useful in applications such as shape and range estimation enhancing the practical utility of our approach.

Recent object detection models for infrared (IR) imagery are based upon deep neural networks (DNNs) and require large amounts of labeled training imagery. However publicly available datasets that can be used for such training are limited in their size and diversity. To address this problem we explore cross-modal style transfer (CMST) to leverage large and diverse color imagery datasets so that they can be used to train DNN-based IR image-based object detectors. We evaluate six contemporary stylization methods on four publicly-available IR datasets - the first comparison of its kind - and find that CMST is highly effective for DNN-based detectors. Surprisingly we find that existing data-driven methods are outperformed by a simple grayscale stylization (an average of the color channels). Our analysis reveals that existing data-driven methods are either too simplistic or introduce significant artifacts into the imagery. To overcome these limitations we propose meta-learning style transfer (MLST) which learns a stylization by composing and tuning well-behaved analytic functions. We find that MLST leads to more complex stylizations without introducing significant image artifacts and achieves the best overall detector performance on our benchmark datasets.

In practice domain shifts are likely to occur between training and test data necessitating domain adaptation (DA) to adjust the pre-trained source model to the target domain. Recently universal domain adaptation (UniDA) has gained attention for addressing the possibility of an additional category (label) shift between the source and target domain. This means new classes can appear in the target data some source classes may no longer be present or both at the same time. For practical applicability UniDA methods must handle both source-free and online scenarios enabling adaptation without access to the source data and performing batch-wise updates in parallel with prediction. In an online setting preserving knowledge across batches is crucial. However existing methods often require substantial memory which is impractical because memory is limited and valuable in particular on embedded systems. Therefore we consider memory-efficiency as an additional constraint. To achieve memory-efficient online source-free universal domain adaptation (SF-UniDA) we propose a novel method that continuously captures the distribution of known classes in the feature space using a Gaussian mixture model (GMM). This approach combined with entropy-based out-of-distribution detection allows for the generation of reliable pseudo-labels. Finally we combine a contrastive loss with a KL divergence loss to perform the adaptation. Our approach not only achieves state-of-the-art results in all experiments on the DomainNet and Office-Home datasets but also significantly outperforms the existing methods on the challenging VisDA-C dataset setting a new benchmark for online SF-UniDA. Our code is available at https://github.com/pascalschlachter/GMM.

Contrastive learning has significantly improved representation quality enhancing knowledge transfer across tasks in continual learning (CL). However catastrophic forgetting remains a key challenge as contrastive based methods primarily focus on "soft relationships" or "softness" between samples which shift with changing data distributions and lead to representation overlap across tasks. Recently the newly identified Neural Collapse phenomenon has shown promise in CL by focusing on "hard relationships" or "hardness" between samples and fixed prototypes. However this approach overlooks "softness" crucial for capturing intra-class variability and this rigid focus can also pull old class representations toward current ones increasing forgetting. Building on these insights we propose Focal Neural Collapse Contrastive (FNC^2) a novel representation learning loss that effectively balances both soft and hard relationships. Additionally we introduce the Hardness-Softness Distillation (HSD) loss to progressively preserve the knowledge gained from these relationships across tasks. Our method outperforms state-of-the-art approaches particularly in minimizing memory reliance. Remarkably even without the use of memory our approach rivals rehearsal-based methods offering a compelling solution for data privacy concerns.

High-definition (HD) maps provide environmental information for autonomous driving systems and are essential for safe planning. While existing methods with single-frame input achieve impressive performance for online vectorized HD map construction they still struggle with complex scenarios and occlusions. We propose MemFusionMap a novel temporal fusion model with enhanced temporal reasoning capabilities for online HD map construction. Specifically we contribute a working memory fusion module that improves the model's memory capacity to reason across a history of frames. We also design a novel temporal overlap heatmap to explicitly inform the model about the temporal overlap information and vehicle trajectory in the Bird's Eye View space. By integrating these two designs MemFusionMap significantly outperforms existing methods while also maintaining a versatile design for scalability. We conduct extensive evaluation on open-source benchmarks and demonstrate a maximum improvement of 5.4% in mAP over state-of-the-art methods. The project page for MemFusionMap is https://song-jingyu.github.io/MemFusionMap.

Diffusion models excel in generating images that closely resemble their training data but are also susceptible to data memorization raising privacy ethical and legal concerns particularly in sensitive domains such as medical imaging. We hypothesize that this memorization stems from the overparameterization of deep models and propose that regularizing model capacity during fine-tuning can mitigate this issue. Firstly we empirically show that regulating the model capacity via Parameter-efficient fine-tuning (PEFT) mitigates memorization to some extent however it further requires the identification of the exact parameter subsets to be fine-tuned for high-quality generation. To identify these subsets we introduce a bi-level optimization framework MemControl that automates parameter selection using memorization and generation quality metrics as rewards during fine-tuning. The parameter subsets discovered through MemControl achieve a superior tradeoff between generation quality and memorization. For the task of medical image generation our approach outperforms existing state-of-the-art memorization mitigation strategies by fine-tuning as few as 0.019% of model parameters. Moreover we demonstrate that the discovered parameter subsets are transferable to non-medical domains. Our framework is scalable to large datasets agnostic to reward functions and can be integrated with existing approaches for further memorization mitigation. To the best of our knowledge this is the first study to empirically evaluate memorization in medical images and propose a targeted yet universal mitigation strategy. The code is available at https://github.com/Raman1121/Diffusion_Memorization_HPO

Diffusion models have emerged as frontrunners in text-to-image generation but their fixed image resolution during training often leads to challenges in high-resolution image generation such as semantic deviations and object replication. This paper introduces MegaFusion a novel approach that extends existing diffusion-based text-to-image models towards efficient higher-resolution generation without additional fine-tuning or adaptation. Specifically we employ an innovative truncate and relay strategy to bridge the denoising processes across different resolutions allowing for high-resolution image generation in a coarse-to-fine manner. Moreover by integrating dilated convolutions and noise re-scheduling we further adapt the model's priors for higher resolution. The versatility and efficacy of MegaFusion make it universally applicable to both latent-space and pixel-space diffusion models along with other derivative models. Extensive experiments confirm that MegaFusion significantly boosts the capability of existing models to produce images of megapixels and various aspect ratios while only requiring about 40% of the original computational cost. Code is available at https://haoningwu3639.github.io/MegaFusion/.

Magnetic Resonance Imaging (MRI) is instrumental in clinical diagnosis offering diverse contrasts that provide comprehensive diagnostic information. However acquiring multiple MRI contrasts is often constrained by high costs long scanning durations and patient discomfort. Current synthesis methods typically focused on single-image contrasts fall short in capturing the collective nuances across various contrasts. Moreover existing methods for multi-contrast MRI synthesis often fail to accurately map feature level information across multiple imaging contrasts. We introduce McCaD (Multi-Contrast MRI Conditioned Adaptive Adversarial Diffusion) a novel framework leveraging an adversarial diffusion model conditioned on multiple contrasts for high-fidelity MRI synthesis. McCaD significantly enhances synthesis accuracy by employing a multiscale feature-guided mechanism incorporating denoising and semantic encoders. An adaptive feature maximization strategy and a spatial feature-attentive loss have been introduced to capture more intrinsic features across multiple contrasts. This facilitates a precise and comprehensive feature-guided denoising process. Extensive experiments on tumor and healthy multi-contrast MRI datasets demonstrated that the McCaD outperforms state-of-the-art baselines quantitively and qualitatively. The code is available at https://github.com/sanuwanihewa/McCaD.

Achieving accurate material segmentation for 3-channel RGB images is challenging due to the considerable variation in material appearance. Hyperspectral images which are sets of spectral measurements sampled at multiple wavelengths theoretically offer distinct information for material identification as variations in the intensity of electromagnetic radiation reflected by a surface depend on the material composition of a scene. However existing hyperspectral datasets are impoverished in terms of the number of images and material categories for the dense material segmentation task and collecting and annotating hyperspectral images with a spectral camera is prohibitively expensive. To address this we propose a novel model the MatSpectNet to segment materials with recovered hyperspectral images from RGB images. The network leverages the principles of colour perception in modern cameras to constrain the reconstructed hyperspectral images and employs the domain adaptation method to generalise the hyperspectral reconstruction capability from a spectral recovery dataset to material segmentation datasets. The reconstructed hyperspectral images are further filtered using learnt response curves and enhanced with human perception. The performance of MatSpectNet is evaluated on the LMD dataset as well as the OpenSurfaces dataset. Our experiments demonstrate that MatSpectNet attains a 1.60% increase in average pixel accuracy and a 3.42% improvement in mean class accuracy compared with the most recent publication. Additional experiments and the project code are published on https://github.com/heng-yuwen/MatSpectNet.

Video tasks are compute-heavy and thus pose a challenge when deploying in real-time applications particularly for tasks that require state-of-the-art Vision Transformers (ViTs). Several research efforts have tried to address this challenge by leveraging the fact that large portions of the video undergo very little change across frames leading to redundant computations in frame-based video processing. In particular some works leverage pixel or semantic differences across frames however this yields limited latency benefits with significantly increased memory overhead. This paper in contrast presents a strategy for masking regions in video frames that leverages the semantic information in images and the temporal correlation between frames to significantly reduce FLOPs and latency with little to no penalty in performance over baseline models. In particular we demonstrate that by leveraging extracted features from previous frames ViT backbones directly benefit from region masking skipping up to 80% of input regions improving FLOPs and latency by 3.14x and 1.5x. We improve memory and latency over the state-of-the-art (SOTA) by 2.3x and 1.14x while maintaining similar detection performance. Additionally our approach demonstrates promising results on convolutional neural networks (CNNs) and provides latency improvements over the SOTA up to 1.3x using specialized computational kernels. Code is available at: https://github.com/sreetamasarkar/MaskVD

Magnetic Resonance Imaging (MRI) is one of the most important medical imaging modalities as it provides superior resolution of soft tissues albeit with a notable limitation in scanning speed. The advent of deep learning has catalyzed the development of cutting-edge methods for the expedited reconstruction of MRI scans utilizing convolutional neural networks and more recently vision transformers. Recently proposed structured state space models (e.g. Mamba) have gained some traction due to their efficiency and low computational requirements compared to transformer models. We propose an innovative MRI reconstruction framework that employs structured state space models at its core aimed at amplifying both long-range contextual sensitivity and reconstruction efficacy. Comprehensive experiments on public brain MRI datasets show that our model sets new benchmarks beating state-of-the-art reconstruction baselines. Code is available at https://github.com/yilmazkorkmaz1/MambaRecon.

The goal of style transfer is given a content image and a style source generating a new image preserving the content but with the artistic representation of the style source. Most of the state-of-the-art architectures use transformers or diffusion-based models to perform this task despite the heavy computational burden that they require. In particular transformers use self- and cross-attention layers which have large memory footprint while diffusion models require high inference time. To overcome the above this paper explores a novel design of Mamba an emergent State-Space Model (SSM) called Mamba-ST to perform style transfer. To do so we adapt Mamba linear equation to simulate the behavior of cross-attention layers which are able to combine two separate embeddings into a single output but drastically reducing memory usage and time complexity. We modified the Mamba's inner equations so to accept inputs from and combine two separate data streams. To the best of our knowledge this is the first attempt to adapt the equations of SSMs to a vision task like style transfer without requiring any other module like cross-attention or custom normalization layers. An extensive set of experiments demonstrates the superiority and efficiency of our method in performing style transfer compared to transformers and diffusion models. Results show improved quality in terms of both ArtFID and FID metrics. Code is available at https://github.com/FilippoBotti/MambaST.

We present Make-A-Texture a new framework that efficiently synthesizes high-resolution texture maps from textual prompts for given 3D geometries. Our approach progressively generates textures that are consistent across multiple viewpoints with a depth-aware inpainting diffusion model in an optimized sequence of viewpoints determined by an automatic view selection algorithm. A significant feature of our method is its remarkable efficiency achieving a full texture generation within an end-to-end runtime of just 3.07 seconds on a single NVIDIA H100 GPU significantly outperforming existing methods. Such an acceleration is achieved by optimizations in the diffusion model and a specialized backprojection method. Moreover our method reduces the artifacts in the backprojection phase by selectively masking out non-frontal faces and internal faces of open-surfaced objects. Experimental results demonstrate that Make-A-Texture matches or exceeds the quality of other state-of-the-art methods. Our work significantly improves the applicability and practicality of texture generation models for real-world 3D content creation including interactive creation and text-guided texture editing.

Leveraging large-scale Text-to-Image (TTI) models have become a common technique to generate training or reference data in the field of image synthesis video editing 3D reconstruction. However semantic structural visual hallucinations which contain perceptually critical defects remain a concern especially in non-photorealistic rendering domains such as cartoon pixelization-style character. We propose a novel semantic structural hallucination detection system in cartoon-style images generated by TTI models collecting a new cartoon-hallucination dataset. Our approach leverages pose-aware in-context visual learning (PA-ICVL) with public Vision-Language Models (VLMs) utilizing both RGB images and pose information. By incorporating pose guidance from a fine-tuned pose estimator we enable VLMs to make more accurate decisions. Experimental results demonstrate significant improvements in identifying visual hallucinations compared to baseline methods relying solely on RGB images. Within selected two VLMs GPT-4v Gemini pro vision our proposed PA-ICVL improves the hallucination detection with 50% to 78% 57% to 80% respectively. This research advances a capability of TTI models toward real-world applications by mitigating visual hallucinations via in-context visual learning expanding their potential in non-photorealistic domains. Besides when VLM confront ambiguous tasks this results showcase thought-provoking insights how users boost the domain-adaptive capability of VLM by harnessing additional conditions. The dataset and demo VLMs are provided in the corresponding Git repository: https://github.com/gh-BumsooKim/Cartoon-Hallucinations-Detection.

Text-based video editing has recently attracted considerable interest in changing the style or replacing the objects with a similar structure. Beyond this we demonstrate that properties such as shape size location motion etc. can also be edited in videos. Our key insight is that the keyframe's transformations of the specific internal feature (e.g. edge maps of objects or human pose) can easily propagate to other frames to provide generation guidance. We thus propose MagicStick a controllable video editing method that edits the video properties by utilizing the transformation on the extracted internal control signals. In detail to keep the appearance we inflate both the pre-trained image diffusion model and ControlNet to the temporal dimension and train low-rank adaptions (LoRA) layers to fit the specific scenes. Then in editing we perform an inversion and editing framework. Differently finetuned ControlNet is introduced in both inversion and generation for attention guidance with the proposed attention remix between the spatial attention maps of inversion and editing. Yet succinct our method is the first method to show the ability of video property editing from the pre-trained text-to-image model. We present experiments on numerous examples within our unified framework. We also compare with shape-aware text-based editing and handcrafted motion video generation demonstrating our superior temporal consistency and editing capability than previous works.

In 3D reconstruction mirrors introduce significant challenges by creating distorted and fragmented spaces resulting in inaccurate and unreliable 3D models. As 3D reconstruction typically relies on multi-view images to capture different perspectives of a scene detecting and labeling mirrors in multi-view images before reconstruction can effectively address this issue. However existing methods focus solely on single-image detection overlooking the rich information provided by multi-view setups. To overcome this limitation we propose MVMD a novel Multi-View Mirror Detection method along with the first database specifically designed for mirror detection in multi-view scenes. The design of MVMD is grounded in the inherent associations between objects seen from different views and those reflected inside and outside of mirrors. These relationships are learned through cross- and self-attention mechanisms. MVMD consists of three key blocks: the Inter-Views Block tracks the shifts of objects within mirrors caused by changes in viewpoint; the Intra-View Block detects object reflections inside mirrors; and the Refinement Block sharpens mirror boundaries and enhances detected details. Experimental results show that our method improves accuracy by up to 2.6% and IoU by up to 11.1% compared to single-image mirror detection techniques. This substantial improvement makes MVMD particularly effective for computer vision tasks especially in enhancing the accuracy of 3D reconstruction in mirror-dense environments. Code and data are available at: https://github.com/mvmdwacv25.

While videos can be falsified in many different ways most existing forensic networks are specialized to detect only a single manipulation type (e.g. deepfake inpainting). This poses a significant issue as the manipulation used to falsify a video is not known a priori. To address this problem we propose MVFNet - a multipurpose video forensics network capable of detecting multiple types of manipulations including inpainting deepfakes splicing and editing. Our network does this by extracting and jointly analyzing a broad set of forensic feature modalities that capture both spatial and temporal anomalies in falsified videos. To reliably detect and localize fake content of all shapes and sizes our network employs a novel Multi-Scale Hierarchical Transformer module to identify forensic inconsistencies across multiple spatial scales. Experimental results show that our network obtains state-of-the-art performance in general scenarios where multiple different manipulations are possible and rivals specialized detectors in targeted scenarios.

Visual artifacts are often introduced into streamed video content due to prevailing conditions during content production and delivery. Since these can degrade the quality of the user's experience it is important to automatically and accurately detect them in order to enable effective quality measurement and enhancement. Existing detection methods often focus on a single type of artifact and/or determine the presence of an artifact through thresholding objective quality indices. Such approaches have been reported to offer inconsistent prediction performance and are also impractical for real-world applications where multiple artifacts co-exist and interact. In this paper we propose a Multiple Visual Artifact Detector MVAD for video streaming which for the first time is able to detect multiple artifacts using a single framework that is not reliant on video quality assessment models. Our approach employs a new Artifact-aware Dynamic Feature Extractor (ADFE) to obtain artifact-relevant spatial features within each frame for multiple artifact types. The extracted features are further processed by a Recurrent Memory Vision Transformer (RMViT) module which captures both short-term and long-term temporal information within the input video. The proposed network architecture is optimized in an end-to-end manner based on a new large and diverse training database that is generated by simulating the video streaming pipeline and based on Adversarial Data Augmentation. This model has been evaluated on two video artifact databases Maxwell and BVI-Artifact and achieves consistent and improved prediction results for ten target visual artifacts when compared to seven existing single and multiple artifact detectors. The source code and training database will be available at https://chenfeng-bristol.github.io/MVAD/.

Panoramic observation using fisheye cameras is significant in virtual reality (VR) and robot perception. However panoramic images synthesized by traditional methods lack depth information and can only provide three degrees-of-freedom (3DoF) rotation rendering in VR applications. To fully preserve and exploit the parallax information within the original fisheye cameras we introduce MSI-NeRF which combines deep learning omnidirectional depth estimation and novel view synthesis. We construct a multi-sphere image as a cost volume through feature extraction and warping of the input images. We further build an implicit radiance field using spatial points and interpolated 3D feature vectors as input which can simultaneously realize omnidirectional depth estimation and 6DoF view synthesis. Leveraging the knowledge from depth estimation task our method can learn scene appearance by source view supervision only. It does not require novel target views and can be trained conveniently on existing panorama depth estimation datasets. Our network has the generalization ability to reconstruct unknown scenes efficiently using only four images. Experimental results show that our method outperforms existing methods in both depth estimation and novel view synthesis tasks.