WACV 2025 论文列表

Deformable image registration is crucial for aligning medical images in a nonlinear fashion across different modalities allowing for precise spatial correspondence between varying anatomical structures. This paper presents NestedMorph a novel network utilizing a Nested Attention Fusion approach to improve intra-subject deformable registration between T1-weighted (T1w) MRI and diffusion MRI (dMRI) data. NestedMorph integrates high-resolution spatial details from an encoder with semantic information from a decoder using a multi-scale framework enhancing both local and global feature extraction. Our model notably outperforms existing methods including CNN-based approaches like VoxelMorph MIDIR and CycleMorph as well as Transformer-based models such as TransMorph and ViT-V-Net and traditional techniques like NiftyReg and SyN. Evaluations using the HCP dataset demonstrate that NestedMorph achieves superior performance across key metrics including SSIM HD95 and SDlogJ with the highest SSIM of 0.89 the lowest HD95 of 2.5 and SDlogJ of 0.22. These results highlight NestedMorph's ability to capture both local and global image features effectively leading to superior registration performance. The promising outcomes of this study underscore NestedMorph's potential to significantly advance deformable medical image registration providing a robust framework for future research and clinical applications. The source code and our implementation are available at: https://github.com/AS-Lab/Marthi-et-al-2024-NestedMorph-Deformable-Medical-Image-Registration

In image editing employing diffusion models it is crucial to preserve the reconstruction fidelity to the original image while changing its style. Although existing methods ensure reconstruction fidelity through optimization a drawback of these is the significant amount of time required for optimization. In this paper we propose negative-prompt inversion a method capable of achieving equivalent reconstruction solely through forward propagation without optimization thereby enabling ultrafast editing processes. We experimentally demonstrate that the reconstruction fidelity of our method is comparable to that of existing methods allowing for inversion at a resolution of 512 pixels and with 50 sampling steps within approximately 5 seconds which is more than 30 times faster than null-text inversion. Reduction of the computation time by the proposed method further allows us to use a larger number of sampling steps in diffusion models to improve the reconstruction fidelity with a moderate increase in computation time.

We address domain-agnostic slice-to-volume (S2V) registration the alignment of 2D sliced/tomographic images into 3D volumes without prior knowledge of structure shape or orientation. While S2V registration is well-studied in medical imaging which often relies on auxiliary information (e.g. landmarks segmentation masks pre-defined orientations canonical/atlas volumes) applications such as micro-structure characterization in materials science lack such domain-specific aids. This leaves the task inherently ill-posed due to noise unstructured regions repetitive patterns rotational and translational symmetries. To address this challenge we present "Needles & Haystacks" a novel multi-domain algorithm development dataset with 158436 unique registration problems and ground-truth solutions based on diverse and openly licensed real-world volumetric data. Additionally we provide an online platform with 8461 test problems for reproducible evaluation of competing methods. We also propose strong baseline solutions with public implementations and highlight opportunities for further algorithmic advancements.

While neural radiance fields (NeRF) led to a breakthrough in photorealistic novel view synthesis handling mirroring surfaces still denotes a particular challenge as they introduce severe inconsistencies in the scene representation. Previous attempts either focus on reconstructing single reflective objects or rely on strong supervision guidance in terms of additional user-provided annotations of visible image regions of the mirrors thereby limiting the practical usability. In contrast in this paper we present NeRF-MD a method which shows that NeRFs can be considered as mirror detectors and which is capable of reconstructing neural radiance fields of scenes containing mirroring surfaces without the need for prior annotations. To this end we first compute an initial estimate of the scene geometry by training a standard NeRF using a depth reprojection loss. Our key insight lies in the fact that parts of the scene corresponding to a mirroring surface will still exhibit a significant photometric inconsistency whereas the remaining parts are already reconstructed in a plausible manner. This allows us to detect mirror surfaces by fitting geometric primitives to such inconsistent regions in this initial stage of the training. Using this information we then jointly optimize the radiance field and mirror geometry in a second training stage to refine their quality. We demonstrate the capability of our method to allow the faithful detection of mirrors in the scene as well as the reconstruction of a single consistent scene representation and demonstrate its potential in comparison to baseline and mirror-aware approaches.

Federated learning (FL) enables multiple clients to train models collectively while preserving data privacy. However FL faces challenges in terms of communication cost and data heterogeneity. One-shot federated learning has emerged as a solution by reducing communication rounds improving efficiency and providing better security against eavesdropping attacks. Nevertheless data heterogeneity remains a significant challenge impacting performance. This work explores the effectiveness of diffusion models in one-shot FL demonstrating their applicability in addressing data heterogeneity and improving FL performance. Additionally we investigate the utility of our diffusion model approach FedDiff compared to other one-shot FL methods under differential privacy (DP). Furthermore to improve generated sample quality under DP settings we propose a pragmatic Fourier Magnitude Filtering (FMF) method enhancing the effectiveness of the generated data for global model training. Code available at https://github.com/mmendiet/FedDiff.

Audio Description (AD) is a narration designed to enhance accessibility for visually impaired individuals by conveying the key visual elements of a video. Thus automating AD generation for long-form videos such as movies and dramas provides high social value but is a challenging task. First AD must reflect the narrative context of the entire movie including the storyline names of characters and places and the cultural setting. Second to avoid disrupting the immersive experience of the movie AD must not overlap with the characters' dialogues requiring the delivery of numerous visual elements in concise sentences. This paper presents NarrAD a training-free AD generation framework that satisfies both of the requirements by leveraging rich narrative context in movie scripts and curating information across narration slots. Experiments on the MAD dataset demonstrate that our approach outperforms prior works in both captioning and LLM-based metrics. In the user study with 600 subjects NarrAD achieves the highest user experience and movie comprehension. NarrAD's AD samples are available at https://bit.ly/4aSwOTr.

In this work we present a novel multi-view photometric stereo (MVPS) method. Like many works in 3D reconstruction we are leveraging neural shape representations and learnt renderers. However our work differs from the state-of-the-art multi-view PS methods such as PS-NeRF or Supernormal in that we explicitly leverage per-pixel intensity renderings rather than relying mainly on estimated normals. We model point light attenuation and explicitly raytrace cast shadows in order to best approximate the incoming radiance for each point. The estimated incoming radiance is used as input to a fully neural material renderer that uses minimal prior assumptions and it is jointly optimised with the surface. Estimated normals and segmentation maps are also incorporated in order to maximise the surface accuracy. Our method is among the first (along with Supernormal) to outperform the classical MVPS approach proposed by the DiLiGenT-MV benchmark and achieves average 0.2mm Chamfer distance for objects imaged at approx 1.5m distance away with approximate 400x400 resolution. Moreover our method shows high robustness to the sparse MVPS setup (6 views 6 lights) greatly outperforming the SOTA competitor (0.38mm vs 0.61mm) illustrating the importance of neural rendering in multi-view photometric stereo.

Continual learning (CL) in medical imaging presents a unique challenge requiring models to adapt to new domains while retaining previously acquired knowledge. We introduce NCAdapt a Neural Cellular Automata (NCA) based method designed to address this challenge. NCAdapt features a domain-specific multi-head structure integrating adaptable convolutional layers into the NCA backbone for each new domain encountered. After initial training the NCA backbone is frozen and only the newly added adaptable convolutional layers consisting of 384 parameters are trained along with domain-specific NCA convolutions. We evaluate NCAdapt on hippocampus segmentation tasks benchmarking its performance against Lifelong nnU-Net and U-Net models with state-of-the-art (SOTA) CL methods. Our lightweight approach achieves SOTA performance underscoring its effectiveness in addressing CL challenges in medical imaging. Upon acceptance we will make our code base publicly accessible to support reproducibility and foster further advancements in medical CL.

Scene text image super-resolution (STISR) enhances the resolution and quality of low-resolution images. Unlike previous studies that treated scene text images as natural images recent methods using a text prior (TP) extracted from a pre-trained text recognizer have shown strong performance. However two major issues emerge: (1) Explicit categorical priors like TP can negatively impact STISR if incorrect. We reveal that these explicit priors are unstable and propose replacing them with Non-CAtegorical Prior (NCAP) using penultimate layer representations. (2) Pre-trained recognizers used to generate TP struggle with low-resolution images. To address this most studies jointly train the recognizer with the STISR network to bridge the domain gap between low- and high-resolution images but this can cause an overconfidence phenomenon in the prior modality. We highlight this issue and propose a method to mitigate it by mixing hard and soft labels. Experiments on the TextZoom dataset demonstrate an improvement by 3.5% while our method significantly enhances generalization performance by 14.8% across four text recognition datasets. Our method generalizes to all TP-guided STISR networks.

The generation of transferable adversarial perturbations typically involves training a generator to maximize embedding separation between clean and adversarial images at a single mid-layer of a source model. In this work we build on this approach and introduce Neuron Attack for Transferability (NAT) a method designed to target specific neuron within the embedding. Our approach is motivated by the observation that previous layer-level optimizations often disproportionately focus on a few neurons representing similar concepts leaving other neurons within the attacked layer minimally affected. NAT shifts the focus from embedding-level separation to a more fundamental neuron-specific approach. We find that targeting individual neurons effectively disrupts the core units of the neural network providing a common basis for transferability across different models. Through extensive experiments on 41 diverse ImageNet models and 9 fine-grained models NAT achieves fooling rates that surpass existing baselines by over 14% in cross-model and 4% in cross-domain settings. Furthermore by leveraging the complementary attacking capabilities of the trained generators we achieve impressive fooling rates within just 10 queries. Our code is available at: https://krishnakanthnakka.github.io/NAT/

In recent years generative 3D face models (e.g. EG3D) have been developed to tackle the problem of synthesizing photo-realistic faces. However these models are often unable to capture facial features unique to each individual highlighting the importance of personalization. Some prior works have shown promise in personalizing generative face models but these studies primarily focus on 2D settings. Also these methods require both fine-tuning and storing a large number of parameters for each user posing a hindrance to achieving scalable personalization. Another challenge of personalization is the limited number of training images available for each individual which often leads to overfitting when using full fine-tuning methods. Our proposed approach My3DGen generates a personalized 3D prior of an individual using as few as 50 training images. My3DGen allows for novel view synthesis semantic editing of a given face (e.g. adding a smile) and synthesizing novel appearances all while preserving the original person's identity. We decouple the 3D facial features into global features and personalized features by freezing the pre-trained EG3D and training additional personalized weights through low-rank decomposition. As a result My3DGen introduces only 240K personalized parameters per individual leading to a 127x reduction in trainable parameters compared to the 30.6M required for fine-tuning the entire parameter space. Despite this significant reduction in storage our model preserves identity features without compromising the quality of downstream applications both quantitatively and qualitatively.

Images obtained from different modalities can effectively enhance the accuracy and reliability of the detection model by complementing specialized information from visible (RGB) and infrared (IR) images. However integrating information from multiple modalities faces the following challenges: 1) distinct characteristics of RGB and IR images lead to the problem of modality imbalance 2) fusing multimodal information can greatly affect the detection accuracy as some of the unique information provided by each modality is lost during the integration process and 3) RGB and IR images are fused while preserving the noise of each modality. To address these issues we propose a novel multispectral object detection network which contains two main components; 1) Cross-modal Information Complementary (CIC) module and 2) Cosine Similarity Channel Resampling (CSCR) module. The proposed method addresses the modality imbalance problem and efficiently fuses RGB and IR images in the feature level. Extensive experimental results on LLVIP FLIR M3FD VEDAI and KAIST benchmark datasets verify the effectiveness and generalization performance of the proposed multispectral object detection network compared with other state-of-the-art methods.

Motivated by depression's significant impact on global health this work proposes MultiDepNet a novel multimodal interpretable depression detection system integrating visual physiological audio and textual data. Through dedicated feature extraction methods (MTCNN for video TS-CAN for physiological ResNet-18 for audio and RoBERTa for text modalities) and a strategic fusion of modality-specific networks including CNN-RNN Transformer MLP and ResNet-18 it achieves significant advancements in depression detection. Its performance evaluated across four benchmark datasets (AVEC 2013 AVEC 2014 DAIC and E-DAIC) demonstrates average MAE of 5.64 RMSE of 7.15 accuracy of 74.19 precision of 0.7373 recall of 0.7378 and F1 of 0.7376. It also implements a MultiViz-based interpretability mechanism that computes each modality's contribution to the model's performance. The results reveal the visual modality to be the most significant contributing 37.88% towards depression detection.

Multimodal fusion learning has shown significant promise in classifying various diseases such as skin cancer and brain tumors. However existing methods face three key limitations. First they often lack generalizability to other diagnosis tasks due to their focus on a particular disease. Second they do not fully leverage multiple health records from diverse modalities to learn robust complementary information. And finally they typically rely on a single attention mechanism missing the benefits of multiple attention strategies within and across various modalities. To address these issues this paper proposes a dual robust information fusion attention mechanism (DRIFA) that leverages two attention modules - i.e. multi-branch fusion attention module and the multimodal information fusion attention module. DRIFA can be integrated with any deep neural network forming a multi modal fusion learning framework denoted as DRIFA-Net. We show that the multi-branch fusion attention of DRIFA learns enhanced representations for each modality such as dermoscopy pap smear MRI and CT-scan whereas multimodal information fusion attention module learns more refined multimodal shared representations - improving the network's generalization across multiple tasks and enhancing overall performance. Additionally to estimate the uncertainty of DRIFA-Net predictions we have employed an ensemble Monte Carlo dropout strategy. Extensive experiments on five publicly available datasets with diverse modalities demonstrate that our approach consistently outperforms state-of-the-art methods. The code is available at https://github.com/misti1203/DRIFA-Net.

Recently text-to-image generation with diffusion models has made significant advancements in both higher fidelity and generalization capabilities compared to previous baselines. However generating holistic multi-view consistent images from prompts still remains an important and challenging task. To address this challenge we propose a diffusion process that attends to time-dependent spatial frequencies of features with a novel attention mechanism as well as novel noise initialization technique and cross-attention loss. This Fourier-based attention block focuses on features from non-overlapping regions of the generated scene in order to better align the global appearance. Our noise initialization technique incorporates shared noise and low spatial frequency information derived from pixel coordinates and depth maps to induce noise correlations across views. The cross-attention loss further aligns features sharing the same prompt across the scene. Our technique improves SOTA on several quantitative metrics with qualitatively better results when compared to other state-of-the-art approaches for multi-view consistency.

Multi-view multi-label classification (MvMLC) has recently garnered significant research attention due to its wide range of real-world applications. However incompleteness in views and labels is a common challenge often resulting from data collection oversights and uncertainties in manual annotation. Furthermore the task of learning robust multi-view representations that are both view-consistent and view-specific from diverse views still a challenge problem in MvMLC. To address these issues we propose a novel framework for incomplete multi-view multi-label classification (iMvMLC). Our method factorizes multi-view representations into two independent sets of factors: view-consistent and view-specific and we correspondingly design a graph disentangling loss to fully reduce redundancy between these representations. Additionally our framework innovatively decomposes consistent representation learning into three key sub-objectives: (i) how to extract view-shared information across different views (ii) how to eliminate intra-view redundancy in consistent representations and (iii) how to preserve task-relevant information. To this end we design a robust task-relevant consistency learning module that collaboratively learns high-quality consistent representations leveraging a masked cross-view prediction (MCP) strategy and information theory. Notably all modules in our framework are developed to function effectively under conditions of incomplete views and labels making our method adaptable to various multi-view and multi-label datasets. Extensive experiments on five datasets demonstrate that our method outperforms other leading approaches.

In this study we propose a multi-task learning model of classification and generation for set-structured data. The proposed model learns data generation and classification in a single neural network by integrating a classification layer into a variational autoencoder while maintaining permutation invariance and equivariance nature which are characteristics of set-structured data. The proposed model allows for semi-supervised learning in set-structured data classification and can also be applied to confidence calibration using the input data distribution estimated by the generative model. In the experiments we evaluated the performance of the proposed model in a semi-supervised classification task on set-structured datasets and compared it with a baseline model consisting only of a classifier. The results demonstrated that simultaneous learning of the classification and generation effectively improves the classification accuracy and confidence reliability for set-structured data even with a limited number of labeled data.

Despite remarkable progress in knowledge transfer across visual and textual domains extending these achievements to indoor localization particularly for learning transferable representations among Received Signal Strength (RSS) fingerprint datasets remains a challenge. This is due to inherent discrepancies among these RSS datasets largely including variations in building structure the input number and disposition of WiFi anchors. Accordingly specialized networks which were deprived of the ability to discern transferable representations readily incorporate environment-sensitive clues into the learning process hence limiting their potential when applied to specific RSS datasets. Initially we design an Expert Training phase which features multiple surrogate generative teachers all serving as a global adapter that homogenizes the input disparities among independent source RSS datasets while preserving their unique characteristics. In a subsequent Expert Distilling phase we continue introducing a triplet of underlying constraints that requires minimizing the differences in essential knowledge between the specialized network and surrogate teachers through refining its representation learning on the target dataset. This process implicitly fosters a representational alignment in such a way that is less sensitive to specific environmental dynamics. Extensive experiments conducted on three benchmark WiFi RSS fingerprint datasets underscore the effectiveness of the framework that significantly exerts the full potential of specialized networks in localization

From camera to screen researchers have developed a well-established system for capturing and reproducing the color experience of human eyes. In this study we aim to upgrade this process by transiting from conventional RGB to multi-spectral image (MSI) color reproduction. While MSI offers evident advantages in color matching we find out it is not trivial to make good use of more spectral information for color constancy. Therefore we present a regularized color reproduction system that incorporates a spectral prior-guided optimization strategy to establish a sensor-optimized RGB projection for color matching along with a learning-based chromatic adaptation model for color constancy. Specifically we define the RGB projection through an end-to-end optimization under the guidance of sensor spectral sensitivities. Subsequently we devise a chromatic adaptation neural network that estimates the scene illuminance and an illuminance-adaptive matrix for auto white balancing and dynamic color correction respectively. Comprehensive experiments show the superiority of our system compared to alternative solutions.