WACV 2025 Papers

Magnetic Resonance Imaging (MRI) is a powerful imaging technique widely used for visualizing structures within the human body and in other fields such as plant sciences. However there is a demand to develop fast 3D-MRI reconstruction algorithms to show the fine structure of objects from under-sampled acquisition data i.e. k-space data. This emphasizes the need for efficient solutions that can handle limited input while maintaining high-quality imaging. In contrast to previous methods only using 2D we propose a 3D MRI reconstruction method that leverages a regularized 3D diffusion model combined with optimization method. By incorporating diffusion-based priors our method improves image quality reduces noise and enhances the overall fidelity of 3D MRI reconstructions. We conduct comprehensive experiments analysis on clinical and plant science MRI datasets. To evaluate the algorithm effectiveness for under-sampled k-space data we also demonstrate its reconstruction performance with several undersampling patterns as well as with in- and out-of-distribution pre-trained data. In experiments we show that our method improves upon tested competitors

In visual Reinforcement Learning (RL) learning from pixel-based observations poses significant challenges on sample efficiency primarily due to the complexity of extracting informative state representations from high-dimensional data. Previous methods such as contrastive-based approaches have made strides in improving sample efficiency but fall short in modeling the nuanced evolution of states. To address this we introduce MOOSS a novel framework that leverages a temporal contrastive objective with the help of graph-based spatial-temporal masking to explicitly model state evolution in visual RL. Specifically we propose a self-supervised dual-component strategy that integrates (1) a graph construction of pixel-based observations for spatial-temporal masking coupled with (2) a multi-level contrastive learning mechanism that enriches state representations by emphasizing temporal continuity and change of states. MOOSS advances the understanding of state dynamics by disrupting and learning from spatial-temporal correlations which facilitates policy learning. Our comprehensive evaluation on multiple continuous and discrete control benchmarks shows that MOOSS outperforms previous state-of-the-art visual RL methods in terms of sample efficiency demonstrating the effectiveness of our method.

Spiking Neural Networks (SNNs) have emerged as a compelling alternative to traditional Artificial Neural Networks (ANNs) due to their energy efficiency and biological plausibility. However current SNN models often rely on ANN architectures that may not fully exploit the unique properties of SNNs. Neural Architecture Search (NAS) approaches have been shown to automate the identification of suitable architectures for various applications. Nevertheless very few works have been presented on NAS for SNNs and particularly for identifying architectures that achieve high accuracy while capitalizing on the energy efficiency property of SNNs. In this paper we present a Multi-Objective Neural Architecture Search for Efficient SNNs (MONAS-ESNN) approach that utilizes a training-free NAS to discover SNN architectures that optimize both accuracy and energy efficiency. The proposed MONAS-ESNN uses the NSGA-II evolutionary algorithm to optimize both objective functions while leveraging the unique temporal dynamics of SNNs. We also introduce a new Adjusted Sparsity-Aware Hamming Distance (ASAHD) that enhances the evaluation of potential architectures by representing diverse spike activation patterns for different types of spiking neurons. Experimental results on CIFAR-10 CIFAR-100 and Tiny-ImageNet-200 datasets demonstrate that MONAS-ESNN identifies SNN architectures that have higher accuracy and are more efficient as measured by the number of generated spikes than existing methods. Therefore the proposed MONAS-ESNN can automate the search and discovery of SNN architectures with higher accuracy fewer generated spikes and faster convergence paving the way for more energy-efficient neural networks.

Recently the astonishing performance of large language models (LLMs) in natural language comprehension and generation tasks triggered lots of exploration of using them as central controllers to build agent systems. Multiple studies focus on bridging the LLMs to external tools to extend the application scenarios. However the current LLMs' ability to perceive tool use is limited to a single text query which may result in ambiguity in understanding the users' real intentions. LLMs are expected to eliminate that by perceiving the information in the visual- or auditory-grounded instructions. Therefore in this paper we propose MLLM-Tool a system incorporating open-source LLMs and multi-modal encoders so that the learned LLMs can be conscious of multi-modal input instruction and then select the function-matched tool correctly. To facilitate the evaluation of the model's capability we collect a dataset featuring multi-modal input tools from HuggingFace. Another essential feature of our dataset is that it also contains multiple potential choices for the same instruction due to the existence of identical functions and synonymous functions which provides more potential solutions for the same query. The experiments reveal that our MLLM-Tool is capable of recommending appropriate tools for multi-modal instructions. Codes and data are available at github.com/MLLMTool/MLLM-Tool.

Previous studies on federated learning (FL) often encounter performance degradation due to data heterogeneity among different clients. In light of the recent advances in multimodal large language models (MLLMs) such as GPT-4v and LLaVA which demonstrate their exceptional proficiency in multimodal tasks such as image captioning and multimodal question answering. We introduce a novel federated learning framework named Multimodal Large Language Model Assisted Federated Learning (MLLM-LLaVA-FL) which employs powerful MLLMs at the server end to address the heterogeneous and long-tailed challenges. Owing to the advanced cross-modality representation capabilities and the extensive open-vocabulary prior knowledge of MLLMs our framework is adept at harnessing the extensive yet previously underexploited open-source data accessible from websites and powerful server-side computational resources. Hence the MLLM-LLaVA-FL not only enhances the performance but also avoids increasing the risk of privacy leakage and the computational burden on local devices distinguishing it from prior methodologies. Our framework has three key stages. Initially we conduct global visual-text pretraining of the model. This pretraining is facilitated by utilizing the extensive open-source data available online with the assistance of MLLMs. Subsequently the pretrained model is distributed among various clients for local training. Finally once the locally trained models are transmitted back to the server a global alignment is carried out under the supervision of MLLMs to further enhance the performance. Experimental evaluations on established benchmarks show that our framework delivers promising performance in the typical scenarios with data heterogeneity and long-tail distribution across different clients in FL.

Estimating the Most Important Person (MIP) in any social event setup is a challenging problem mainly due to contextual complexity and scarcity of labeled data. Moreover the causality aspects of MIP estimation are quite subjective and diverse. To this end we aim to address the problem by annotating a large-scale 'in-the-wild' dataset for identifying human perceptions about the 'Most Important Person (MIP)' in an image. The paper provides a thorough description of our proposed Multimodal Large Language Model (MLLM) based data annotation strategy and a thorough data quality analysis. Further we perform a comprehensive benchmarking of the proposed dataset utilizing state-of-the-art MIP localization methods indicating a significant drop in performance compared to existing datasets. The performance drop shows that the existing MIP localization algorithms must be more robust with respect to 'in-the-wild' situations. We believe the proposed dataset will play a vital role in building the next-generation social situation understanding methods. The dataset and associated code will be made available for research purposes.

Accurate tumor segmentation in whole slide image (WSI) is essential for histopathological diagnosis and research but the traditional manual analysis is labor-intensive and prone to variability. Furthermore many artificial models focus on specific magnification images limiting the detailed information available for segmentation. To address these challenges we propose MFTrans a novel multi-resolution fusion transformer with a CNN-based architecture designed for efficient tumor segmentation in WSI. Inspired by the diagnostic procedures of expert pathologists MFTrans integrates both high- and low-magnification images capturing detailed local features and broader contextual relationships through a dual-branch architecture. The model employs a global token transformer and cross-attention mechanism to fuse hierarchical features from dual branches to improve segmentation performance. We evaluate MFTrans on three real-world WSI datasets: Camelyon16 PAIP2019 and Catholic Uijeongbu St. Mary's hospital dataset demonstrating its superior segmentation performance over state-of-the-art methods in balanced and imbalanced setups. These results highlight MFTrans's effectiveness in medical image analysis and its generalizability across different datasets making it a robust tool for automated cancer diagnostics. Our code is available at https://github.com/aimed-gist/MFTrans.

In this work we present MFTIQ a novel dense long-term tracking model that advances the Multi-Flow Tracker (MFT) framework to address challenges in point-level visual tracking in video sequences. MFTIQ builds upon the flow-chaining concepts of MFT integrating an Independent Quality (IQ) module that separates correspondence quality estimation from optical flow computations. This decoupling significantly enhances the accuracy and flexibility of the tracking process allowing MFTIQ to maintain reliable trajectory predictions even in scenarios of prolonged occlusions and complex dynamics. Designed to be "plug-and-play" MFTIQ can be employed with any off-the-shelf optical flow method without the need for fine-tuning or architectural modifications. Experimental validations on the TAP-Vid Davis dataset show that MFTIQ with RoMa optical flow not only surpasses MFT but also performs comparably to state-of-the-art trackers while having substantially faster processing speed. Code and models available at https://github.com/serycjon/MFTIQ.

Recently neural radiance fields (NeRFs) have shown remarkable performance in generating photorealistic novel views in 3D modeling. The traditional NeRF typically requires extensive training and long rendering times inspiring many recent works to utilize efficient data structures such as feature grids to ease the computational burden of multilayer perceptron networks. However Those approaches require storing features in dense grids that demands a substantial amount of memory space resulting in a notable memory bottleneck. Consequently it leads to a significant increase in training time. To address this issue in this work we propose MFNeRF a memory-efficient NeRF framework that employs a mixed-feature hash table to improve memory efficiency and reduce training time while maintaining reconstruction quality. Specifically we first design a mixed-feature hash encoding method to adaptively mix parts of multi-level feature grids and map them to a single hash table. Following that in order to obtain the correct index of a grid point we further develop an index transformation method that transforms indices of an arbitrary-level grid to those of a canonical grid. Extensive benchmarking against the state-of-the-art methods including InstantNGP TensoRF and DVGO indicates that our MFNeRF achieves faster training and rendering times on the same GPU hardware with a significantly smaller memory while maintaining similar or even higher reconstruction quality. Compared to the InstantNGP-Big model our method could achieve 89% improvement in the figure of merit defined in terms of PSNR*FPS/MB.

Leveraging human perception into training of convolutional neural networks (CNN) has boosted generalization capabilities of such models in open-set recognition tasks. One of the active research questions is where (in the model architecture or training pipeline) and how to efficiently incorporate always-limited human perceptual data into training strategies of models. In this paper we introduce MENTOR (huMan pErceptioN-guided preTraining fOr increased geneRalization) which addresses this question through two unique rounds of training CNNs tasked with open-set anomaly detection. First we train an autoencoder to learn human saliency maps given an input image without any class labels. The autoencoder is thus tasked with discovering domain-specific salient features which mimic human perception. Second we remove the decoder part add a classification layer on top of the encoder and train this new model conventionally now using class labels. We show that MENTOR successfully raises the generalization performance across three different CNN backbones in a variety of anomaly detection tasks (demonstrated for detection of unknown iris presentation attacks synthetically-generated faces and anomalies in chest X-ray images) compared to traditional pretraining methods (e.g. sourcing the weights from ImageNet) and as well as state-of-the-art methods that incorporate human perception guidance into training. In addition we demonstrate that MENTOR can be flexibly applied to existing human perception-guided methods and subsequently increasing their generalization with no architectural modifications.

Photometric Stereo (PS) is a technique for estimating surface normals from images illuminated by multiple light sources. However when the target object has a complex shape or the light sources are not appropriately arranged certain regions may experience severe shadows leading to insufficient information for accurate estimation. In this paper we propose a Monocular-Depth-guided Coarse Normal attention for Photometric Stereo (MDCN-PS). The MDCN-PS can effectively combine monocular depth from a single image with PS with multiple light sources by a Photometric Stereo network Adaptor (PS Adaptor) with Coarse Normal Attention. The key is to use the coarse normals obtained from Monocular Depth Estimation as supplementary information which can improve accuracy in regions where the light source is limited due to severe shadows or inhomogeneous light source distribution. Comprehensive experiments on real-world and synthetic datasets show that the proposed method achieved an accuracy improvement of 1.2 points in real-world datasets when limited to two input images and of 3.1 points in synthetic datasets in mean angular error compared to existing methods. Qualitative results also demonstrated that our method improves accuracy in areas with insufficient lighting patterns due to shadows.

Medical imaging analysis faces challenges such as data scarcity high annotation costs and privacy concerns. This paper introduces the Medical AI for Synthetic Imaging (MAISI) an innovative approach using the diffusion model to generate synthetic 3D computed tomography (CT) images to address those challenges. MAISI leverages the foundation volume compression network and the latent diffusion model to produce high-resolution CT images (up to a landmark volume dimension of 512 x 512 x 768 ) with flexible volume dimensions and voxel spacing. By incorporating ControlNet MAISI can process organ segmentation including 127 anatomical structures as additional conditions and enables the generation of accurately annotated synthetic images that can be used for various downstream tasks. Our experiment results show that MAISI's capabilities in generating realistic anatomically accurate images for diverse regions and conditions reveal its promising potential to mitigate challenges using synthetic data.

Masked Autoencoders (MAEs) are an important divide in self-supervised learning (SSL) due to their independence from augmentation techniques for generating positive (and/or negative) pairs as in contrastive frameworks. Their masking and reconstruction strategy also nicely aligns with SSL approaches in natural language processing. Most MAEs are built upon Transformer-based architectures where visual features are not regularized as opposed to their convolutional neural network (CNN) based counterparts which can potentially hinder their performance. To address this we introduce MAGMA a novel batch-wide layer-wise regularization loss applied to representations of different Transformer layers. We demonstrate that by plugging in the proposed regularization loss one can significantly improve the performance of MAE-based models. We further demonstrate the impact of the proposed loss on optimizing other generic SSL approaches (such as VICReg and SimCLR) broadening the impact of the proposed approach. Our code base can be found at https://github.com/adondera/magma.

Decoupling lighting from geometry using unconstrained photo collections is notoriously challenging. Solving it would benefit many users as creating complex 3D assets takes days of manual labor. Many previous works have attempted to address this issue often at the expense of output fidelity which questions the practicality of such methods. We introduce LumiGauss - a technique that tackles 3D reconstruction of scenes and environmental lighting through 2D Gaussian Splatting. Our approach yields high-quality scene reconstructions and enables realistic lighting synthesis under novel environment maps. We also propose a method for enhancing the quality of shadows common in outdoor scenes by exploiting spherical harmonics properties. Our approach facilitates seamless integration with game engines and enables the use of fast precomputed radiance transfer. We validate our method on the NeRF-OSR dataset demonstrating superior performance over baseline methods. Moreover LumiGauss can synthesize realistic images for unseen environment maps. Our code: https: //github.com/joaxkal/lumigauss.

Research in efficient vision backbones is evolving into models that are a mixture of convolutions and transformer blocks. A smart combination of both architecture-wise and component-wise is mandatory to excel in the speed-accuracy trade-off. Most publications focus on maximizing accuracy and utilize MACs (multiply accumulate operations) as an efficiency metric. The latter however often do not measure accurately how fast a model actually is due to factors like memory access cost and degree of parallelism. We analyzed common modules and architectural design choices for backbones not in terms of MACs but rather in actual throughput and latency as the combination of the latter two is a better representation of the efficiency of models in real applications. We applied the conclusions taken from that analysis to create a recipe for increasing hardware-efficiency in macro design. Additionally we introduce a simple slimmed-down version of Multi-Head Self-Attention that aligns with our analysis. We combine both macro and micro design to create a new family of hardware-efficient backbone networks called LowFormer. LowFormer achieves a remarkable speedup in terms of throughput and latency while achieving similar or better accuracy than current state-of-the-art efficient backbones. In order to prove the generalizability of our hardware-efficient design we evaluate our method on GPU mobile GPU and ARM CPU. We further show that the downstream tasks object detection and semantic segmentation profit from our hardware-efficient architecture. Code and models are available at https://github.com/altair199797/LowFormer.

Convolutional Neural Networks (CNNs) that have excelled in diverse computer vision tasks are vulnerable to backdoor attacks enabling attacker-controlled predictions via specific triggers. Restricted to spatial domains recent research exploits perceptual traits by embedding triggers in the frequency domain yielding pixel-level indistinguishable perturbations. In black-box settings restricted access to model and training process necessitates advanced trigger designs. Current frequency-based attacks manipulate magnitude spectra introducing discrepancies between clean and poisoned data though vulnerable to common image processing operations like compression and filtering. In this paper we propose a robust low-frequency backdoor attack (LFBA) in black-box setup that minimally perturbs spectrum components and maintains the perceptual similarity in spatial space simultaneously. Our methodology capitalizes on the insight that optimal triggers can be located in low-frequency regions to maximize attack effectiveness robustness against image transformation operations and stealthiness in dual space. To effectively explore the discrete frequency space we utilize simulated annealing (SA) a form of evolutionary algorithm to optimize the properties of trigger including the frequency bands to be manipulated and the perturbation of each band under restricted attack scenario. Extensive experiments on both CNNs and Vision Transformers (ViT) confirm the effectiveness and robustness of LFBA against image processing operations and state-of-the-art backdoor defenses. Furthermore LFBA exhibits inherent stealthiness in both spatial and frequency spaces making it resistant to human and frequency inspection.

The computer vision community has explored dyadic interactions for atomic actions such as pushing carrying-object etc. However with the advancement in deep learning models there is a need to explore more complex dyadic situations such as loose interactions. These are interactions where two people perform certain atomic activities to complete a global action irrespective of temporal synchronisation and physical engagement like cooking-together for example. Analysing these types of dyadic-interactions has several useful applications in the medical domain for social-skills development and mental health diagnosis. To achieve this we propose a novel dual-path architecture to capture the loose interaction between two individuals. Our model learns global abstract features from each stream via a CNNs backbone and fuses them using a new Global-Layer-Attention module based on a cross-attention strategy. We evaluate our model on real-world autism diagnoses such as our Loose-Interaction dataset and the publicly available Autism dataset for loose interactions. Our network achieves baseline results on the Loose-Interaction and SOTA results on the Autism datasets. Moreover we study different social interactions by experimenting on a publicly available dataset i.e. NTU-RGB+D (interactive classes from both NTU-60 and NTU-120). We have found that different interactions require different network designs. We also compare a slightly different version of our method (details in Section 3.6) by incorporating time information to address tight interactions achieving SOTA results.

Deep neural networks are likely to learn unintended spurious correlations between training data and labels when dealing with biased data potentially limiting the generalization to unseen samples not presenting the same bias. In this context model debiasing approaches can be devised aiming at reducing the model's dependency on such unwanted correlations either leveraging the knowledge of bias information or not. In this work we focus on the latter and more realistic scenario showing the importance of accurately predicting the bias-conflicting and bias-aligned samples to obtain compelling performance in bias mitigation. On this ground we propose to conceive the problem of model bias from an out-of-distribution perspective introducing a new bias identification method based on anomaly detection. We claim that when data is mostly biased bias-conflicting samples can be regarded as outliers with respect to the bias-aligned distribution in the feature space of a biased model thus allowing for precisely detecting them with an anomaly detection method. Coupling the proposed bias identification approach with bias-conflicting data upsampling and augmentation in a two-step strategy we reach state-of-the-art performance on synthetic and real benchmark datasets. Ultimately our proposed approach shows that the data bias issue does not necessarily require complex debiasing methods given that an accurate bias identification procedure is defined. Source code is available at https://github.com/Malga-Vision/MoDAD

Despite the importance of long-term memory in marketing and brand building until now there has been no large-scale study on the memorability of ads. All previous memorability studies have been conducted on short-term recall on specific content types like action videos. On the other hand long-term memorability is crucial for advertising industry and ads are almost always highly multimodal. Therefore we release the first memorability dataset LAMBDA consisting of 1749 participants and 2205 ads covering 276 brands. Running statistical tests over different participant subpopulations and ad types we find many interesting insights into what makes an ad memorable e.g. fast-moving ads are more memorable than those with slower scenes; people who use ad-blockers remember a lower number of ads than those who don't. Next we present a model Henry to predict the memorability of a content. Henry achieves state-of-the-art performance across all prominent literature memorability datasets. It shows strong generalization performance with better results in 0-shot on unseen datasets. Finally with the intent of memorable ad generation we present a scalable method to build a high-quality memorable ad generation model by leveraging automatically annotated data. Our approach SEED (Self rEwarding mEmorability Modeling) starts with a language model trained on LAMBDA as seed data and progressively trains an LLM to generate more memorable ads. We show that the generated advertisements have 44% higher memorability scores than the original ads. We release this large-scale ad dataset UltraLAMBDA consisting of 5 million ads. Our code and the datasets LAMBDA and UltraLAMBDA are open-sourced at https://behavior-in-the-wild.github.io/memorability.