WACV 2025 Papers

Positional encodings are a common component of neural scene reconstruction methods and provide a way to bias the learning of neural fields towards coarser or finer representations. Current neural surface reconstruction methods use a "one-size-fits-all" approach to encoding choosing a fixed set of encoding functions and therefore bias across all scenes. Current state-of-the-art surface reconstruction approaches leverage grid-based multi-resolution hash encoding in order to recover high-detail geometry. We propose a learned approach which allows the network to choose its encoding basis as a function of space by masking the contribution of features stored at separate grid resolutions. The resulting spatially adaptive approach allows the network to fit a wider range of frequencies without introducing noise. We test our approach on standard benchmark surface reconstruction datasets and achieve state-of-the-art performance on two benchmark datasets.

We present a novel deep learning-based approach to the 3D reconstruction of clothed humans using weak supervision via 2D normal maps. Given a single RGB image or multiview images our network is optimized to infer a person-specific signed distance function (SDF) discretized on a tetrahedral mesh surrounding the body in a rest pose. Subsequently estimated human pose and camera parameters are used to generate a normal map from the SDF. A key aspect of our approach is the direct use of the Marching Tetrahedra algorithm in end-to-end optimization and in order to do so we derive analytical gradients to facilitate straightforward differentiation (and thus backpropagation). Additionally predicted normal maps allow us to leverage pretrained image-to-normal networks in order to minimize a surface error instead of a photometric error. We demonstrate the efficacy of our approach on both labeled and in-the-wild data in the context of existing clothed human reconstruction methods.

Cross-View Geo-Localisation within urban regions is challenging in part due to the lack of geo-spatial structuring within current datasets and techniques. We propose utilising graph representations to model sequences of local observations and the connectivity of the target location. Modelling as a graph enables generating previously unseen sequences by sampling with new parameter configurations. To leverage this newly available information we propose a GNN-based architecture producing spatially strong embeddings and improving discriminability over isolated image embeddings. We outline SpaGBOL introducing three novel contributions. 1) The first graph-structured dataset for Cross-View Geo-Localisation containing multiple streetview images per node to improve generalisation. 2) Introducing GNNs to the problem we develop the first system that exploits the correlation between node proximity and feature similarity. 3) Leveraging the unique properties of the graph representation - we demonstrate a novel retrieval filtering approach based on neighbourhood bearings. SpaGBOL achieves state-of-the-art accuracies on the unseen test graph - with relative Top-1 retrieval improvements on previous techniques of 11% and 50% when filtering with Bearing Vector Matching on the SpaGBOL dataset. Code and dataset available: https://www.github.com/tavisshore/SpaGBOL.

Development of optical technology has enabled imaging of two-dimensional (2D) sound fields. This acousto-optic sensing enables understanding of the interaction between sound and objects such as reflection and diffraction. Moreover it is expected to be used an advanced measurement technology for sonars in self-driving vehicles and assistive robots. However the low sound-pressure sensitivity of the acousto-optic sensing results in high intensity of noise on images. Therefore denoising is an essential task to visualize and analyze the sound fields. In addition to denoising segmentation of sound and object silhouette is also required to analyze interactions between them. In this paper we propose sound-field-images-with-object-silhouette denoising and segmentation (SoundSil-DS) that jointly perform denoising and segmentation for sound fields and object silhouettes on a visualized image. We developed a new model based on the current state-of-the-art denoising network. We also created a dataset to train and evaluate the proposed method through acoustic simulation. The proposed method was evaluated using both simulated and measured data. We confirmed that our method can applied to experimentally measured data. These results suggest that the proposed method may improve the post-processing for sound fields such as physical model-based three-dimensional reconstruction since it can remove unwanted noise and separate sound fields and other object silhouettes. Our code is available at https://github.com/nttcslab/soundsil-ds.

Accurately localizing 3D sound sources and estimating their semantic labels - where the sources may not be visible but are assumed to lie on the physical surface of objects in the scene - have many real applications including detecting gas leak and machinery malfunction. The audio-visual weak- correlation in such setting poses new challenges in deriving innovative methods to answer if or how we can use cross- modal information to solve the task. Towards this end we propose to use an acoustic-camera rig consisting of a pinhole RGB-D camera and a coplanar four-channel microphone array (Mic-Array). By using this rig to record audio-visual signals from multiviews we can use the cross-modal cues to estimate the sound sources 3D locations. Specifically our framework SoundLoc3D treats the task as a set prediction problem each element in the set corresponds to a potential sound source. Given the audio-visual weak-correlation the set representation is initially learned from a single view microphone array signal and then refined by actively incorporating physical surface cues revealed from multiview RGB-D images. We demonstrate the efficiency and superiority of SoundLoc3D on large-scale simulated dataset and further show its robustness to RGB-D measurement inaccuracy and ambient noise interference.

Accurate intraday solar irradiance forecasting is crucial for optimizing dispatch planning and electricity trading. For this purpose we introduce a novel and effective approach that includes three distinguishing components from the literature: 1) the uncommon use of single-frame public camera imagery; 2) solar irradiance time series scaled with a proposed normalization step which boosts performance; and 3) a lightweight multimodal model called Solar Multimodal Transformer (SMT) that delivers accurate short-term solar irradiance forecasting by combining images and scaled time series. Benchmarking against Solcast a leading solar forecasting service provider our model improved prediction accuracy by 25.95%. Our approach allows for easy adaptation to various camera specifications offering broad applicability for real-world solar forecasting challenges.

Pedestrian trajectory prediction remains a challenge for autonomous systems particularly due to the intricate dynamics of social interactions. Accurate forecasting requires a comprehensive understanding not only of each pedestrian's previous trajectory but also of their interaction with the surrounding environment an important part of which are other pedestrians moving dynamically in the scene. To learn effective socially-informed representations we propose a model that uses a reconstructor alongside a conditional variational autoencoder-based trajectory forecasting module. This module generates pseudo-trajectories which we use as augmentations throughout the training process. To further guide the model towards social awareness we propose a novel social loss that aids in forecasting of more stable trajectories. We validate our approach through extensive experiments demonstrating strong performances in comparison to state of-the-art methods on the ETH/UCY and SDD benchmarks.

Accurately estimating the 3D pose of the camera wearer in egocentric video sequences is crucial to modeling human behavior in virtual and augmented reality applications. The task presents unique challenges due to the limited visibility of the user's body caused by the front-facing camera mounted on their head. Recent research has explored the utilization of the scene and ego-motion but it has overlooked humans' interactive nature. We propose a novel framework for Social Egocentric Estimation of body MEshes (SEE-ME). Our approach is the first to estimate the wearer's mesh using only a latent probabilistic diffusion model which we condition on the scene and for the first time on the social wearer-interactee interactions. Our in depth study sheds light on when social interaction matters most for ego-mesh estimation: it quantifies the impact of interpersonal distance and gaze direction. Overall SEE-ME surpasses the current best technique reducing the pose estimation error (MPJPE) by 53%. The code is available at https://github.com/L-Scofano/SEEME/tree/main.

Keratoconus an ocular condition marked by progressive corneal thinning and outward bulging presents diagnostic challenges due to the high cost and lack of portability in conventional corneal topographers. These limitations restrict accessibility for many necessitating affordable and mobile alternatives. Innovations like SmartKC offer a low-cost and portable alternative however there still remains some gaps in performance when compared to commercial topographers. In this paper we introduce SmartKC++ a series of innovative methodological improvements to the image processing pipeline of SmartKC aimed at significantly enhancing its diagnostic precision and reliability. Our comprehensive evaluation on a dataset comprising 303 eye images reveals that SmartKC++ boosts the accuracy of automated keratoconus diagnosis by 7.69% relative to SmartKC.

Deep learning (DL) methods have shown remarkable successes in medical image segmentation often using large amounts of annotated data for model training. However acquiring a large number of diverse labeled 3D medical image datasets is highly difficult and expensive. Recently mask propagation DL methods were developed to reduce the annotation burden on 3D medical images. For example Sli2Vol [59] proposed a self-supervised framework (SSF) to learn correspondences by matching neighboring slices via slice reconstruction in the training stage; the learned correspondences were then used to propagate a labeled slice to other slices in the test stage. But these methods are still prone to error accumulation due to the inter-slice propagation of reconstruction errors. Also they do not handle discontinuities well which can occur between consecutive slices in 3D images as they emphasize exploiting object continuity. To address these challenges in this work we propose a new SSF called Sli2Vol+ for segmenting any anatomical structures in 3D medical images using only a single annotated slice per training and testing volume. Specifically in the training stage we first propagate an annotated 2D slice of a training volume to the other slices generating pseudo-labels (PLs). Then we develop a novel Object Estimation Guided Correspondence Flow Network to learn reliable correspondences between consecutive slices and corresponding PLs in a self-supervised manner. In the test stage such correspondences are utilized to propagate a single annotated slice to the other slices of a test volume. We demonstrate the effectiveness of our method on various medical image segmentation tasks with different datasets showing better generalizability across different organs modalities and modals. Code is available at https://github.com/adlsn/Sli2Volplus

Integrating aerial imagery-based scene generation into applications like autonomous driving and gaming enhances realism in 3D environments but challenges remain in creating detailed content for occluded areas and ensuring real-time consistent rendering. In this paper we introduce Skyeyes a novel framework that can generate photorealistic sequences of ground view images using only aerial view inputs thereby creating a ground roaming experience. More specifically we combine a 3D representation with a view consistent generation model which ensures coherence between generated images. This method allows for the creation of geometrically consistent ground view images even with large view gaps. The images maintain improved spatial-temporal coherence and realism enhancing scene comprehension and visualization from aerial perspectives. To the best of our knowledge there are no publicly available datasets that contain pairwise geo-aligned aerial and ground view imagery. Therefore we build a large synthetic and geo-aligned dataset using Unreal Engine. Both qualitative and quantitative analyses on this synthetic dataset display superior results compared to other leading synthesis approaches. See the project page for more results: https://chaoren2357.github.io/website-skyeyes/

The emergence of diffusion models has enabled the generation of diverse high-quality images solely from text prompting subsequent efforts to enhance the controllability of these models. Despite the improvement in controllability pose control remains limited to specific objects (e.g. humans) or poses (e.g. frontal view) due to the fact that pose is generally controlled via camera parameters (e.g. rotation angle) or keypoints (e.g. eyes nose). Specifically camera parameters-conditional pose control models generate unrealistic images depending on the object owing to the small size of 3D datasets for training. Also keypoint-based approaches encounter challenges in acquiring reliable keypoints for various objects (e.g. church) or poses (e.g. back view). To address these limitations we propose depth-based pose control as depth maps are easily obtainable from a single depth estimation model regardless of objects and poses unlike camera parameters and keypoints. However depth-based pose control confronts issues of shape dependency as depth maps influence not only the pose but also the shape of the generated images. To tackle this issue we propose Skip-and-Play (SnP) designed via analysis of the impact of three components of depth-conditional ControlNet on the pose and the shape of the generated images. To be specific based on the analysis we selectively skip parts of the components to mitigate shape dependency on the depth map while preserving the pose. Through various experiments we demonstrate the superiority of SnP over baselines and showcase the ability of SnP to generate images of diverse objects and poses. Remarkably SnP exhibits the ability to generate images even when the objects in the condition (e.g. a horse) and the prompt (e.g. a hedgehog) differ from each other.

Graph based representation has been widely used in modelling spatio-temporal relationships in video understanding. Although effective existing graph-based approaches focus on capturing the human-object relationships while ignoring fine-grained semantic properties of the action components. These semantic properties are crucial for understanding the current situation such as where does the action takes place what tools are used and functional properties of the objects. In this work we propose a graph-based representation called Situational Scene Graph (SSG) to encode both human-object relationships and the corresponding semantic properties. The semantic details are represented as predefined roles and values inspired by situation frame which is originally designed to represent a single action. Based on our proposed representation we introduce the task of situational scene graph generation and propose a multi-stage pipeline Inter active and Complementary Network (InComNet) to address the task. Given that the existing datasets are not applicable to the task we further introduce a SSG dataset whose annotations consist of semantic role-value frames for human objects and verb predicates of human-object relations. Finally we demonstrate the effectiveness of our proposed SSG representation by testing on different downstream tasks. Experimental results show that the unified representation can not only benefit predicate classification and semantic role-value classification but also benefit reasoning tasks on human centric situation understanding. Code and data are available at https://github.com/LUNAProject22/SSG.

In industrial quality control detecting anomalies in visual textures is essential for ensuring product quality and operational efficiency. Early identification of defects prevents faulty items from reaching consumers reduces waste and maintains high standards of production. Numerous unsupervised anomaly detection methods heavily depend on the integration of multiple layers from various pre-trained models a selection often made through empirical means. We propose SingleNet an innovative knowledge distillation approach tailored for fast unsupervised texture anomaly detection using a single layer from a compact pre-trained model. Contrary to the previous knowledge distillation approaches our network leverages fast Fourier convolutions (FFC) to reconstruct a degraded version of the teacher extracted features. At test time we employed a frequency-aware filtering mechanism to reduce reconstruction artifacts caused by discrepancies between teacher and student architectures. Empirical results demonstrate the efficacy of our approach attaining state-of-the-art performance across evaluated datasets coupled with expedited high-speed inference.

Computer-assisted surgical (CAS) systems enhance surgical execution and outcomes by providing advanced support to surgeons. These systems often rely on deep learning models trained on complex challenging-to-annotate data. While synthetic data generation can address these challenges enhancing the realism of such data is crucial. This work introduces a multi-stage pipeline for generating realistic synthetic data featuring a fully-fledged surgical simulator that automatically produces all necessary annotations for modern CAS systems. This simulator generates a wide set of annotations that surpass those available in public synthetic datasets. Additionally it offers a more complex and realistic simulation of surgical interactions including the dynamics between surgical instruments and deformable anatomical environments outperforming existing approaches. To further bridge the visual gap between synthetic and real data we propose a lightweight and flexible image-to-image translation method based on Stable Diffusion (SD) and Low-Rank Adaptation (LoRA). This method leverages a limited amount of annotated data enables efficient training and maintains the integrity of annotations generated by our simulator. The proposed pipeline is experimentally validated and can translate synthetic images into images with real-world characteristics which can generalize to real-world context thereby improving both training and CAS guidance. The code and the dataset are available at https://github.com/SanoScience/SimuScope.

Out-of-context (OOC) misinformation poses a significant challenge in multimodal fact-checking where images are paired with texts that misrepresent their original context to support false narratives. Recent research in evidence-based OOC detection has seen a trend towards increasingly complex architectures incorporating Transformers foundation models and large language models. In this study we introduce a simple yet robust baseline which assesses MUltimodal SimilaritiEs (MUSE) specifically the similarity between image-text pairs and external image and text evidence. Our results demonstrate that MUSE when used with conventional classifiers like Decision Tree Random Forest and Multilayer Perceptron can compete with and even surpass the state-of-the-art on the NewsCLIPpings and VERITE datasets at less than 1% of their computational complexity. Furthermore integrating MUSE in our proposed "Attentive Intermediate Transformer Representations" (AITR) significantly improved performance by 3.3% and 7.5% on NewsCLIPpings and VERITE respectively. Nevertheless the success of MUSE relying on surface-level patterns and shortcuts without examining factuality and logical inconsistencies raises critical questions about how we define the task construct datasets collect external evidence and overall how we assess progress in the field. We release our code at: https://github.com/stevejpapad/outcontext-misinfo-progress.

Vision-based sign language recognition is an extensively researched problem aimed at advancing communication between deaf and hearing individuals. Numerous Sign Language Recognition (SLR) datasets have been introduced to promote research in this field spanning multiple languages vocabulary sizes and signers. However most existing popular datasets focus predominantly on the frontal view of signers neglecting visual information from other perspectives. In practice many sign languages contain words that have similar hand movements and expressions making it challenging to differentiate between them from a single frontal view. Although a few studies have proposed sign language datasets using multi-view data these datasets remain limited in vocabulary size and scale hindering their generalizability and practicality. To address this issue we introduce a new large-scale multi-view sign language recognition dataset spanning 1000 glosses and 30 signers resulting in over 84000 multi-view videos. To the best of our knowledge this is the first multi-view sign language recognition dataset of this scale. In conjunction with offering a comprehensive dataset we perform extensive experiments to assess the performance of state-of-the-art Sign Language Recognition models utilizing on our dataset. The findings indicate that utilizing multi-view data substantially enhances model accuracy across all models with a maximum performance improvement of up to 19.75% compared to models trained on single-view data. Our dataset and baseline models are publicly accessible on GitHub.

Multi-modal semantic segmentation significantly enhances AI agents' perception and scene understanding especially under adverse conditions like low-light or overexposed environments. Leveraging additional modalities (X-modality) like thermal and depth alongside traditional RGB provides complementary information enabling more robust and reliable segmentation. In this work we introduce Sigma a Siamese Mamba network for multi-modal semantic segmentation utilizing the Selective Structured State Space Model Mamba. Unlike conventional methods that rely on CNNs with their limited local receptive fields or Vision Transformers (ViTs) which offer global receptive fields at the cost of quadratic complexity our model achieves global receptive fields coverage with linear complexity. By employing a Siamese encoder and innovating a Mamba fusion mechanism we effectively select essential information from different modalities. A decoder is then developed to enhance the channel-wise modeling ability of the model. Our method Sigma is rigorously evaluated on both RGB-Thermal and RGB-Depth segmentation tasks demonstrating its superiority and marking the first successful application of State Space Models (SSMs) in multi-modal perception tasks. Code is available at https://github.com/zifuwan/Sigma.

In the study of animal behavior researchers often record long continuous videos accumulating into large-scale datasets. However the behaviors of interest are often rare compared to routine behaviors. This incurs a heavy cost on manual annotation forcing users to sift through many samples before finding their needles. We propose a pipeline to efficiently sample rare behaviors from large datasets enabling the creation of training datasets for rare behavior classifiers. Our method only needs an unlabeled animal pose or acceleration dataset as input and makes no assumptions regarding the type number or characteristics of the rare behaviors. Our pipeline is based on a recent graph-based anomaly detection model for human behavior which we apply to this new data domain. It leverages anomaly scores to automatically label normal samples while directing human annotation efforts toward anomalies. In research data anomalies may come from many different sources (e.g. signal noise versus true rare instances). Hence the entire labeling budget is focused on the abnormal classes letting the user review and label samples according to their needs. We tested our approach on three datasets of freely-moving animals acquired in the laboratory and the field. We found that graph-based models are particularly useful when studying motion-based behaviors in animals yielding good results while using a small labeling budget. Our method consistently outperformed traditional random sampling offering an average improvement of 70% in performance and creating datasets even when the behavior of interest was only 0.02% of the data. Even when the performance gain was minor (e.g. when the behavior is not rare) our method still reduced the annotation effort by half.