WACV 2025 论文列表

We introduce RealSync a novel dataset designed to significantly enhance the training and evaluation of models for audio-visual synchronization (AV Sync) tasks. Sourced from high-quality YouTube channels RealSync covers a wide range of content domains providing an improved scale diversity and alignment with broadcast content compared to existing datasets. It features extended-length video samples catering to the critical need for more comprehensive real-world training and evaluation materials. Alongside this dataset we present StreamSync a model tailored for real-world AV Sync applications. StreamSync is designed to be backbone agnostic and incorporates a streaming mechanism that processes consecutive video segments dynamically iteratively refining synchronization predictions. This innovative approach enables StreamSync to outperform existing models offering superior synchronization accuracy with minimal computational cost per iteration. Together our dataset and the StreamSync model establish a new benchmark for AVSync research promising to drive the development of more robust and practical AVSync methods. https://github.com/jvoas655/StreamSync

We study the challenging problem of simultaneously localizing a sequence of queries in the form of instructional diagrams in a video. This requires understanding not only the individual queries but also their interrelationships. However most existing methods focus on grounding one query at a time ignoring the inherent structures among queries such as the general mutual exclusiveness and the temporal order. Consequently the predicted timespans of different step diagrams may overlap considerably or violate the temporal order thus harming the accuracy. In this paper we tackle this issue by simultaneously grounding a sequence of step diagrams. Specifically we propose composite queries constructed by exhaustively pairing up the visual content features of the step diagrams and a fixed number of learnable positional embeddings. Our insight is that self-attention among composite queries carrying different content features suppress each other to reduce timespan overlaps in predictions while the cross-attention corrects the temporal misalignment via content and position joint guidance. We demonstrate the effectiveness of our approach on the IAW dataset for grounding step diagrams and the YouCook2 benchmark for grounding natural language queries significantly outperforming existing methods while simultaneously grounding multiple queries.

Visual datasets are generally constructed from the samples available at the time of their collection and are not further updated. However these static datasets do not reflect the distribution changes that occur in real data. We analyze how different collection times lead to a shift in class distribution by collecting a set of Flickr images published over 14 years. The proposed "Visual Classes through Time" (VCT-107) dataset contains images tagged by their publication date and includes 107 classes covering various topics (human-made objects animals plants food etc.). Images from each class are divided into five collection periods to study the impact of time on classification accuracy. When training different classification models using linear probing we observe an accuracy loss when training on data from one period and testing on other periods. This happens even in the case of a strongly pre-trained model like DinoV2 ViT-B/14. Intuitively the performance loss is generally more significant when the collection periods between the training and test data are further apart. Our analysis reveals that the temporal shift varies between classes with the largest shifts observed for human-made objects and the smallest for natural concepts such as animal species. Our results stress the importance of regularly updating models to adapt to time-induced changes in the distribution of visual classes even when using a strongly pre-trained model. We release the VCT-107 dataset to facilitate research on temporal shifts.

Longitudinal medical image processing is a significant task to understand the dynamic changes of disease by taking and comparing image series over time providing insights into how conditions evolve and enabling more accurate diagnosis and treatment planning. While recent advancements in biomedical Vision-Language Pre-training (VLP) have enabled label-efficient representation learning with paired medical images and reports existing methods primarily pair a single image with the corresponding textual report limiting their ability to capture temporal relationships. To address this limitation it is essential to learn temporal-aware cross-modal representations from sequential medical images and text reports that highlight the temporal changes occurring between examinations. Specifically we introduce TempA-VLP a temporal-aware vision language pre-training framework with a cross-exam encoder to integrate the information from both prior and current examinations. This approach enables the model to capture dynamic representations that reflect disease progression over time which allows us to (i) achieve state-of-the-art performance in disease progression classification (ii) localize dynamic progression regions across consecutive examinations as demonstrated in our new task dynamic phrase grounding on the Chest-Imagenome Gold dataset and (iii) highlight progression localized regions often relevant to lesion areas which in turn improves disease classification tasks on a single image.

We present TaxaBind a unified embedding space for characterizing any species of interest. TaxaBind is a multimodal embedding space across six modalities: ground-level images of species geographic location satellite image text audio and environmental features useful for solving ecological problems. To learn this joint embedding space we leverage ground-level images of species as a binding modality. We propose multimodal patching a technique for effectively distilling the knowledge from various modalities into the binding modality. We construct two large datasets for pretraining: iSatNat with species images and satellite images and iSoundNat with species images and audio. Additionally we introduce TaxaBench-8k a diverse multimodal dataset with six paired modalities for evaluating deep learning models on ecological tasks. Experiments with TaxaBind demonstrate its strong zero-shot and emergent capabilities on a range of tasks including species classification cross-model retrieval and audio classification. The datasets and models are made available at https://github.com/mvrl/TaxaBind.

Many industrial image segmentation systems require training on large annotated datasets but there is little standardization for producing training annotations. Data is often obtained via crowdsourcing but many labeling task configurations are set by convenience and may have unintended effects on data quality. In this work we (1) demonstrate a new software tool for running crowdsourced image segmentation experiments (2) present a dataset capturing variation in segmentation annotations produced under different task configurations and (3) experimentally evaluate the quality of the annotations produced by these different configurations. We show annotation quality can be significantly improved by paying annotators per annotated object rather than per image and by leveraging paintbrush-style drawing tools rather than polygon or curve drag tools. We also show that some task complexity is required to maintain annotator engagement and sufficient task performance. Finally we show that many configuration-related annotation errors degrade model training performance but that models can tolerate error error patterns that are common across crowdsourced annotation schemes.

We study the problem of creating a character model that can be controlled in real time from a single image of an anime character. A solution would greatly reduce the cost of creating avatars computer games and other interactive applications. Talking Head Anime 3 (THA3) is an open source project that attempts to directly address the problem. It takes as input (1) an image of an anime character's upper body and (2) a 45-dimensional pose vector and outputs a new image of the same character taking the specified pose. The range of possible movements is expressive enough for personal avatars and certain types of game characters. THA3's main limitation is its speed. It can achieve interactive frame rates (approximately 20 FPS) only if it is run on a very powerful GPU (Nvidia Titan RTX or better). Based on the insight that avatars and game characters do not need to change their appearance every so often we propose a technique to distill the system into a small student neural network (< 2 MB) specific to a particular character. The student model can generate 512x512 animation frames in real time (no more than 30 FPS) using consumer gaming GPUs while preserving the image quality of the teacher model. For the first time our technique makes the whole system practical for real-time applications.

The performance of perception tasks is heavily influenced by imaging systems. However designing cameras with high task performance is costly requiring extensive camera knowledge and experimentation with physical hardware. Additionally cameras and perception tasks are mostly designed in isolation whereas recent methods that jointly design cameras and tasks have shown improved performance. Therefore we present a novel end-to-end optimization approach that co-designs cameras with specific vision tasks. This method combines derivative-free and gradient-based optimizers to support both continuous and discrete camera parameters within manufacturing constraints. We leverage recent computer graphics techniques and physical camera characteristics to simulate the cameras in virtual environments making the design process cost-effective. We validate our simulations against physical cameras and provide a procedurally generated virtual environment. Our experiments demonstrate that our method designs cameras that outperform common off-the-shelf options and more efficiently compared to the state-of-the-art approach requiring only 2 minutes to design a camera on an example experiment compared with 67 minutes for the competing method. Designed to support the development of cameras under manufacturing constraints multiple cameras and unconventional cameras we believe this approach can advance the fully automated design of cameras. Code is available on our project page at https://roboticimaging.org/Projects/TaCOS/.

Camera-based estimation of vital signs is a promising method for non-contact health monitoring which analyzes minute changes in video data. However the creation of accurate models for this task is challenging due to the scarcity of datasets that possess synchronized vital sign recordings. Our research enhances an existing non-contrastive unsupervised learning technique for extracting rPPG signals which does not necessitate ground-truth signals during the training process. We have incorporated new time-domain loss functions and added a feature stabilization block to improve the model's stability and accuracy in detecting low-level features. Additionally we have devised a metric to evaluate the feature instability in the model's final layer. Our experiments on four public datasets demonstrate that our method surpasses the performance of current state-of-the-art methods. These advancements make our approach a significant breakthrough in the development of scalable deep-learning models for camera-based heart-rate estimation.

Thermal cameras offer unique detection capabilities in building inspections search and rescue operations and autonomous vehicle perception. Of the different types of thermal cameras uncooled microbolometers are often chosen due to their relative affordability small size and low power consumption. However microbolometers suffer from motion blur rolling shutter distortions and fixed pattern noise which limit the conditions of their use. Nearly all prior methods for microbolometer image restoration account for only one of these degradations and current techniques addressing microbolometer blur and rolling shutter are limited. This paper presents TRNeRF a thermal image restoration method that jointly addresses all three degradations by incorporating the microbolometer image formation model with Neural Radiance Fields (NeRFs). To evaluate TRNeRF this paper introduces a new real-world dataset that is uniquely designed to support two novel quantitative evaluation strategies for thermal image restoration. Experiments demonstrate that TRNeRF is able to recover sharp global shutter and clear thermal images even under extremely aggressive camera motion that causes existing methods to fail. The code and dataset are available at: https://umautobots.github.io/trnerf.

Despite advancements in visual question answering challenges persist with documents like financial reports often structured in complicated tabular structures with complex numerical computations. An alternative approach the pipeline-driven methodology includes table recognition (TR) and table question-answering (TQA). Recent advancements in TR support this approach with better accuracy and interpretability. However real-world tables usually represent hierarchical tables. They pose additional challenges due to merged cells and indents necessitating a specific approach for hierarchical relationship extraction. In this paper we propose TRH2TQA (Table Recognition with Hierarchical Relationships to Table Question-Answering) for business table images. It consists of three modules on table images with question-answer pairs. First the TR module extracts structure and textual content from table images into HTML format. Second post-structure extraction is applied to identify header and hierarchical relationships using predicted column span and bounding box. Finally this information is combined with natural language questions in the TQA module to generate the answer through the decoder. In extensive experiments TRH2TQA outperforms in question-answering performance on the VQAonBD 2023 dataset.

Attention guides our gaze to fixate the proper location of the scene and holds it in that location for the deserved amount of time given current processing demands before shifting to the next one. As such gaze deployment crucially is a temporal process. Existing computational models have made significant strides in predicting spatial aspects of observer's visual scanpaths (where to look) while often putting on the background the temporal facet of attention dynamics (when). In this paper we present TPP-Gaze a novel and principled approach to model scanpath dynamics based on Neural Temporal Point Process (TPP) that jointly learns the temporal dynamics of fixations position and duration integrating deep learning methodologies with point process theory. We conduct extensive experiments across five publicly available datasets. Our results show the overall superior performance of the proposed model compared to state-of-the-art approaches. Source code and trained models are publicly available at: https://github.com/phuselab/tppgaze.

Regressing text in natural scenes with polygonal representations is challenging due to shape prediction difficulties. To address this we introduce Text Polygon Detection with Split Transformers (TPD-STR) which directly regresses polygonal points. TPD-STR incorporates the Decoder Split (DS) architecture to separate polygonal point regression and textness classification and the Positional Information Propagation (PIP) module to enhance classification. Both modules are effective and compatible with existing methods. TPD-STR achieves state-of-the-art (SOTA) performance among regression-based methods surpassing segmentation-based methods on MSRA-TD500 without external data. Adding DS and PIP to existing models further improves performance. Experiments demonstrate the model's ability to detect text instances effectively.

Active Visual Exploration (AVE) optimizes the utilization of robotic resources in real-world scenarios by sequentially selecting the most informative observations. However modern methods require a high computational budget due to processing the same observations multiple times through the autoencoder transformers. As a remedy we introduce a novel approach to AVE called TOken REcycling (TORE). It divides the encoder into extractor and aggregator components. The extractor processes each observation separately enabling the reuse of tokens passed to the aggregator. Moreover to further reduce the computations we decrease the decoder to only one block. Through extensive experiments we demonstrate that TORE outperforms state-of-the-art methods while reducing computational overhead by up to 90%.

An image classifier may depend on incidental features stemming from a strong correlation between the feature and the classification target in the training dataset. Recently Last Layer Retraining (LLR) with group-balanced datasets is shown to be efficient in mitigating the spurious correlation of classifiers. However the acquisition of image-based group-balanced datasets is costly which hinders the general applicability of the LLR method. In this work we propose to perform LLR based on text datasets built with large language models to debias a general image classifier. To that end we demonstrate that text can generally be a proxy for its corresponding image beyond the image-text joint embedding space which is achieved with a linear projector that ensures orthogonality between its weight and the modality gap of the joint embedding space. In addition we propose a systematic validation procedure that checks whether the generated words are compatible with the embedding space of CLIP and the image classifier which is shown to be effective for improving debiasing performance. We dub these procedures as TLDR (Text-based Last layer retraining for Debiasing image classifieRs) and show our method achieves the performance that is competitive with the LLR methods that require group-balanced image dataset for retraining. Furthermore TLDR outperforms other baselines that involve training the last layer without any group annotated dataset. Codes: https://github.com/beotborry/TLDR

The potential of Open-Vocabulary Semantic Segmentation (OVSS) in few-shot scenarios is not fully explored due to the complexity of extending few-shot concepts to semantic segmentation tasks. To address this challenge we propose Training-Free Mask Matching (TFM^2) an efficient mask-based adapter method that enhances OVSS models for the few-shot open vocabulary semantic segmentation task. TFM^2 is a key-value cache that explicitly designed for image masks. We introduce three modules to construct and refine the mask cache subsequently enhancing the OVSS mask classification performance. Comprehensive experiments demonstrate that TFM^2 improves the performance of state-of-the-art OVSS methods by a margin of 1% to 5% across different settings. Moreover TFM^2 is not limited to any specific methods or backbones. This work underscores the importance and potential of few-shot data in OVSS and presents a significant step toward leveraging this potential.

Video Object Segmentation (VOS) has emerged as an increasingly important problem with availability of larger datasets and more complex and realistic settings which involve long videos with global motion (e.g. in egocentric settings) depicting small objects undergoing both rigid and non-rigid (including state) deformations. While a number of recent approaches have been explored for this task these data characteristics still present challenges. In this work we propose a novel clip-based DETR-style encoder-decoder architecture which focuses on systematically analyzing and addressing aforementioned challenges. Specifically we propose a novel transformation-aware loss that focuses learning on portions of the video where an object undergoes significant deformations - a form of "soft" hard examples mining. Further we propose a multiplicative time-coded memory beyond vanilla additive positional encoding which helps propagate context across long videos. Finally we incorporate these in our proposed holistic multi-scale video transformer for tracking via multi-scale memory matching and decoding to ensure sensitivity and accuracy for long videos and small objects. Our model enables on-line inference with long videos in a windowed fashion by breaking the video into clips and propagating context among them. We illustrate that short clip length and longer memory with learned time-coding are important design choices for improved performance. Collectively these technical contributions enable our model to achieve new state-of-the-art (SoTA) performance on two complex egocentric datasets - VISOR [13] and VOST [44] while achieving comparable to SoTA results on the conventional VOS benchmark DAVIS'17 [38]. Detailed ablations vali date our design choices and provide insights into the impor tance of parameter choices and impact on performance.

We propose a novel TACLE (TAsk and CLass-awarE) framework for the relatively unexplored and challenging problem of exemplar-free semi-supervised class incremental learning. In this scenario at each new task the model has to learn new classes from both (few) labeled and unlabeled data without access to exemplars from previous classes. In addition to leveraging the capabilities of pre-trained models TACLE proposes a novel task-adaptive threshold thereby maximizing the utilization of the available unlabeled data as incremental learning progresses. Additionally to enhance the performance of the under-represented classes within each task we propose a class-aware weighted cross-entropy loss. We also exploit the unlabeled data for classifier alignment which further enhances the model performance. Extensive experiments on benchmark datasets namely CIFAR10 CIFAR100 and ImageNet-Subset100 demonstrate the effectiveness of the proposed TACLE framework. We further showcase its effectiveness when the unlabeled data is imbalanced and also for the extreme case of one labeled example per class. code: https://github.com/rokmr/TACLE

Automatically generating realistic musical performance motion can greatly enhance digital media production often involving collaboration between professionals and musicians. However capturing the intricate body hand and finger movements required for accurate musical performances is challenging. Existing methods often fall short due to the complex mapping between audio and motion typically requiring additional inputs like scores or MIDI data. In this work we present SyncViolinist a multi-stage end-to-end framework that generates synchronized violin performance motion solely from audio input. Our method overcomes the challenge of capturing both global and fine-grained performance features through two key modules: a bowing/fingering module and a motion generation module. The bowing/fingering module extracts detailed playing information from the audio which the motion generation module uses to create precise coordinated body motions reflecting the temporal granularity and nature of the violin performance. We demonstrate the effectiveness of SyncViolinist with significantly improved qualitative and quantitative results from unseen violin performance audio outperforming state-of-the-art methods. Extensive subjective evaluations involving professional violinists further validate our approach. The code and dataset are available at https://github.com/Kakanat/SyncViolinist.