WACV 2025 论文列表

We consider the problem of constraining diffusion model outputs with a user-supplied reference image. Our key objective is to extract multiple attributes (e.g. color object layout style) from this single reference image and then generate new samples and novel compositions with them. We first perform an extensive attribute distribution analysis that leads to the discovery of an extended conditioning space consisting of multiple textual conditions. These textual conditions vary both per-layer of the U-Net as well as per-timestep of the denoising process. We observe that although the extended conditioning space provides greater control over different attributes of the generated image often a subset of these attributes are captured in the same set of U-Net layers and/or across same denoising timesteps. For instance color and style are captured across same U-Net layers whereas layout and color are captured across same timestep stages. Existing works in multi-attribute constrained image generation proposed extending textual inversion by learning per-layer or per-timestep tokens but they suffer from attribute entanglement for the reasons described above. To address the aforementioned gap we introduce our second contribution where we design a new multi-attribute textual inversion algorithm MATTE with associated disentanglement-enhancing regularization losses that operates jointly across both layer and timestep dimensions and explicitly leads to four disentangled tokens (color style layout and object). We conduct extensive qualitative and quantitative evaluations to demonstrate the effectiveness of the proposed approach.

Recent advances in Large Language Models (LLMs) have enabled the semantic description of textures in natural language aiming to capture them in richer detail. However most methods are confined to either depending on supervised training with pairs of images and manually annotated visual attributes that most texture datasets lack or using Vision-Language Models (VLMs) such as CLIP. In this paper we develop an encoder-agnostic Weakly supervised Texture Description Generator (WTDG) that employs a novel Scaled Ranked Kullback-Leibler divergence (SR-KL) loss between image and text modalities. Within the SR-KL loss formulation we leverage category information which is always available as ground-truths for all benchmark texture recognition datasets. We further extend our proposed WTDG to assist in texture recognition by using its generated texture descriptions. Thus we develop a multimodal framework called Tex^2 which is adept at simultaneous generation of texture description and recognition. Our approach exhibits promising performance in describing and recognizing textures on benchmark datasets.

Visual images corrupted by various types and levels of degradations are commonly encountered in practical image compression. However most existing image compression methods are tailored for clean images therefore struggling to achieve satisfying results on these images. Joint compression and restoration methods typically focus on a single type of degradation and fail to address a variety of degradations in practice. To this end we propose a unified framework for all-in-one image compression and restoration which incorporates the image restoration capability against various degradations into the process of image compression. The key challenges involve distinguishing authentic image content from degradations and flexibly eliminating various degradations without prior knowledge. Specifically the proposed framework approaches these challenges from two perspectives: ie content information aggregation and degradation representation aggregation. Extensive experiments demonstrate the following merits of our model: 1) superior rate-distortion (RD) performance on various degraded inputs while preserving the performance on clean data; 2) strong generalization ability to real-world and unseen scenarios; 3) higher computing efficiency over compared methods. Our code is available at https://github.com/ZeldaM1/All-in-one.

We consider the problem of customizing text-to-image diffusion models with user-supplied reference images. Given new prompts the existing methods can capture the key concept from the reference images but fail to align the generated image with the prompt. In this work we seek to address this key issue by proposing new methods that can easily be used in conjunction with existing customization methods that optimize the embeddings and weights at various intermediate stages of the text encoding process before being fed into the noise prediction model of a text-to-image diffusion model. The first contribution of this paper is a dissection of the various stages of the text encoding process leading up to the conditioning vector for text-to-image models. We take a holistic view of existing customization methods and notice that key and value outputs from this process differs substantially from their corresponding baseline (non-customized) models. While this difference does not impact the concept being customized it leads to other parts of the generated image not being aligned with the prompt. Further we also observe that these keys and values allow independent control various aspects of the final generation enabling semantic manipulation of the output. Taken together the features spanning these keys and values serve as the basis for our next contribution where we fix the aforementioned issues with existing methods. We propose a new post-processing algorithm AlignIT that infuses the keys and values for the concept of interest while ensuring the keys and values for all other tokens in the input prompt are unchanged. Our proposed method can be plugged in directly to existing customization methods leading to a substantial performance improvement in the alignment of the final result with the input prompt while retaining the customization quality. We conduct extensive experiments across various different customization methods and a wide variety of reference images and show consistent improvements both qualitatively and quantitatively.

Agricultural domains are being transformed by recent advances in AI and computer vision that support quantitative visual evaluation. Using aerial and ground imaging over a time series we develop a framework for characterizing the ripening process of cranberry crops a crucial component for precision agriculture tasks such as comparing crop breeds (high-throughput phenotyping) and detecting disease. Using drone imaging we capture images from 20 waypoints across multiple bogs and using ground-based imaging (hand-held camera) we image same bog patch using fixed fiducial markers. Both imaging methods are repeated to gather a multi-week time series spanning the entire growing season. Aerial imaging provides multiple samples to compute a distribution of albedo values. Ground imaging enables tracking of individual berries for a detailed view of berry appearance changes. Using vision transformers (ViT) for feature detection after segmentation we extract a high dimensional feature descriptor of berry appearance. Interpretability of appearance is critical for plant biologists and cranberry growers to support crop breeding decisions (e.g. comparison of berry varieties from breeding programs). For interpretability we create a 2D manifold of cranberry appearance by using a UMAP dimensionality reduction on ViT features. This projection enables quantification of ripening paths and a useful metric of ripening rate. We demonstrate the comparison of four cranberry varieties based on our ripening assessments. This work is the first of its kind and has future impact for cranberries and for other crops including wine grapes olives blueberries and maize. Aerial and ground datasets are made publicly available.

Significant progress has been made in advancing large multimodal conversational models (LMMs) capitalizing on vast repositories of image-text data available online. Despite this progress these models often encounter substantial domain gaps hindering their ability to engage in complex conversations across new domains. Recent efforts have aimed to mitigate this issue albeit relying on domain-specific image-text data to curate instruction-tuning data. However many domains such as agriculture lack such vision-language data. In this work we propose an approach to construct instruction-tuning data that harnesses vision-only data for the agriculture domain. We utilize diverse agricultural datasets spanning multiple domains curate class-specific information and employ large language models (LLMs) to construct an expert-tuning set resulting in a 70k AgroInstruct. Subsequently we expert-tuned and created AgroGPT an efficient LMM that can hold complex agriculture-related conversations and provide useful insights. We also develop AgroEvals for evaluation and compare AgroGPT's performance with large open and closed-source models. AgroGPT excels at identifying fine-grained agricultural concepts can act as an agriculture expert and provides helpful information for multimodal agriculture questions. The code datasets and models are available at https://github.com/awaisrauf/agroGPT.

Analysis of 3D segmentation models especially in the context of medical imaging is often limited to segmentation performance metrics that overlook the crucial aspect of explainability and bias. Currently effectively explaining these models with saliency maps is challenging due to the high dimensions of input images multiplied by the ever-growing number of segmented class labels. To this end we introduce Agg^2Exp a methodology for aggregating fine-grained voxel attributions of the segmentation model's predictions. Unlike classical explanation methods that primarily focus on the local feature attribution Agg^2Exp enables a more comprehensive global view on the importance of predicted segments in 3D images. Our benchmarking experiments show that gradient-based voxel attributions are more faithful to the model's predictions than perturbation-based explanations. As a concrete use-case we apply Agg^2Exp to discover knowledge acquired by the Swin UNEt TRansformer model trained on the TotalSegmentator v2 dataset for segmenting anatomical structures in computed tomography medical images. Agg^2Exp facilitates the explanatory analysis of large segmentation models beyond their predictive performance. The source code is publicly available at https://github.com/MI2DataLab/agg2exp.

Drones excel at capturing aerial-view images especially in human unreachable areas. Automatically interpreting and describing these images enables decision-making easier without the need to review the images extensively. Traditional image captioning models struggle with aerial imagery due to diverse orientations perspectives and unclear objects. Integrating the capabilities of Large Vision Language Models (LVLMs) with drone images can improve description utility benefiting strategic missions like surveillance search and rescue etc. However the lack of image-caption datasets for drone imagery poses a significant challenge for training and evaluating drone image captioning. To address this gap we contribute the first Aerial-view Image Captioning dataset (AeroCaps) containing four captions per image. Another major hurdle for the task is the hallucinatory nature of LVLMs. To this end we perform the first extensive analysis of hallucinations on aerial imagery by two SOTA LVLMs - LLaVA and InstructBLIP on our proposed dataset and VisDrone. We explore the reasons behind such hallucinations. We release the LVLM-generated image captions along with our hallucination-labelled annotations as the Labelled Illusion Dataset (LID) for further research. Additionally we review how effective advanced LLMs like GPT-4 are in evaluating the degree of hallucinations made by other LVLMs like LLaVA.

The significant improvements in point cloud representation learning have increased its applicability in many real-life applications resulting in the need for lightweight better-performing models. One widely proposed efficient method is knowledge distillation where a lightweight model uses knowledge from large models. Very few works exist on distilling the knowledge for point clouds. Most of the work focuses on cross-modal-based approaches that make the method expensive to train. This paper proposes PointKAD an adversarial knowledge distillation framework for point cloud-based tasks. PointKAD includes adversarial feature distillation and response distillation with the help of discriminators to extract and distill the representation of feature maps and logits. We conduct extensive experimental studies on both synthetic (ModelNet40) and real (ScanObjectNN) datasets to show that PointKAD achieves state-of-the-art results compared to the existing knowledge distillation methods for point cloud classification. Additionally we present results on the part segmentation task highlighting the efficacy of the PointKAD framework. Our experiments further reveal that PointKAD is capable of transferring knowledge across different tasks and datasets showcasing its versatility. Furthermore we demonstrate that PointKAD can be applied to a cross-modal training setup achieving competitive performance with cross-modal-based point cloud methods for classification.

Deformable vision transformers reduce the expensive quadratic time-complexity of attention modeling by using sparse attention structures making it possible to use transformers in large-scale vision applications such as multi-view vision systems. We show that existing adversarial attacks against conventional vision transformers do not transfer to deformable transformers primarily due to the data-dependent dynamic nature of sparse attention. In this work we present for the first time adversarial attacks against deformable vision transformers by getting control of their attention-inferring module. We develop a novel collaborative attack where a source patch manipulates attention to point to a target patch containing the adversarial noise which fools the model. We observe that our attack alters less than 1% of the patched area in the input field completely disrupting object detection and resulting in 0% AP in single-view object detection using MS COCO and 0% MODA in multi-view object detection using Wildtrack.

Pruning aims to accelerate and compress models by removing redundant parameters identified by specifically designed importance scores which are usually imperfect. This removal is irreversible often leading to subpar performance in pruned models. Dynamic sparse training while attempting to adjust sparse structures during training for continual reassessment and refinement has several limitations including criterion inconsistency between pruning and growth unsuitability for structured sparsity and short-sighted growth strategies. Our paper introduces an efficient innovative paradigm to enhance a given importance criterion for either unstructured or structured sparsity. Our method separates the model into an active structure for exploitation and an exploration space for potential updates. During exploitation we optimize the active structure whereas in exploration we reevaluate and reintegrate parameters from the exploration space through a pruning and growing step consistently guided by the same given importance criterion. To prepare for exploration we briefly "reactivate" all parameters in the exploration space and train them for a few iterations while keeping the active part frozen offering a preview of the potential performance gains from reintegrating these parameters. We show on various datasets and configurations that existing importance criterion even simple as magnitude can be enhanced with ours to achieve state-of-the-art performance and training cost reductions. Notably on ImageNet with ResNet50 ours achieves an +1.3 increase in Top-1 accuracy over prior art at 90% ERKsparsity. Compared with the SOTA latency pruning method HALP we reduced its training cost by over 70% while attaining a faster and more accurate pruned model.

Chart comprehension presents significant challenges for machine learning models due to the diverse and intricate shapes of charts. Existing multimodal methods often overlook these visual features or fail to integrate them effectively for Chart Question Answering. To address this we introduce ChartFormer a unified framework that enhances chart component recognition by accurately identifying and classifying components such as bars lines pies titles legends and axes. Additionally we propose a novel Question-guided Deformable Co-Attention (QDCAt) mechanism which fuses chart features encoded by ChartFormer with the given question leveraging the question's guidance to ground the correct answer. Extensive experiments demonstrate a 3.2% improvement in mAP over the baselines for chart component recognition. For ChartQA and OpenCQA tasks our approach achieves improvements of 15.4% in accuracy and 0.8 in BLEU score respectively underscoring the robustness of our solution for detailed visual data interpretation across various applications. The source code and dataset are publicly available at https://github.com/VT-NLP/chartQA.

3D Gaussian Splatting has recently achieved notable success in novel view synthesis for dynamic scenes and geometry reconstruction in static scenes. Building on these advancements early methods have been developed for dynamic surface reconstruction by globally optimizing entire sequences. However reconstructing dynamic scenes with significant topology changes emerging or disappearing objects and rapid movements remains a substantial challenge particularly for long sequences. To address these issues we propose AT-GS a novel method for reconstructing high-quality dynamic surfaces from multi-view videos through per-frame incremental optimization. To avoid local minima across frames we introduce a unified and adaptive gradient-aware densification strategy that integrates the strengths of conventional cloning and splitting techniques. Additionally we reduce temporal jittering in dynamic surfaces by ensuring consistency in curvature maps across consecutive frames. Our method achieves superior accuracy and temporal coherence in dynamic surface reconstruction delivering high-fidelity space-time novel view synthesis even in complex and challenging scenes. Extensive experiments on diverse multi-view video datasets demonstrate the effectiveness of our approach showing clear advantages over baseline methods.

Visual anomaly detection targets to detect images that notably differ from normal pattern and it has found extensive application in identifying defective parts within the manufacturing industry. These anomaly detection paradigms predominantly focus on training detection models using only clean unlabeled normal samples assuming an absence of contamination; a condition often unmet in real-world scenarios. The performance of these methods significantly depends on the quality of the data and usually decreases when exposed to noise. We introduce a systematic adaptive method that employs deviation learning to compute anomaly scores end-to-end while addressing data contamination by assigning relative importance to the weights of individual instances. In this approach the anomaly scores for normal instances are designed to approximate scalar scores obtained from the known prior distribution. Meanwhile anomaly scores for anomaly examples are adjusted to exhibit statistically significant deviations from these reference scores. Our approach incorporates a constrained optimization problem within the deviation learning framework to update instance weights resolving this problem for each mini-batch. Comprehensive experiments on the MVTec and VisA benchmark datasets indicate that our proposed method surpasses competing techniques and exhibits both stability and robustness in the presence of data contamination.

Continual learning allows systems to continuously learn and adapt to the tasks in an evolving real-world environment without forgetting previous tasks. Developing deep learning models that can continually learn over a sequence of tasks is challenging. We propose a novel method AdaPrefix which addresses this and empowers continual learning capability in pretrained large models (PLMs). AdaPrefix provide a continual learning method for transformer-based deep learning models by appropriately integrating the parameter-efficient methods adapters and prefixes. AdaPrefix is an effective approach for smaller PLMs and achieves better results than state-of-the-art approaches. We further improve upon AdaPrefix by proposing AdaPrefix++ enabling knowledge transfer across the tasks. It leverages hypernetworks to generate prefixes and continually learns the hypernetwork parameters to facilitate knowledge transfer. AdaPrefix++ has a smaller parameter growth compared to AdaPrefix and is more effective and valuable for continual learning in PLMs. We performed several experiments on various benchmark datasets to demonstrate the performance of our approach for different PLMs and continual learning scenarios.

Video-to-video synthesis poses significant challenges in maintaining character consistency smooth temporal transitions and preserving visual quality during fast motion. While recent fully cross-frame self-attention mechanisms have improved character consistency across multiple frames they come with high computational costs and often include redundant operations especially for videos with higher frame rates. To address these inefficiencies we propose an adaptive motion-guided cross-frame attention mechanism that selectively reduces redundant computations. This enables a greater number of cross-frame attentions over more frames within the same computational budget thereby enhancing both video quality and temporal coherence. Our method leverages optical flow to focus on moving regions while sparsely attending to stationary areas allowing for the joint editing of more frames without increasing computational demands. Traditional frame interpolation techniques struggle with motion blur and flickering in intermediate frames which compromises visual fidelity. To mitigate this we introduce KV-caching for jointly edited frames reusing keys and values across intermediate frames to preserve visual quality and maintain temporal consistency throughout the video. With our adaptive cross-frame self-attention approach we achieve a threefold increase in the number of keyframes processed compared to existing methods all within the same computational budget as fully cross-frame attention baselines. This results in significant improvements in prediction accuracy and temporal consistency outperforming state-of-the-art approaches. Code is made publicly available at https://github.com/tanvir-utexas/AdaVE/tree/main.

Transformer has recently gained tremendous popularity in unsupervised domain adaptation tasks due to its superior generalization ability. State-of-the-art methods leverage mixup to build an intermediate domain to reduce domain gap. However such strategy becomes less effective when the domain gap becomes large as the domain gap between intermediate domain and source domain is not minimized and the constructed intermediate domain is non-informative. How to address the adaptation problem when domain gap becomes large is an important research problem in domain adaptation. In this paper we propose an adversarial and adaptive mixup (Ad^2mix) framework which gradually aligns the intermediate domain towards source domain to fully unleash the potential of both the transformer architecture and mixup to address the large domain gap problem. Specifically we formulate a general framework for intermediate domain learning with mixup. We propose adversarial mixup with a specially designed mixup alike adversarial adaptation operation to reduce the domain gap between the intermediate domain and source domain. To construct an informative intermediate domain unlike existing methods which utilize a Beta distribution to generate mixup coefficients to interpolate source and target data we adaptively assign mixup coefficient for each target data instance based on their transferability and discriminativity information. Our framework creates a natural curriculum of intermediate domains from near source domain to near target domain for gradual adaptation. Extensive experimental studies and evaluations on three public domain adaptation benchmark datasets and one medical domain adaptation task demonstrate the superiority of our framework.

In the rapidly evolving landscape of digital marketing effective customer engagement through advertisements is crucial for brands. Thus computational understanding of ads is pivotal for recommendation authoring and customer behaviour simulation. Despite advancements in knowledge-guided visual-question-answering (VQA) models existing frameworks often lack domain-specific responses and suffer from a dearth of benchmark datasets for advertisements. To address this gap we introduce ADVQA the first dataset for ad-related VQA sourced from Facebook and X (twitter) which facilitates further research in ad comprehension. It comprises open-ended questions and detailed context obtained automatically from web articles. Moreover we present AdQuestA a novel multimodal framework for knowledge-guided open-ended question-answering tailored to advertisements. AdQuestA leverages a Retrieval Augmented Generation (RAG) to obtain question-aware ad context as explicit knowledge and image-grounded implicit knowledge effectively exploiting inherent relationships for reasoning. Extensive experiments corroborate its efficacy yielding state-of-the-art performance on the ADVQA dataset even surpassing 10X larger models such as GPT-4 on this task. Our framework not only enhances understanding of ad content but also advances the broader landscape of knowledge-guided VQA models.

Multi-class semantic segmentation remains a cornerstone challenge in computer vision. Yet dataset creation remains excessively demanding in time and effort especially for specialized domains. Active Learning (AL) mitigates this challenge by selecting data points for annotation strategically. However existing patch-based AL methods often overlook boundary pixels' critical information essential for accurate segmentation. We present OREAL a novel patch-based AL method designed for multi-class semantic segmentation. OREAL enhances boundary detection by employing maximum aggregation of pixel-wise uncertainty scores. Additionally we introduce one-vs-rest entropy a novel uncertainty score function that computes class-wise uncertainties while achieving implicit class balancing during dataset creation. Comprehensive experiments across diverse datasets and model architectures validate our hypothesis.