WACV 2025 论文列表

The study of social interactions and collective behaviors through multi-agent video analysis is crucial in biology. While self-supervised keypoint discovery has emerged as a promising solution to reduce the need for manual keypoint annotations existing methods often struggle with videos containing multiple interacting agents especially those of the same species and color. To address this we introduce B-KinD-multi a novel approach that leverages pre-trained video segmentation models to guide keypoint discovery in multi-agent scenarios. This eliminates the need for time-consuming manual annotations on new experimental settings and organisms. Extensive evaluations demonstrate improved keypoint regression and downstream behavioral classification in videos of flies mice and rats. Furthermore our method generalizes well to other species including ants bees and humans highlighting its potential for broad applications in automated keypoint annotation for multi-agent behavior analysis. Code available under: B-KinD-Multi

Tuning parameters of a non-differentiable or black-box compute is challenging. Existing methods rely mostly on random sampling or grid sampling from the parameter space. Further with all the current methods it is not possible to supply any input specific parameters to the black-box. To the best of our knowledge for the first time we are able to learn input-specific parameters for a black box in this work. As a test application we choose a popular image denoising method BM3D as our black-box compute. Then we use a differentiable surrogate model (a neural network) to approximate the black-box behaviour. Next another neural network is used in an end-to-end fashion to learn input instance-specific parameters for the black-box. Motivated by prior advances in surrogate-based optimization we applied our method to the Smartphone Image Denoising Dataset (SIDD) and the Color Berkeley Segmentation Dataset (CBSD68) for image denoising. The results are compelling demonstrating a significant increase in PSNR and a notable improvement in SSIM nearing 0.93. Experimental results underscore the effectiveness of our approach in achieving substantial improvements in both model performance and optimization efficiency. For code and implementation details please refer to our GitHub repository: https://github.com/arnisha-k/instance-specific-param.

Multispectral (MS) snapshot cameras equipped with a MS filter array (MSFA) capture multiple spectral bands in a single shot resulting in a raw mosaic image where each pixel holds only one channel value. The fully-defined MS image is estimated from the raw one through demosaicing which inevitably introduces spatio-spectral artifacts. Moreover training on fully-defined MS images can be computationally intensive particularly with deep neural networks (DNNs) and may result in features lacking discrimination power due to suboptimal learning of spatio-spectral interactions. Furthermore outdoor MS image acquisition occurs under varying lighting conditions leading to illumination-dependent features. This paper presents an original approach to learn discriminant and illumination-robust features directly from raw images. It involves: raw spectral constancy to mitigate the impact of illumination MSFA-preserving transformations suited for raw image augmentation to train DNNs on diverse raw textures and raw-mixing to capture discriminant spatio-spectral interactions in raw images. Experiments on MS image classification show that our approach outperforms both handcrafted and recent deep learning-based methods while also requiring significantly less computational effort. The source code is available at https://github.com/AnisAmziane/RawTexture.

Human experts demonstrate proficiency not only in disentangling anatomical structures from disease conditions but also in intertwining anatomical and disease information to accurately diagnose a variety of disorders. However deep learning models despite their prowess in acquiring intricate representation often struggle to learn representation where distinct semantic aspects of the data (both anatomy and pathology) are entangled particularly in medical images which present a rich array of anatomical structures and potential pathological conditions. We envision that a deep model when trained to comprehend medical images akin to human perception would offer powerful representation with higher generalizability robustness and interpretability. To realize this vision we have developed LeADER a framework for learning anatomy-disease entangled representation from medical images. As a proof of concept we have trained LeADER on 1M chest radiographs gathered from 10 public datasets. Experimental results across 11 medical tasks compared to 8 baselines in zero-shot linear probing limited data regimes and full fine-tuning settings demonstrate LeADER's superior performance over the Google CXR Foundation Model large-scale medical models and fully/self-supervised baselines across diverse downstream tasks. This enhanced performance is attributed to the significance of entangling anatomy-specific and disease-specific representations via our framework which enables the simultaneous acquisition of both anatomical and disease knowledge yet overlooked in existing supervised/self-supervised learning methods. All code and models are available at GitHub.com/JLiangLab/LeADER.

Large-scale vision-language pre-trained (VLP) models (e.g. CLIP) are renowned for their versatility as they can be applied to diverse applications in a zero-shot setup. However when these models are used in specific domains their performance often falls short due to domain gaps or the under-representation of these domains in the training data. While fine-tuning VLP models on custom datasets with human-annotated labels can address this issue annotating even a small-scale dataset (e.g. 100k samples) can be an expensive endeavor often requiring expert annotators if the task is complex. To address these challenges we propose LatteCLIP an unsupervised method for fine-tuning CLIP models on classification with known class names in custom domains without relying on human annotations. Our method leverages Large Multimodal Models (LMMs) to generate expressive textual descriptions for both individual images and groups of images. These provide additional contextual information to guide the fine-tuning process in the custom domains. Since LMM-generated descriptions are prone to hallucination or missing details we introduce a novel strategy to distill only the useful information and stabilize the training. Specifically we learn rich per-class prototype representations from noisy generated texts and dual pseudo-labels. Our experiments on 10 domain-specific datasets show that LatteCLIP outperforms pre-trained zero-shot methods by an average improvement of +4.74 points in top-1 accuracy and other state-of-the-art unsupervised methods by +3.45 points.

Recent advancements in cooperative perception have showcased substantial improvements compared to single-agent perception. Nonetheless the inherent latency present in such systems can dramatically impair their effectiveness. In this paper we propose a Latency Robust Cooperative Perception framework named LRCP to compensate for the effect of temporal asynchrony. The intuition of LRCP is to directly fuse asynchronous bird's-eye view (BEV) features instead of estimating aligned features. To achieve this we first propose a novel flow prediction module that uses cached past BEV features to predict the flow with a non-discrete time delay at the BEV feature level. Then the predicted flow is employed to guide the spatial sampling location of interests. Our approach substantially enhances the robustness of temporal asynchronous cooperative perception. Specifically we achieved robust performance across a range of latencies up to 500 ms with a performance degradation of only 1 percent point for AP@0.5 metric and 4 percent points for AP@0.7 metric at 500 ms on two public datasets (V2X-Sim and Dair-V2X). Code to reproduce our results is available at https://github. com/JesseWong333/LRCP.

The increasing relevance of panoptic segmentation is tied to the advancements in autonomous driving and AR/VR applications. However the deployment of such models has been limited due to the expensive nature of dense data annotation giving rise to unsupervised domain adaptation (UDA). A key challenge in panoptic UDA is reducing the domain gap between a labeled source and an unlabeled target domain while harmonizing the subtasks of semantic and instance segmentation to limit catastrophic interference. While considerable progress has been achieved existing approaches mainly focus on the adaptation of semantic segmentation. In this work we focus on incorporating instance-level adaptation via a novel instance-aware cross-domain mixing strategy IMix. IMix significantly enhances the panoptic quality by improving instance segmentation performance. Specifically we propose inserting high-confidence predicted instances from the target domain onto source images retaining the exhaustiveness of the resulting pseudo-labels while reducing the injected confirmation bias. Nevertheless such an enhancement comes at the cost of degraded semantic performance attributed to catastrophic forgetting. To mitigate this issue we regularize our semantic branch by employing CLIP-based domain alignment (CDA) exploiting the domain-robustness of natural language prompts. Finally we present an end-to-end model incorporating these two mechanisms called LIDAPS achieving state-of-the-art results on all popular panoptic UDA benchmarks. https://github.com/elhamAm/LIDAPS

Annotation errors are a challenge not only during training of machine learning models but also during their evaluation. Label variations and inaccuracies in datasets often manifest as contradictory examples that deviate from established labeling conventions. Such inconsistencies when significant prevent models from achieving optimal performance on metrics such as mean Average Precision (mAP). We introduce the notion of "label convergence" to describe the highest achievable performance under the constraint of contradictory test annotations essentially defining an upper bound on model accuracy. Recognizing that noise is an inherent characteristic of all data our study analyzes five real-world datasets including the LVIS dataset to investigate the phenomenon of label convergence. We approximate that label convergence is between 62.63-67.52 mAP@[0.5:0.95:0.05] for LVIS with 95% confidence attributing these bounds to the presence of real annotation errors. With current state-of-the-art (SOTA) models at the upper end of the label convergence interval for the well-studied LVIS dataset we conclude that model capacity is sufficient to solve current object detection problems. Therefore future efforts should focus on three key aspects: (1) updating the problem specification and adjusting evaluation practices to account for unavoidable label noise (2) creating cleaner data especially test data and (3) including multi-annotated data to investigate annotation variation and make these issues visible from the outset.

Source-free domain adaptation (SFDA) utilizes a pre-trained source model with unlabeled target data. Self-supervised SFDA techniques generate pseudolabels from the pre-trained source model but these pseudolabels often contain noise due to domain discrepancies between the source and target domains. Traditional self-supervised SFDA techniques rely on deterministic model predictions using the softmax function leading to unreliable pseudolabels. In this work we propose to introduce predictive uncertainty and softmax calibration for pseudolabel refinement using evidential deep learning. The Dirichlet prior is placed over the output of the target network to capture uncertainty using evidence with a single forward pass. Furthermore softmax calibration solves the translation invariance problem to assist in learning with noisy labels. We incorporate a combination of evidential deep learning loss and information maximization loss with calibrated softmax in both prior and non-prior target knowledge SFDA settings. Extensive experimental analysis shows that our method outperforms other state-of-the-art methods on benchmark datasets. The code is available at https://visdomlab.github.io/EKS/.

Traditional dataset distillation primarily focuses on image representation while often overlooking the important role of labels. In this study we introduce Label-Augmented Dataset Distillation (LADD) a new dataset distillation framework enhancing dataset distillation with label augmentations. LADD sub-samples each synthetic image generating additional dense labels to capture rich semantics. These dense labels require only a 2.5% increase in storage (ImageNet subsets) with significant performance benefits providing strong learning signals. Our label-generation strategy can complement existing dataset distillation methods and significantly enhance their training efficiency and performance. Experimental results demonstrate that LADD outperforms existing methods in terms of computational overhead and accuracy. With three high-performance dataset distillation algorithms LADD achieves remarkable gains by an average of 14.9% in accuracy. Furthermore the effectiveness of our method is proven across various datasets distillation hyperparameters and algorithms. Finally our method improves the cross-architecture robustness of the distilled dataset which is important in the application scenario.

We focus on the problem of Gallbladder Cancer (GBC) detection from Ultrasound (US) images. The problem presents unique challenges to modern Deep Neural Network (DNN) techniques due to low image quality arising from noise textures and viewpoint variations. Tackling such challenges would necessitate precise localization performance by the DNN to identify the discerning features for the downstream malignancy prediction. While several techniques have been proposed in the recent years for the problem all of these methods employ complex custom architectures. Inspired by the success of foundational models for natural image tasks along with the use of adapters to fine-tune such models for the custom tasks we investigate the merit of one such design ViT-Adapter for the GBC detection problem. We observe that ViT-Adapter relies predominantly on a primitive CNN-based spatial prior module to inject the localization information via cross-attention which is inefficient for our problem due to the small pathology sizes and variability in their appearances due to non-regular structure of the malignancy. In response we propose LQ-Adapter a modified Adapter design for ViT which improves localization information by leveraging learnable content queries over the basic spatial prior module. Our method surpasses existing approaches enhancing the mean IoU (mIoU) scores by 5.4% 5.8% and 2.7% over ViT-Adapters DINO and FocalNet-DINO respectively on the US image-based GBC detection dataset and establishing a new state-of-the-art (SOTA). Additionally we validate the applicability and effectiveness of LQ-Adapter on the Kvasir-Seg dataset for polyp detection from colonoscopy images. Superior performance of our design on this problem as well showcases its capability to handle diverse medical imaging tasks across different datasets. Source code and trained models are publicly released.

Reinforcement learning (RL) based autonomous driving has emerged as a promising alternative to data-driven imitation learning approaches. However crafting effective reward functions for RL poses challenges due to the complexity of defining and quantifying good driving behaviors across diverse scenarios. Recently large pretrained models have gained significant attention as zero-shot reward models for tasks specified with desired linguistic goals. However the desired linguistic goals for autonomous driving such as "drive safely" are ambiguous and incomprehensible by pretrained models. On the other hand undesired linguistic goals like "collision" are more concrete and tractable. In this work we introduce LORD a novel large models based opposite reward design through undesired linguistic goals to enable the efficient use of large pretrained models as zero-shot reward models. Through extensive experiments our proposed framework shows its efficiency in leveraging the power of large pretrained models for achieving safe and enhanced autonomous driving. Moreover the proposed approach shows improved generalization capabilities as it outperforms counterpart methods across diverse and challenging driving scenarios.

Scene Graph Generation (SGG) converts visual scenes into structured graph representations providing deeper scene understanding for complex vision tasks. However existing SGG models often overlook essential spatial relationships and struggle with generalization in open-vocabulary contexts. To address these limitations we propose LLaVASpaceSGG a multimodal large language model (MLLM) designed for open-vocabulary SGG with enhanced spatial relation modeling. To train it we collect the SGG instruction-tuning dataset named SpaceSGG. This dataset is constructed by combining publicly available datasets and synthesizing data using open-source models within our data construction pipeline. It combines object locations object relations and depth information resulting in three data formats: spatial SGG description question-answering and conversation. To enhance the transfer of MLLMs' inherent capabilities to the SGG task we introduce a two-stage training paradigm. Experiments show that LLaVASpaceSGG outperforms other open-vocabulary SGG methods boosting recall by 8.6% and mean recall by 28.4% compared to the baseline. Our codebase dataset and trained models are publicly accessible on GitHub at the following URL: https://github.com/Endlinc/LLaVA-SpaceSGG.

Training deep neural networks (DNNs) using traditional backpropagation (BP) presents challenges in terms of computational complexity and energy consumption particularly for on-device learning where computational resources are limited. Various alternatives to BP including random feedback alignment forward-forward and local classifiers have been explored to address these challenges. These methods have their advantages but they can encounter difficulties when dealing with intricate visual tasks or demand considerable computational resources. In this paper we propose a novel Local Learning rule inspired by neural activity Synchronization phenomena (LLS) observed in the brain. LLS utilizes fixed periodic basis vectors to synchronize neuron activity within each layer enabling efficient training without the need for additional trainable parameters. We demonstrate the effectiveness of LLS and its variations LLS-M and LLS-MxM on multiple image classification datasets achieving accuracy comparable to BP with reduced computational complexity and minimal additional parameters. Specifically LLS achieves comparable performance with up to 300x fewer multiply-accumulate (MAC) operations and half the memory requirements of BP. Furthermore the performance of LLS on the Visual Wake Word (VWW) dataset highlights its suitability for on-device learning tasks making it a promising candidate for edge hardware implementations.

The employment of large language models (LLMs) for task planning and reasoning has emerged as a focal point of interest within the robotics research community. However directly applying LLMs even with large token-sized prompts does not achieve the task planning performance required for an industrial-grade domain-specific use-case (DSU). This work aims to overcome the obstacles of a robotic task planner for DSUs by introducing a novel planning framework LLM-RSPF (Large Language Model-based Robotic System Planning Framework). Central to the LLM-RSPF is a novel robotic system ontology that organizes the components of the robotic system in a coherent and a systematic manner. The ontology empowers the LLM-RSPF to efficiently capture a contextual representation of the DSU using the LLMs. Subsequently the research introduces a LLM-tuning regimen referred as chain of hierarchical thought (CoHT) specifically crafted to complement the proposed system ontology. Integrating these two components the LLM-RSPF aims to enhance the accuracy robustness and throughput of a robotic system in a cost-effective manner. In addition the research presents an empirical methodology to generate the LLM-tuning dataset size for a guaranteed performance. The LLM-RSPF is validated on a retail order-fulfillment use-case thereby illustrating the efficacy of the framework. Through rigorous evaluation the LLM-RSPF demonstrates exceptional performance on the generated dataset effectively meeting the DSU objectives.

Recent advancements in vision-language models (VLMs) have found important applications in medical imaging particularly in digital pathology. VLMs demand large-scale datasets of image-caption pairs which is often hard to obtain in medical domains. State-of-the-art VLMs in digital pathology have been pre-trained on datasets that are significantly smaller than their computer vision counterparts. Furthermore the caption of a pathology slide often refers to a small sub-set of features in the image--an important point that is ignored in existing VLM pre-training schemes. Another important issue that is under-appericated is that the performance of state-of-the-art VLMs in zero-shot classification tasks can be sensitive to the choice of the prompts. In this paper we first employ language rewrites using a large language model (LLM) to enrich a public pathology image-caption dataset and make it publicly available. Our extensive experiments demonstrate that by training with language rewrites we can boost the performance of a state-of-the-art digital pathology VLM on downstream tasks such as zero-shot classification and text-to-image and image-to-text retrieval. We further leverage LLMs to demonstrate the sensitivity of zero-shot classification results to the choice of prompts and propose a scalable approach to characterize this when comparing models. Finally we present a novel context modulation layer that adjusts the image embeddings for better aligning with the paired text and use context-specific language rewrites for training this layer. In our results we show that the proposed context modulation framework can further yield substantial performance gains.

Photometric stereo is a powerful technique for estimating per-pixel surface normals from images under varied illumination. Although several methods address photometric stereo with different image (or light) counts ranging from one to two to a hundred very few focus on learning optimal lighting configuration. Finding an optimal configuration is challenging due to the large number of possible lighting directions. Moreover exhaustive sampling of all possibilities is impractical due to time and resource constraints. Photometric stereo methods have demonstrated promising performance on existing datasets which feature limited light directions sparsely sampled from the light space. Therefore can we optimally utilize these datasets for illumination planning? In this work we introduce LIPIDS - Learning-based Illumination Planning In Discretized light Space to achieve minimal and optimal lighting configurations for photometric stereo under arbitrary light distribution. We propose a Light Sampling Network (LSNet) that optimizes the lighting direction for a fixed number of lights by minimizing the normal loss through a normal regression network. The learned light configurations can directly estimate surface normals during inference even using an off-the-shelf photometric stereo method. Extensive qualitative and quantitative analysis on synthetic and real-world datasets show that photometric stereo under learned lighting configurations through LIPIDS either surpasses or is nearly comparable to existing illumination planning methods across different photometric stereo backbones.

Diffusion models (DMs) have gained prominence due to their ability to generate high-quality varied images with recent advancements in text-to-image generation. The research focus is now shifting towards the controllability of DMs. A significant challenge within this domain is localized editing where specific areas of an image are modified without affecting the rest of the content. This paper introduces LIME for localized image editing in diffusion models. LIME does not require user-specified regions of interest (RoI) or additional text input but rather employs features from pre-trained methods and a straightforward clustering method to obtain precise editing mask. Then by leveraging cross-attention maps it refines these segments for finding regions to obtain localized edits. Finally we propose a novel cross-attention regularization technique that penalizes unrelated cross-attention scores in the RoI during the denoising steps ensuring localized edits. Our approach without re-training fine-tuning and additional user inputs consistently improves the performance of existing methods in various editing benchmarks. The project page can be found at https://enisimsar.github.io/LIME/.

We introduce Knockoff Branch: adding few neurons as a knockoff container for learning stolen features. Model stealing attacks extract the functionality from the victim model by querying APIs. Prior work substantially enhanced transferability and improved query efficiency between the adversary model and the victim model. However there is still a limited understanding of the knockoff itself. For knockoff the model is either compared to the same type but with different structures or different types and capacities. For this reason we propose a framework to analyze the knockoff quality for a single model specifically reinvestigating transformer-based extraction. We observed that 1) when the adversary can access the public pretrained model full fine-tuning is not necessary. This allows a knockoff to require only about 0.5% of trainable parameters and 20 epochs. 2) Although querying by out-of-distribution datasets leads to a sub-optimal knockoff this issue can be mitigated by scaling branch features even without using complicated sampling strategies. Our proposed method is lightweight and achieves high accuracy at most similar to white-box knowledge distillation (higher performance than the victim model). https://github.com/onlyin-hung/knockoff-branch.