paperlist.ai

AI conference papers

AI conference paper library

Search papers from 30+ top conferences, including NeurIPS, ICLR, ICML, CVPR, ACL and AAAI. Filter by keyword, then open any paper for its abstract, authors and links.

arXiv

Face-D(^2)CL: Multi-Domain Synergistic Representation with Dual Continual Learning for Facial DeepFake Detection

Yushuo Zhang, Yu Cheng, Yongkang Hu, Jiuan Zhou, Jiawei Chen, Zhaoxia Yin

Facial forgery techniques are advancing rapidly, posing severe threats to public trust and information security while imposing higher demands on the continual adaptation of DeepFake detection models. Although continual learning enables models to adapt to emerging forgery methods, existing approaches still face two key bottlenecks. On the one hand, they lack sufficient feature representation capacity to handle increasingly diverse and complex forgery traces. On the other hand, continual adaptation to new forgery distributions leads to severe catastrophic forgetting of prior knowledge, which substantially degrades detection performance. To address these issues, we propose Face-D(^2)CL, a framework for facial DeepFake detection. It leverages multi-domain synergistic representation to fuse spatial and frequency-domain features, enabling comprehensive capture of diverse forgery traces. Additionally, it employs a dual continual learning mechanism that combines Real/Fake-aware Elastic Weight Consolidation (RF-EWC) and Domain-wise Orthogonal Gradient Constraint (D-OGC). RF-EWC distinguishes the parameter importance for real versus fake samples, while D-OGC ensures that updates to task-specific expert modules do not interfere with previously learned knowledge. This synergy allows the model to achieve a dynamic balance between robust anti-forgetting capabilities and agile adaptability to emerging facial forgery paradigms, all without relying on historical data replay. Extensive experiments demonstrate that our method surpasses current state-of-the-art (SOTA) approaches in both stability and plasticity, achieving a 60.7% relative reduction in the average detection error rate. On unseen forgery domains, it further improves the average detection AUC by 7.9% compared to the current SOTA method.

arXiv

EmoScene: A Dual-space Dataset for Controllable Affective Image Generation

Li He, Longtai Zhang, Wenqiang Zhang, Yan Wang, Lizhe Qi

Text-to-image diffusion models achieve high visual fidelity, yet fine-grained affective control remains difficult because textual emotion cues often fail to specify the visual perceptual factors underlying affective expression. Existing visual-affect datasets are likewise often limited to discrete labels, specific domains, or limited supervision of perceptual attributes. We introduce EmoScene, a large-scale dual-space dataset for controllable affective image generation, containing 1.2M images across more than 300 scene categories. Its affective space jointly represents discrete emotions and continuous valence--arousal--dominance (VAD), its perceptual space records measurable appearance attributes, and contextual descriptions ground both in scene semantics. EmoScene combines multi-model annotation with human-in-the-loop quality control. A random audit of 30,519 images yields 91.22\% agreement on discrete emotion labels, while an independent multi-rater evaluation yields Fleiss' $κ=0.85$. After controlling for source and scene composition, affective dimensions and perceptual attributes exhibit stable associations across data sources, reflecting statistical tendencies rather than deterministic visual rules. To demonstrate the dataset's utility, we further develop AffectCtrl, which learns residuals in the conditioning space of frozen diffusion models to support categorical emotion generation and continuous control over VAD, brightness, and saturation. AffectCtrl achieves 85.75\% categorical emotion accuracy, outperforms EmotiCrafter in valence and arousal control under a shared evaluation protocol, and obtains Pearson correlations of 0.673--0.765 across all five continuous axes. These results demonstrate that EmoScene provides a scalable data foundation for analyzing and controlling affective expression in visual generation.

arXiv

XSPA: Crafting Imperceptible X-Shaped Sparse Adversarial Perturbations for Transferable Attacks on VLMs

Chengyin Hu, Jiaju Han, Xuemeng Sun, Qike Zhang, luwei yang, Lehan Sun, Jiahuan Long, Yiwei Wei, Jiujiang Guo

Vision-language models (VLMs) share visual-textual representations across zero-shot classification, image captioning, and visual question answering (VQA), creating a pathway through which subtle perturbations may cause failures across tasks. We introduce X-shaped Sparse Pixel Attack (XSPA), a structured attack that restricts perturbations to two intersecting diagonal lines. Within this fixed support, XSPA jointly optimizes a classification objective, cross-task semantic guidance, perturbation magnitude, and linewise smoothness. Under the default setting, it changes about 1.04\% of image pixels. On COCO, XSPA reduces zero-shot accuracy by 52.33 points on OpenAI CLIP ViT-L/14 and 67.00 points on OpenCLIP ViT-B/16; GPT-4-based caption consistency and VQA correctness decrease by up to 58.60 and 44.25 points, respectively. Matched-budget experiments show that an adaptive saliency Top-k support achieves higher attack success than the fixed X-shaped support, while XSPA provides better SSIM and LPIPS but worse PSNR and MAE. Same-image analysis finds no three-task targeted success, indicating that strong surrogate classification effects do not reliably translate into targeted output-space failures. These results position XSPA as a controlled stress test for studying how sparse fixed geometry affects VLM robustness, rather than as a universally superior attack.

arXiv

Gimbal360: Canonicalizing Planar Diffusion for Spherical Panorama Completion

Yuqin Lu, Haofeng Liu, Yang Zhou, Yihua Dai, Guiqing Li, Shengfeng He, Jun Liang

Diffusion models provide powerful priors for 2D image completion, but these priors are learned on bounded planar images and do not transfer directly to $360^\circ$ panoramas. Perspective observations and spherical panoramas differ in both projective geometry and topology: viewpoint-dependent distortion complicates spatial correspondence, while Equirectangular Projection (ERP) panoramas exhibit intrinsic $S^1$ periodicity that standard Euclidean architectures do not preserve. We present Gimbal360, a unified framework that adapts planar diffusion priors to spherical panoramic completion by standardizing these geometric and topological structures. Our Canonical Viewing Space expresses projective distortion as a fixed function of latitude, providing a consistent interface between perspective inputs and spherical panoramas. To map unposed in-the-wild images into this space, Differentiable Projective Canonicalization projects a dense correspondence field onto a 3-DoF rigid projection manifold without requiring camera parameters at inference. We further introduce Topologically Equivariant Generation, which enforces latent shift equivariance to preserve continuity across the periodic ERP boundary. Together, these designs allow diffusion to operate in a representation whose geometry and topology are explicitly aligned with the spherical domain. We also introduce Horizon360, a curated large-scale dataset of gravity-aligned panoramic environments. Extensive experiments show that Gimbal360 achieves state-of-the-art visual fidelity and seam continuity in $360^\circ$ scene completion.

arXiv

ABRA: Teleporting Fine-Tuned Knowledge Across Domains for Open-Vocabulary Object Detection

Mattia Bernardi, Chiara Cappellino, Matteo Mosconi, Enver Sangineto, Angelo Porrello, Simone Calderara

Although recent Open-Vocabulary Object Detection architectures, such as Grounding DINO, demonstrate strong zero-shot capabilities, their performance degrades significantly under domain shifts. Moreover, many domains of practical interest, such as nighttime or foggy scenes, lack large annotated datasets, preventing direct fine-tuning. In this paper, we introduce Aligned Basis Relocation for Adaptation(ABRA), a method that transfers class-specific detection knowledge from a labeled source domain to a target domain where no training images containing these classes are accessible. ABRA formulates this adaptation as a geometric transport problem in the weight space of a pretrained detector, aligning source and target domain experts to transport class-specific knowledge. Extensive experiments across challenging domain shifts demonstrate that ABRA successfully teleports class-level specialization under multiple adverse conditions. Our code will be made public upon acceptance.

arXiv

EgoIntent: A Pre-Outcome Micro-Step Benchmark for Understanding What, Why, and Next

Ye Pan, Chi Kit Wong, Yuanhuiyi Lyu, Hanqian Li, Chenfei Liao, Jiahao Huo, Lutao Jiang, Zixin Zhang, Jiacheng Chen, Yuqian Fu, Xu Zheng

Egocentric video provides a natural modality for studying human behavior, but conventional visual understanding captures mainly observable scenes, objects, and actions rather than the latent goals that organize them. Existing intent benchmarks typically focus on coarse event-level goals and overlook how intent evolves across procedural steps. We introduce EgoIntent, a step-level intent-understanding benchmark comprising 3,014 pre-outcome micro-steps from 32 egocentric videos across 15 indoor and outdoor daily-life scenarios. Each step is manually annotated along three complementary dimensions: Local Intent (What), the immediate goal; Procedural Intent (Why), the role of the step in the broader procedure; and Next-Plan (Next), the action most likely to follow. Multiple rounds of human review refine temporal boundaries and annotation quality. We evaluate 15 multimodal large language models using reference-based scores and complementary reference-free diagnostics. Controlled studies on four representative models show that only one model gains significantly from correct temporal order, while a single boundary frame outperforms the full ordered clip for three models. Step-only input performs best for all four models, and adding 15 seconds of history significantly degrades three. Revealing the current outcome improves Local Intent by 7.81 points, while revealing the following step improves Next-Plan by 13.17 points. These findings indicate that current models can achieve strong intent-prediction scores through static boundary cues without robustly exploiting temporal order or procedural history.

arXiv

OSMDA: OpenStreetMap-based Domain Adaptation for Remote Sensing VLMs

Stefan Maria Ailuro, Mario Markov, Mohammad Mahdi, Delyan Boychev, Luc Van Gool, Danda Pani Paudel

Vision-Language Models (VLMs) adapted to remote sensing rely heavily on domain-specific image-text supervision, yet high-quality annotations for satellite and aerial imagery remain scarce and expensive to produce. Prevailing pseudo-labeling pipelines address this gap by distilling knowledge from large frontier models, but this dependence on large teachers is costly, limits scalability, and caps achievable performance at the ceiling of the teacher. We propose OSMDA: a self-contained domain adaptation framework that eliminates this dependency. Our key insight is that a capable base VLM can serve as its own annotation engine: by pairing aerial images with rendered OpenStreetMap (OSM) tiles, we leverage optical character recognition and chart comprehension capabilities of the model to generate captions enriched by OSM's vast auxiliary metadata. The model is then fine-tuned on the resulting corpus with satellite imagery alone, yielding OSMDA-VLM, a domain-adapted VLM that requires no manual labeling and no stronger external VLM teacher. We conduct exhaustive evaluations spanning six zero-shot and five in-distribution benchmarks across vision-language tasks, where OSMDA leads to substantial improvement. We further compare against nine competitive baselines, demonstrating that our method achieves superior overall performance, while being substantially cheaper to train than teacher-dependent alternatives. These results suggest that, given a strong foundation model, alignment with crowd-sourced geographic data is a practical and scalable path towards remote sensing domain adaptation. Dataset and model weights will be made publicly available upon acceptance.

arXiv

Less is More: Compact-Token Masked Feature Prediction for Skeleton Representation Learning

Jeonghyeok Do, Yun Chen, Geunhyuk Youk, Munchurl Kim

Current skeleton representation learning paradigms face distinct limitations: Contrastive Learning (CL) often overlooks fine-grained motion details, while Masked Auto-Encoders (MAE) rely on coordinate-level reconstruction. This reconstruction inherently demands dense token sequences and heavy decoders, wasting pre-training computation on discarded components and forcing downstream inference to process dense token grids. To resolve these bottlenecks, we propose SLiM (Skeleton Less is More), a compact-token framework that unifies masked feature prediction and contrastive learning via a shared encoder. By shifting the objective from raw coordinate reconstruction to decoder-free, teacher-guided feature prediction, SLiM breaks the reliance on dense tokenization and enables effective learning with a highly compact token grid. Crucially, to prevent trivial shortcut learning arising from strong inter-joint dependencies of human, we introduce Semantic Tube Masking together with Skeleton-Aware Augmentations to enforce deep skeletal-temporal reasoning and anatomical consistency. Extensive experiments across multiple downstream protocols demonstrate that SLiM achieves state-of-the-art performance while structurally reducing inference computation by 7.89$\times$ compared to dense-token MAE baselines.

arXiv

Geometric-Topological Perception and Motion Prior for Real-Time Satellite Video Object Tracking

Zixiao Wen, Guangyao Zhou, Jiawei Li, Xiantai Xiang, Zhen Yang, Yuxin Hu, Yuhan Liu

Satellite video object tracking (SVOT) remains fundamentally challenging due to texture scarcity, arbitrary rotation, aspect ratio changes, and severe occlusions. While recent state-of-the-art trackers excel in general scenarios, their reliance on rich appearance details or rigid spatial matching mechanisms leads to significant performance degradation in the satellite domain. To bridge this gap, we propose SiamGM, a real-time spatial-temporal unified tracking framework tailored for satellite videos. Instead of conventionally stacking modules, we synergize geometric-topological perception with temporal-kinematic prior, addressing the core limitations of SVOT. Spatially, we closely couple a Topological Attention Module (TAM) with a Geometry-Constrained Label Assignment (GCLA) method during the training phase, where a Local Graph Propagation (LGP) mechanism enforces neighborhood-consistent template-search correspondences. This joint paradigm establishes fine-grained structural correspondences and explicitly suppresses surrounding background noise. Temporally, rather than passively relying on visual cues, we propose an Online Motion Model Refinement (OMMR) strategy during the tracking phase. Gated by an online self-calibrated confidence indicator, OMMR adaptively fuse long-term stable and short-term instantaneous motion estimation for robust tracking. Evaluations on two challenging SatSOT and SV248S benchmarks confirm that SiamGM outperforms not only recent Siamese-based satellite trackers but also state-of-the-art Transformer trackers in both precision and success metrics. Notably, this highly unified architecture introduces only minor computational overhead, enabling real-time tracking at 130 frames per second (FPS).

arXiv

Move What Matters: Parameter-Efficient Domain Adaptation via Optimal Transport Flow for Collaborative Perception

Zesheng Jia, Jin Wang, Siao Liu, Lingzhi Li, Ziyao Huang, Yunjiang Xu, Jianping Wang

Efficient domain adaptation remains a fundamental challenge for deploying multi-agent systems across diverse environments in Vehicle-to-Everything (V2X) collaborative perception. Despite the success of Parameter-Efficient Fine-Tuning (PEFT) in natural language processing and conventional vision tasks, directly applying PEFT to collaborative perception recovers only a limited portion of the performance lost to domain shift. In this work, we identify two complementary bottlenecks that limit this recovery: (i) inter-frame redundancy within a collaborative sequence, which makes the effectiveness of scarce labels sensitive to frame selection, and (ii) foreground cues that become less linearly decodable in deeper-stage representations of a frozen backbone. To address these issues, we propose FlowAdapt, a parameter-efficient framework grounded in optimal transport. Wasserstein Greedy Sampling casts frame selection as minimizing the $W_{\infty}$ distance from the sequence to the retained subset, which equals its covering radius, so a farthest first traversal returns a subset provably within twice the optimum. Progressive Knowledge Transfer then routes compressed early-stage features into the deeper stages, gating each stage-local correction by that early evidence. Extensive experiments across target domains and fusion architectures show that FlowAdapt achieves state-of-the-art adaptation performance with about 1\% trainable parameters, and maintains this lead under localization noise.

arXiv

Chart Specification: Structural Representations for Incentivizing VLM Reasoning in Chart-to-Code Generation

Minggui He, Mingchen Dai, Jian Zhang, Yilun Liu, Shimin Tao, Pufan Zeng, Osamu Yoshie, Yuya Ieiri

Vision-Language Models (VLMs) have shown promise in generating plotting code from chart images, yet achieving structural fidelity remains challenging. Existing approaches largely rely on supervised fine-tuning, encouraging surface-level token imitation rather than faithful modeling of underlying chart structure, which often leads to hallucinated or semantically inconsistent outputs. We propose Chart Specification, a structured intermediate representation that shifts training from text imitation to semantically grounded supervision. Chart Specification filters syntactic noise to construct a structurally balanced training set and supports a Spec-Align Reward that provides fine-grained, verifiable feedback on structural correctness, enabling reinforcement learning to enforce consistent plotting logic. Experiments on three public benchmarks show that our method consistently outperforms prior approaches. With only 3K training samples, we achieve strong data efficiency, surpassing leading baselines by up to 61.7% on complex benchmarks, and scaling to 4K samples establishes new state-of-the-art results across all evaluated metrics. Overall, our results demonstrate that precise structural supervision offers an efficient pathway to high-fidelity chart-to-code generation. Code and dataset are available at: https://github.com/Mighten/chart-specification-paper

arXiv

SVRepair: Structured Visual Reasoning for Automated Program Repair

Jincheng Wang, Liwei Luo, Xiaoxuan Tang, Jingxuan Xu, Sheng Zhou, Dajun Chen, Wei Jiang, Yong Li

Large language models (LLMs) have recently been applied to Automated Program Repair (APR), yet most existing approaches remain unimodal and fail to use diagnostic signals contained in visual artifacts such as screenshots and control-flow graphs. In practice, many bug reports convey critical information visually (e.g., layout breakage or missing widgets), but directly using such dense visual inputs often causes context loss and noise, making it difficult for MLLMs to ground visual observations into precise fault localization and executable patches. To bridge this semantic gap, we propose \textbf{SVRepair}, a multimodal APR framework with Structured Visual Representation (SVR). SVRepair first fine-tunes a vision-language model, SVR, to uniformly transform heterogeneous visual artifacts into a \emph{semantic scene graph} that captures GUI elements and their structural relations (e.g., hierarchy), providing normalized, code-relevant context for downstream repair. Building on the graph, SVRepair drives a coding agent to localize faults and synthesize patches, and further introduces an iterative visual-artifact segmentation strategy that progressively narrows the input to bug-centered regions to suppress irrelevant context and reduce hallucinations. Across primary repository-level APR benchmarks, SVRepair resolves \textbf{186/517} SWE-Bench M instances (\textbf{35.98\%} over all instances; \textbf{36.47\%} over submitted runs) and \textbf{4/19} visual OmniGIRL instances (\textbf{21.05\%}). On supplementary structured multimodal code reasoning benchmarks, SVRepair reaches \textbf{38.02\%} on MMCode and \textbf{95.73\%} on CodeVision. Code is available at https://github.com/codefuse-ai/CodeFuse-SVR.

arXiv

Breaking Self-Attention Failure: Rethinking Query Initialization for Infrared Small Target Detection

Yuteng Liu, Duanni Meng, Yimian Dai, Maoxun Yuan, Xingxing Wei, Bo Li

Infrared small target detection (IRSTD) faces significant challenges due to low signal-to-noise ratios, extremely small target sizes, and complex cluttered backgrounds. Although DETR-based detectors benefit from global context modeling, their query initialization can become unreliable in IRSTD because only a few encoder tokens correspond to targets, while the majority describe the background. We revisit this phenomenon and reveal that the target-relevant embeddings of IRST are inevitably overwhelmed by dominant background features due to the self-attention mechanism, leading to unreliable query initialization and inaccurate target localization. To address this issue, we propose SEF-DETR, a novel framework that refines query initialization through Patch-wise Spectral Screening (PSS), Frequency-Routed Examination (FRE), and Reliability-Consistency-aware Fusion (RCF). PSS encodes the radial and directional energy distributions of local Fourier spectra and aggregates overlapping patch predictions into a target-relevant density map. Guided by this map, FRE performs sparse deformable re-examination after each encoder layer, while retaining content-driven sampling to distinguish true targets from frequency-domain false alarms. RCF further re-ranks candidate queries according to spatial-frequency consistency and frequency reliability. Extensive experiments on three public IRSTD datasets demonstrate that SEF-DETR achieves superior detection performance over state-of-the-art methods with low computational overhead, providing a robust and efficient DETR-based solution for IRSTD.

arXiv

Accuracy Does Not Guarantee Human-Likeness: Cross-Domain Human-Centered Benchmark in Monocular Depth Estimation

Yuki Kubota, Taiki Fukiage

Deep neural networks (DNNs) are increasingly used as functional models of human vision, yet standard monocular depth estimation (MDE) benchmarks largely evaluate physical accuracy rather than behavioral alignment with humans. We introduce a cross-domain behavioral benchmark that aligns newly collected human absolute-distance judgments, physical ground truth, and model predictions at 16 locations in each of 652 KITTI and 654 NYU Depth V2 scenes. Human-model similarity was measured image-wise by partial correlation between scale-and-shift-aligned estimates while controlling for physical depth, and reference results were obtained for 69 diverse MDE models. Human error patterns were highly reliable, and most models showed positive but substantially lower similarity to humans. Human similarity, however, did not increase monotonically with metric accuracy: within each dataset, high-accuracy models converged toward similar residual error structures that remained distinct from humans, while both human-similarity scores and error-space organization were only weakly preserved across datasets. Thus, physical accuracy and human-like behavior are complementary, domain-sensitive dimensions of MDE performance. Our benchmark will provide a reusable basis for assessing future models along both dimensions.

arXiv

Semantic-Guided Cross-Sensor Super Resolution of Remote Sensing Images: A Gated Dual Conditioning Flow Matching Model

Forouzan Fallah, Wenwen Li, Chia-Yu Hsu, Hyunho Lee, Anna Liljedahl, Yezhou Yang

High spatial resolution satellite imagery is critical for monitoring fine-scale Earth surface processes, but is often limited by cost and revisit time. This work studies cross-sensor super-resolution (SR) to reduce this gap by translating 10 m Sentinel-2 imagery into 2 m Maxar-like imagery in a data-scarce, domain-shifted setting, with a focus on rare geomorphic features such as retrogressive thaw slumps (RTS). We propose RareFlow, a semantic-guided generative AI framework for cross-sensor super-resolution based on a flow-matching formulation, designed to produce visually plausible and physically reliable high-resolution images. RareFlow uses dual conditioning to guide the generation process: (1) a gated ControlNet that preserves scene geometry from low-resolution (LR) input, and (2) text-based semantic guidance that injects contextual information when the target phenomenon is rare. To ensure high-fidelity outputs, we introduce a multifaceted loss function that anchors the output to the high-resolution (HR) ground truth by jointly enforcing frequency alignment, perceptual similarity, and color consistency. RareFlow's performance is systematically evaluated on a newly curated benchmark of multi-sensor satellite imagery for rare Earth feature detection, and its generalizability is demonstrated on two public remote sensing benchmarks, SEN2NAIP and BreizhSR. Human evaluation with domain experts is also conducted to further verify RareFlow's effectiveness in generating high-fidelity super-resolved images for scientific analysis.

arXiv

Restore Text First, Enhance Image Later: Two-Stage Scene Text Image Super-Resolution with Glyph Structure Guidance

Minxing Luo, Linlong Fan, Wang Qiushi, Ge Wu, Yiyan Luo, Yuhang Yu, Jinwei Chen, Yaxing Wang, Qingnan Fan, Jian Yang

Current image super-resolution methods show strong performance on natural images but distort text, creating a fundamental trade-off between image quality and textual readability. To address this, we introduce TIGER (Text-Image Guided supEr-Resolution), a novel two-stage framework that breaks this trade-off through a "text-first, image-later" paradigm. TIGER explicitly decouples glyph restoration from image enhancement: it first reconstructs precise text structures and uses them to guide full-image super-resolution. This ensures high fidelity and readability. To support comprehensive training and evaluation, we present the UZ-ST (UltraZoom-Scene Text) dataset, the first Chinese scene text dataset with extreme zoom. Extensive experiments show TIGER achieves state-of-the-art performance, enhancing readability and image quality.

arXiv

DeCLIP: Decoupled Prompting for Multi-Label Class-Incremental Learning with CLIP

Kaile Du, Zihan Ye, Junzhou Xie, Yixi Shen, Yuyang Li, Fuyuan Hu, Ling Shao, Guangcan Liu, Joost van de weijer, Fan Lyu

Multi-label class-incremental learning (MLCIL) continuously expands the label space while recognizing multiple co-occurring categories, making catastrophic forgetting a central challenge. Recent class-incremental learning methods have increasingly adopted CLIP as their backbone. However, we find that applying CLIP to MLCIL exhibits two critical issues: entanglement of class-specific cues in shared visual representations and high false-positive rates (FPR) under task-level partial labeling. We propose DeCLIP, a replay-free and parameter-efficient framework for CLIP-based MLCIL. DeCLIP uses Decoupled Prompting to learn class-specific positive and negative prompts in both visual and textual modalities, enabling class-conditioned vision-language matching and reducing representation entanglement. Only new-category prompts are optimized, previous prompts remain unchanged, preserving prior knowledge and mitigating catastrophic forgetting without replay. DeCLIP further incorporates Adaptive Similarity Tempering, an inference-time strategy that adapts similarity-tempering strength to the incremental configuration, suppressing false positives without specific tuning. Experiments on MS-COCO, PASCAL VOC, and the real-world NUS-WIDEseq benchmark demonstrate consistent improvements over prior methods with a few trainable parameters.

arXiv

CDG-MAE: Cross-view Masked Modeling using Diffusion Generated Views

Varun Belagali, Pierre Marza, Srikar Yellapragada, Zilinghan Li, Tarak Nath Nandi, Ravi K Madduri, Joel Saltz, Stergios Christodoulidis, Maria Vakalopoulou, Dimitris Samaras

Cross-view masked autoencoding has emerged as a powerful pretext task for learning dense correspondences, which are essential for applications such as video label propagation. The cross-view pretext task is modeled with a masked autoencoder, where a masked target view is reconstructed from an anchor view. However, acquiring effective training data remains a challenge - collecting diverse video datasets is costly, while simple image crops lack the necessary pose variations, underperforming video-based methods. This paper introduces CDG-MAE, a novel MAE-based self-supervised method that uses diverse synthetic views generated from static images via an image-conditioned diffusion model. We present a quantitative method to evaluate the local and global consistency of the generated views to choose the right diffusion model for cross-view self-supervised pretraining. These generated views exhibit substantial changes in pose and perspective, providing a rich training signal that overcomes the limitations of video and crop-based anchors. Furthermore, we enhance the standard single-anchor MAE setting to a multi-anchor masking strategy to increase the difficulty of the pretext task. CDG-MAE substantially narrows the gap to video-based MAE methods, while maintaining the data advantages of image-only MAEs.

arXiv

Deep Learning for Retinal Degeneration Assessment: A Comprehensive Analysis of the MARIO Challenge

Rachid Zeghlache, Ikram Brahim, Pierre-Henri Conze, Mathieu Lamard, Mohammed El Amine Lazouni, Zineb Aziza Elaouaber, Leila Ryma Lazouni, Christopher Nielsen, Ahmad O. Ahsan, Matthias Wilms, Nils D. Forkert, Lovre Antonio Budimir, Ivana Matovinović, Donik Vršnak, Sven Lončarić, Philippe Zhang, Weili Jiang, Yihao Li, Yiding Hao, Markus Frohmann, Patrick Binder, Marcel Huber, Taha Emre, Teresa Finisterra Araújo, Marzieh Oghbaie, Hrvoje Bogunović, Amerens A. Bekkers, Nina M. van Liebergen, Hugo J. Kuijf, Abdul Qayyum, Moona Mazher, Steven A. Niederer, Alberto J. Beltrán-Carrero, Juan J. Gómez-Valverde, Javier Torresano-Rodríquez, Álvaro Caballero-Sastre, María J. Ledesma Carbayo, Yosuke Yamagishi, Yi Ding, Robin Peretzke, Alexandra Ertl, Maximilian Fischer, Jessica Kächele, Sofiane Zehar, Karim Boukli Hacene, Thomas Monfort, Béatrice Cochener, Mostafa El Habib Daho, Anas-Alexis Benyoussef, Gwenolé Quellec

The MARIO challenge, held at MICCAI 2024, focused on advancing the automated detection and monitoring of age-related macular degeneration (AMD) through the analysis of optical coherence tomography (OCT) images. Designed to evaluate algorithmic performance in detecting neovascular activity changes within AMD, the challenge incorporated unique multi-modal datasets. The primary dataset, sourced from Brest, France, was used by participating teams to train and test their models. The final ranking was determined based on performance on this dataset. An auxiliary dataset from Algeria was used post-challenge to evaluate population and device shifts from submitted solutions. Two tasks were involved in the MARIO challenge. The first one was the classification of evolution between two consecutive 2D OCT B-scans. The second one was the prediction of future AMD evolution over three months for patients undergoing anti-vascular endothelial growth factor (VEGF) therapy. Thirty-five teams participated, with the top 12 finalists presenting their methods. This paper outlines the challenge's structure, tasks, data characteristics, and winning methodologies, setting a benchmark for AMD monitoring using OCT, infrared imaging, and clinical data (such as the number of visits, age, gender, etc.). The results of this challenge indicate that artificial intelligence (AI) performs as well as a physician in measuring AMD progression (Task 1) but is not yet able of predicting future evolution (Task 2).

arXiv

Mamba Policy: Towards Efficient 3D Diffusion Policy with Hybrid Selective State Models

Jiahang Cao, Qiang Zhang, Jingkai SUN, Jiaxu Wang, Hao Cheng, Yulin Li, Jun Ma, Kun Wu, Zhiyuan Xu, Yecheng Shao, Wen Zhao, Gang Han, Yijie Guo, Renjing Xu

Diffusion models have been widely employed in the field of 3D manipulation due to their efficient capability to learn distributions, allowing for precise prediction of action trajectories. However, diffusion models typically rely on large parameter UNet backbones as policy networks, which can be challenging to deploy on resource-constrained devices. Recently, the Mamba model has emerged as a promising solution for efficient modeling, offering low computational complexity and strong performance in sequence modeling. In this work, we propose the Mamba Policy, a lighter but stronger policy that reduces the parameter count by over 80% compared to the original policy network while achieving superior performance. Specifically, we introduce the XMamba Block, which effectively integrates input information with conditional features and leverages a combination of Mamba and Attention mechanisms for deep feature extraction. Extensive experiments demonstrate that the Mamba Policy excels on the Adroit, Dexart, and MetaWorld datasets, requiring significantly fewer computational resources. Additionally, we highlight the Mamba Policy's enhanced robustness in long-horizon scenarios compared to baseline methods and explore the performance of various Mamba variants within the Mamba Policy framework. Real-world experiments are also conducted to further validate its effectiveness. Our open-source project page can be found at https://sagecao1125.github.io/mamba_policy/.

Keep browsing

More papers from top AI conferences

30+ top conferences, by conference, year, and topic.