ECCV 2026 论文列表

European Conference on Computer Vision

录用类型

2,834篇论文

按官方录用论文列表统计,可能与会议公布的数字略有出入。

  • Poster
    2,80398.9%
  • 其他 / 未标注
    311.1%

Top 10 主题(点击筛选)

暂无主题数据。

全部 207 个研究方向的趋势

浏览全部论文

SafeSAE-VLA: Interpreting OpenVLA Progress Dynamics with Sparse Feature Analysis

Socrates Osorio, Joy Yang

展开摘要

Understanding whether modern vision-language-action poli-cies internally represent task progress is central to interpretable robotcontrol. This paper presents a progress-based SafeSAE-VLA analysis ofOpenVLA on 750 LIBERO episodes, replacing a degenerate binary targetwith a suite-normalized high-vs.-low split over relative geometric progressand testing layer-20 sparse-autoencoder (SAE) features (dsae = 16,384)with non-parametric differential analysis and false-discovery-rate (FDR)control. The recovered signal is strong and sparse. Of 1,881 active fea-tures, 1,117 are significant (FDR < 0.05), a linear probe reaches 0.918AUROC (area under the ROC curve), and top-20 features alone retain0.894 AUROC. Performance is stable across suites (up to 0.985 on goaland 0.947 on object), while motion-only controls are markedly weaker(0.572–0.711), indicating separability is not explained by gross move-ment magnitude. Dense raw-activation probes are equal or stronger thanSAE readouts (e.g., raw LR 0.975 vs. full-SAE 0.947 under a pooledanalysis with bootstrap confidence intervals). We therefore position thecontribution not as predictive dominance but as showing that nearly allof the progress signal survives in a compact set of named, inspectable,intervention-compatible directions. Layer sweeps, split-robustness checks,and a success-labeled audit (in which the progress-tuned top-20 correctlyscores an at-chance 0.471 while the broader SAE basis reaches 0.968)support this scoping, and direct on-OpenVLA feature-setting interven-tions produce specific, task-local behavioral shifts rather than generalclosed-loop repair.

Poster
RSICCLLM: A Multimodal Large Language Model for Remote Sensing Image Change Captioning

Yelin Wang, Zijia Song, Shuo Ye, Chuanguang Yang, Miaoyu Wang, Yong Xu, Zhulin An, Yongjun Xu, Zitong Yu

展开摘要

Remote Sensing Image Change Captioning (RSICC) aimsto describe changes between bi-temporal remote sensing images andholds significant research and application value. However, most exist-ing methods rely on conventional deep learning architectures, and thelimited model capacity constrains performance. Although large-modelpost-training techniques have achieved great success in general domains,their direct transfer to RSICC remains challenging due to data scarcityand the need for fine-grained change understanding. To address this, wepropose RSICCLLM, the first post-training framework for large vision-language models in RSICC. Specifically, we design a data genera-tion paradigm, release the instruction dataset RSICI, and establish atask-specific RSICC benchmark. We further introduce Difference-aware Supervised Fine-tuning to explicitly extract change repre-sentations and guide the model in perceiving and understanding tem-poral differences. In addition, we propose Dual-Negative PreferenceOptimization (DNPO), which employs two complementary negative-sample construction strategies to construct the preference dataset RSICPand further refine model performance. Extensive experiments validatethe superior capability of RSICCLLM, which achieves outstanding re-sults with only 7B parameters, surpassing models of substantially largerscales. The code and dataset will be made publicly available at https://github.com/keaill/RSICCLLM.

Poster
R2M: Real-Aware Residual Model Merging for Robust and Generalizable Deepfake Detection

Jinhee Park, Guisik Kim, Choongsang Cho, Junseok Kwon

展开摘要

Deepfake generators evolve rapidly, making exhaustive datacollection and repeated retraining impractical. Unlike generic multi-tasksettings, deepfake specialists share a common binary objective (Real vs.Fake) and mainly differ in generator-specific artifacts. However, naiveparameter arithmetic can induce unintended decision-boundary shifts,causing unstable ranking behavior and degraded AUC under domainshift. We propose R2 M, a training-free merging framework that decom-poses task vectors into a shared component and generator-specific resid-uals, linking parameter-space updates to logit-space behavior. Offlinespectral construction identifies shared and residual subspaces, and on-line routing selects residuals via first-order gradient-residual alignment,predicting per-sample logit updates under linearization. R2 M is bothefficient and interpretable: expensive computations are performed of-fline, while online inference requires only a single forward–backward passwith lightweight inner-product routing. The same formulation enablesdiagnostic analysis through margin and routing statistics. Experimentsdemonstrate consistently strong performance across in-domain, cross-domain, and unseen settings, highlighting R2 M as an interpretable andscalable approach to training-free deepfake model merging.

Poster
Evidence Triangulation for Multimodal Fact-Checking in the Wild

Stefanos-Iordanis Papadopoulos, Zacharias Chrysidis, Christos Koutlis*, Symeon Papadopoulos, Panagiotis Petrantonakis

展开摘要

The proliferation of multimedia content on social platformshas fueled multimodal misinformation, where images are used to rein-force false claims. Consequently, Multimodal Fact-Checking (MFC) hasemerged as an increasingly important research area. However, currentprogress is hindered by a reliance on synthetic training data and cu-rated benchmarks that fail to capture the complexity of in-the-wild data.Furthermore, existing detection models rely on restricted intra-modalityconsistency or unconstrained all-to-all fusion, failing to capture nuancedrelations between posts and external evidence. To address these limi-tations, we introduce X-POSE, a benchmark of real-world, community-annotated multimodal posts from X (formerly Twitter), augmented withfull-length news articles retrieved via VLM-optimized search. Addition-ally, we propose TRENT, a novel MFC model that performs evidence tri-angulation using three parallel cross-attention streams alongside a rela-tional fusion mechanism that explicitly models entailment and contradic-tion. Extensive evaluations demonstrate that TRENT consistently out-performs state-of-the-art specialized models and commercial VLMs. Thecode, prompt templates, and dataset are available at https://github.com/stevejpapad/evidence-triangulation.

Poster
PWM-ArtGen: Part World Model for Articulated Object Generation

Wentao Zheng, Ancong Wu

展开摘要

The key challenge in articulated 3D object generation from a single image is accurately predicting the underlying kinematic structure. Existing methods either infer kinematic parameters directly from a static image that lacks dynamic part-level kinematic relationships, or estimate parameters from visual dynamics generated from a single image, which is prone to accumulated errors of two steps. Moreover, the limited scale and diversity of existing annotated datasets further hinder generalization to complex, real-world objects. To overcome these limitations, we propose to learn the joint distribution of visual dynamics and kinematic parameters. Recognizing that articulated objects can be formulated as dynamic systems, we propose a unified Part World Model called PWM-ArtGen. To leverage unannotated data, this model couples action diffusion and image diffusion with independent diffusion timesteps, which enables visual branch co-training. We further curate a photorealistic dataset of 19.7k part-level image pairs without kinematic annotations, to support co-training. Experiments demonstrate that PWM-ArtGen substantially outperforms existing baselines in the resting state and exhibits strong zero-shot generalization to out-of-distribution objects. We will release our code at https://github.com/Wentap123/PWM-ArtGen.

Poster
PosterCopilot: Toward Layout Reasoning and Controllable Editing for Professional Graphic Design

Jiazhe Wei, Ken Li, Tianyu Lao, Haofan Wang, Yueming Lyu, Liang Wang, Caifeng Shan, Chenyang Si

展开摘要

Graphic design forms the cornerstone of modern visual com-munication, serving as a vital medium for promoting cultural and com-mercial events. Recent advances have explored automating this processusing Large Multimodal Models (LMMs), yet existing methods often pro-duce geometrically inaccurate layouts and lack the iterative, layer-specificediting required in professional workflows. To address these limitations,we present PosterCopilot, a framework that advances layout reasoningand controllable editing for professional graphic design. Specifically, weintroduce a progressive three-stage training strategy that equips LMMswith geometric understanding and aesthetic reasoning for layout design,consisting of Perturbed Supervised Fine-Tuning, Reinforcement Learn-ing for Visual-Reality Alignment, and Reinforcement Learning from Aes-thetic Feedback. Furthermore, we develop a complete workflow thatcouples the trained LMM-based design model with generative models,enabling layer-controllable, iterative editing for precise element refine-ment while maintaining global visual consistency. Extensive experimentsdemonstrate that PosterCopilot achieves geometrically accurate and aes-thetically superior layouts, offering fine-grained, layer-wise controllabilityfor professional iterative design.

Poster
OSVE: One Step Video Editing with One Step Diffusion Models

Habin Lim, Gyeong-Moon Park

展开摘要

Text-guided video editing with diffusion models is imprac-tically slow, hindered by costly multi-step sampling and inversion. Wepresent OSVE, the first framework to successfully adapt one-step Text-to-Image (T2I) models for high-quality video editing, addressing the corechallenges of inversion, editability, and temporal consistency. To by-pass slow iterative inversion, we train a learnable encoder that pre-dicts the initial noise for each frame in a single forward pass. This en-coder is trained with a novel Structure-Aware Editing (SAE) loss on acurated dataset of structurally-aligned image pairs, teaching it to pre-serve the source video’s geometry during edits. For temporal coherence,we introduce Unified-Frame Editing (UFE), a technique that concate-nates frame latents to facilitate cross-frame attention in a single gen-eration step. Furthermore, for long videos, a sliding-window strategywith an anchor frame maintains global consistency. Our extensive ex-periments demonstrate that OSVE achieves editing quality compara-ble or superior to state-of-the-art multi-step methods, while operatingapproximately 155–171 times faster. This breakthrough paves the wayfor practical, real-time video editing applications. Code is available athttps://github.com/KU-VGI/OSVE.

Poster
Moiré Video Authentication: A Physical Signature Against AI Video Generation

Yuan Qing, Kunyu Zheng, Lingxiao Li, Boqing Gong, Chang Xiao

展开摘要

Recent advances in video generation have made AI-synthesizedcontent increasingly difficult to distinguish from real footage. We pro-pose a physics-based authentication signature that real cameras producenaturally, but that generative models cannot faithfully reproduce. Ourapproach exploits the Moiré effect: the interference fringes formed whena camera views a compact two-layer grating structure. We utilize theMoiré motion invariant as a verification criterion: under real image for-mation, fringe phase and grating image displacement are linearly cou-pled by optical geometry, independent of viewing distance and gratingstructure. A verifier extracts both signals from video and tests theircorrelation. We validate the invariant on both real-captured and AI-generated videos from multiple state-of-the-art generators, and find thatreal and AI-generated videos produce significantly different correlationsignatures, suggesting a robust means of differentiating them. Our workdemonstrates that deterministic optical phenomena can serve as physi-cally grounded, verifiable signatures against AI-generated video.

Poster
ODONet: Online Dynamic Offset Network for Visual Object Tracking

Qinghua Liu, Wanli Xue, Shengyong Chen

展开摘要

Offset learning has recently demonstrated remarkable prowessin capturing the structural diversity of objects. Consequently, it ap-pears to be a natural fit for modeling the discriminative features ofvisual targets in tracking. Yet existing leading offline offset-based meth-ods struggle to cope with the ever-changing appearance and motionof the target during tracking. To address this limitation, we presentODONet, a Transformer-based tracker that performs online dynamicoffset network. Its Dynamic Visual–Motion Feature Interaction (DV-MFI) method fuses target appearance with historical box information,yielding appearance features and motion-aware embeddings. These arethen fed to an Online Dynamic Offset Propagation (ODOP) methodthat explicitly captures inter-frame motion trends, augments them withoffline offsets, and steers deformable attention toward an adaptive re-ceptive field. To our knowledge, ODONet is the first tracker to prop-agate dynamic offsets online, with excellent performance on publiclyavailable datasets, demonstrating its robustness to appearance and mo-tion drift. Code, models, and additional information are available athttps://github.com/WhiteButterflies/ODONet.

Poster
MindFlow: Harmonizing Cognitive Semantics and Acoustic Dynamics for Facial Animation Generation in Dyadic Conversations

Hejia Chen, Haoxian Zhang, Xu He, Xiaoqiang Liu, Pengfei Wan, Shoulong Zhang, Shuai Li

展开摘要

Generating lifelike facial animation for dyadic conversationsrequires reconciling high-level cognitive intent with precise low-level mo-tor reflexes, yet existing methods fall short in the semantic understandingof dialogue context and in precise dynamic control. In this paper, we pro-pose MindFlow, a dual-pathway generative framework inspired by theVentral-Dorsal pathway model in neuroscience, which decouples genera-tion into two collaborative streams, thereby harmonizing deep semanticreasoning with fine-grained control. In the Ventral module, we trans-form the conventional Sentence-Action approach into a novel Chunk-State approach that models raw acoustic streams as a context-aware,evolving emotional state chain, capturing subtle paralinguistic nuancesand mid-utterance emotional shifts missed by sentence-level modeling.The Dorsal module features a conditional autoregressive flow matchingnetwork for high-fidelity facial motion, driven by high-frequency acousticcues and modulated by emotion states, plus a Selective Acoustic Injectorfor adaptive audio gating to ensure robustness in talking-and-listeningdynamics without interference. Extensive experiments demonstrate thatMindFlow achieves superior semantic appropriateness and motion natu-ralness compared to state-of-the-art baselines.

Poster
PA-VAD: Diffusion-Based Pseudo-Only Video Anomaly Detection via Domain-Aligned Memory Updates

Satoshi Hashimoto, Yanan Wang, Hitoshi Nishimura, Mori Kurokawa

展开摘要

Deploying video anomaly detection (VAD) in the real worldis often constrained by the scarcity, privacy, and cost of collecting realabnormal footage. We propose PA-VAD, a novel pseudo-only frame-work that trains an anomaly detector without using any real abnormalvideos, by pairing real normal videos with diffusion-synthesized pseudo-abnormal videos generated from a small set of real normal images. Be-yond proposing a generation-driven training pipeline, we make a keyempirical discovery: pseudo anomalies exhibit a characteristic spatiotem-poral magnitude bias in feature space, which can dominate MultipleInstance Learning and degrade generalization if left unaddressed. Tocounter this pseudo-induced bias, we introduce the Domain-Aligned Reg-ularized Module (DARM), which combines domain alignment with usage-aware memory updates to balance prototype coverage and stabilize opti-mization under biased pseudo supervision. Extensive experiments demon-strate that PA-VAD achieves 98.2% AUC on ShanghaiTech, 82.5% onUCF-Crime, and 95.1% on XD-Violence, and further improves general-ization to unseen anomaly classes in open-set evaluations. Notably, PA-VAD surpasses the best real-abnormal WVAD baselines on ShanghaiTechand XD-Violence by +0.6% and +0.9% , respectively, and improves overthe UVAD state of the art on UCF-Crime by +1.9% —showing that high-accuracy VAD is attainable without collecting real abnormal videos.

Poster
SPOT-E: Test-Time Entropy Shaping with Visual Spotlights for Frozen VLMs

Bo Yin, Xiaobin Hu, Chengming Xu, Ruolin Shen, Mo Yang, Jiangning Zhang, Peng-Tao Jiang, Cheng Tan, Shuicheng YAN

展开摘要

Vision-language models (VLMs) often underperform on evi-dence intensive tasks because decisive visual evidence are small, localized,and easy to overlook, leading to failures in evidence readout even whenhigh-level reasoning is intact. Prior inference-time visual interventionscan improve grounding without retraining, but they are largely open-loop and lack a mechanism to verify whether highlighted evidence is ac-tually used. We study answer-span prediction entropy as a model-internalfeedback signal and show that naive entropy minimization is ambiguous,since low entropy may arise from evidence-grounded confidence or short-cut collapse. To resolve this ambiguity, we introduce low-entropy anchorsand an entropy-shaping objective that reduces answer uncertainty whilepreserving baseline high-confidence tokens. We instantiate this principlein SPOT-E, a plug-and-play test-time method that produces question-conditioned spotlights, optimized per instance via light-weight tuningbased on Group Relative Policy Optimization (GRPO). Across all bench-marks and different VLM families, SPOT-E yields consistent gains andimproved robustness under visual corruptions. Code is publicly availableat: https://github.com/YinBo0927/SPOT-E

Poster
MemRoPE: Training-Free Infinite Video Generation via Evolving Memory Tokens

Youngrae Kim, Qixin Hu, C.-C. Jay Kuo, Peter Beerel

展开摘要

Autoregressive diffusion enables real-time frame streaming,yet existing sliding-window caches discard past context, causing fidelitydegradation, identity drift, and motion stagnation over long horizons.Current approaches preserve a fixed set of early tokens as attentionsinks, but this static anchor cannot reflect the evolving content of agrowing video. We introduce MemRoPE, a training-free framework withtwo co-designed components. Memory Tokens continuously compress allpast keys into dual long-term and short-term streams via exponentialmoving averages, maintaining both global identity and recent dynamicswithin a fixed-size cache. Online RoPE Indexing caches unrotated keysand applies positional embeddings dynamically at attention time, ensur-ing the aggregation is free of conflicting positional phases. These twomechanisms are mutually enabling: positional decoupling makes tempo-ral aggregation well-defined, while aggregation makes fixed-size cachingviable for unbounded generation. Extensive experiments validate thatMemRoPE outperforms existing methods in temporal coherence, visualfidelity, and subject consistency across minute- to hour-scale generation.

Poster
LogiCo: A Unified Framework for Logical and Structural Anomaly Detection

Ximiao Zhang, Min Xu, Xiuzhuang Zhou

展开摘要

Current anomaly detection methods primarily focus on struc-tural anomalies, while paying insu_x001E_cient attention to anomalies that vi-olate logical constraints. Conversely, top-performing logical anomaly de-tection approaches address this by modeling global semantic consistency,but perform poorly on subtle structural anomalies due to inadequate de-tection granularity. In this paper, we propose LogiCo , a uni_x001C_ed frame-work for Logi cal and structural anomaly detection via Co mponent-levelfeature reconstruction. Unlike existing methods that rely on explicitglobal semantic modeling, LogiCo employs a novel component-level fea-ture reconstruction technique to capture inter-component logical con-straints. Speci_x001C_cally, LogiCo maps pre-trained image features into a dis-crete component-level feature space and performs collaborative featurereconstruction at both component and patch levels, enabling it to ef-fectively detect both logical and structural anomalies. Furthermore, toaddress the speci_x001C_c challenge of count-related logical anomalies, we inte-grate a segmentation-map discriminator that extends the model's capa-bility to identify quantitative inconsistencies. LogiCo achieves state-of-the-art performance on both logical and structural anomaly detectionacross four benchmarks, including MVTec-LOCO, MVTec-AD, VisA,and Real-IAD, demonstrating its superiority and practical feasibility.The code is available at https://github.com/cnulab/LogiCo.

Poster
Learning on the Manifold: Unlocking Standard Diffusion Transformers with Representation Encoders

Amandeep Kumar, Vishal Patel

展开摘要

Leveraging representation encoders for generative modelingoffers a path for efficient, high-fidelity synthesis. However, standard diffu-sion transformers fail to converge on these representations directly. Whilerecent work attributes this to a capacity bottleneck—proposing computa-tionally expensive “width scaling” of diffusion transformers—we demon-strate that the failure is fundamentally geometric. We identify GeometricInterference as the root cause: standard Euclidean flow matching forcesprobability paths through the low-density interior of the hypersphericalfeature space of representation encoders, rather than following the mani-fold surface. To resolve this, we propose Riemannian Flow Matchingwith Jacobi Regularization (RJF). By constraining the generativeprocess to the manifold geodesics and correcting for curvature-inducederror propagation, RJF enables standard Diffusion Transformer architec-tures to converge without width scaling. Our method RJF enables thestandard DiT-B architecture (131M parameters) to converge effectively,achieving an FID of 3.37 where prior methods fail to converge.

Poster
Per‑Object IoU Forecasting for Deadline‑Aware Real‑Time Embedded Detection Control

Erfan Foorginejad, Akshar Chavan, Marco Brocanelli

展开摘要

Real-time object detection on edge platforms is constrainedby inference latency due to limited resources. Existing runtime controllersadapt detector configurations using coarse frame-level feedback, but failto account for object-level accuracy degradation caused by detection tem-poral mismatch. This leads to suboptimal scheduling decisions under dy-namic workloads. We propose a lightweight per-object IoU decay modelthat predicts accuracy loss and derives closed-form deadline estimatesfor each detected object. These deadline predictions provide fine-grained,deadline-aware feedback that can be integrated into runtime control poli-cies. We integrate the model into a new closed-loop detector selectionframework and evaluate it under streaming conditions on Argoverse-HDand MOT17 across heterogeneous hardware platforms. Experiments onreal hardware testbeds show that our model and control algorithm im-prove deadline prediction accuracy by 56% and streaming AP by 39% onaverage across 71 video scenarios, while incurring acceptable overhead.The code is available at https://github.com/3Eerfan/OLAP-ECCV2026.

Poster
Learning from Reliable Negatives: Confidence-Anchored Test-Time Adaptation for GUI Grounding

Yizhou Liu, Fei Tang, Yuchen Yan, Zhengxi Lu, Songqin Nong, Tao Jiang, Wenhao XU, Wenqi Zhang, Weiming Lu, Jun Xiao, Yongliang Shen

展开摘要

Graphical User Interface (GUI) grounding is essential for au-tonomous agents to map natural language instructions to precise screencoordinates. However, existing supervised fine-tuning and reinforcementlearning methods are constrained by the high cost of annotation, creatinga scalability bottleneck. In this paper, we introduce a label-free test-timetraining paradigm driven by two key insights: (1) confidence patterns incoordinate tokens are a better indicator than full-sequence confidence,and (2) in sparse GUI coordinate spaces, negative samples offer more re-liable learning signals than potentially noisy positive ones. We first pro-pose Confidence-Anchored Learning (CAL), which utilizes coordinate-token confidence to filter pseudo-labels and assign distance-based binaryrewards. Building on this, we develop Confidence-Anchored NegativeLearning (CANL), which exclusively optimizes the model using negativesamples to bypass the risks of incorrect positive samples. Experimen-tal results demonstrate that CANL-7B achieves 92.1% on ScreenSpot-V2. On more challenging ScreenSpot-Pro, CANL-7B reaches 33.8%, an8.9% absolute improvement over the base model. Our findings establishcoordinate-token confidence as a powerful alternative to manual annota-tions for scalable GUI agent development.

Poster
Improved Immiscible Diffusion: Accelerating Diffusion Training by Reducing Miscibility

Yiheng Li, Feng Liang, Dan Kondratyuk, Masayoshi Tomizuka, Kurt Keutzer, Chenfeng Xu

展开摘要

The substantial training cost of di!usion models hinderstheir deployment. Immiscible Di!usion [20] showed that reducing themixing of images’ di!usion destination in the noise space via linear as-signment can accelerate di!usion training. However, concerns regardinglimited image diversity and exploding execution times under large batchsizes limit its feasibility for large-scale training. In this work, we startfrom thoroughly tackling these limitations: For the image diversity, wedemonstrate the bijective nature of the denoising process of vanilla dif-fusion, underlying that being immiscible cannot hurt the diversity. Mov-ing beyond the noise layer, we refine immiscible di!usion’s concept to abroader miscibility reduction at any layer, which enables us to proposea new family of its implementations much more e"cient to execute un-der high batch sizes, including K-nearest neighbor (KNN) noise selectionand image scaling. Overall, the immiscible di!usion family achieves upto 4→ faster training across diverse models and tasks, including uncon-ditional/conditional generation, image editing, and robotics planning.Extensive analysis shows step-by-step on how immiscibility eases denois-ing and improves e"ciency. Besides, our analysis of immiscibility o!ersa novel perspective on how optimal transport (OT) enhances di!usiontraining. By identifying trajectory miscibility as a fundamental bottle-neck, we believe this work establishes a potentially new direction forfuture research in high-e"ciency di!usion training.

Poster
Hyper-Network Neural Functional Maps for Unsupervised Robust 3D Shape Matching

Dongliang Cao, Florian Bernard

展开摘要

Functional maps are the cornerstone of recent non-rigid 3D shape matching methods due to their efficiency and performance. However, existing methods struggle with challenging scenarios, such as partiality, topological noise, and raw point clouds. A primary bottleneck is that significant intrinsic distortion prevents truncated spectral bases from being accurately aligned via linear transformations (i.e., functional maps). To address this, we introduce a hyper-network that predicts nonlinear neural functional maps (NFM), learned in an unsupervised manner, to better align spectral bases. Specifically, we model the NFM as an MLP with skip-connection to refine standard FM and employ a hypernetwork to predict its weights, conditioned on standard FM. Our framework is trained using a novel unsupervised spectral alignment loss. Experiments demonstrate that our approach can be seamlessly integrated into state-of-the-art unsupervised deep functional map pipelines, substantially improving matching accuracy in demanding scenarios.

Poster
From Perspective to Fisheye Depth Estimation and Open-Vocabulary Segmentation

Rit Gangopadhyay, Alex Wong

展开摘要

Vision foundation models are capable of generalizing across3-dimensional (3D) scenes with high-fidelity estimates; their empiricalsuccess can be attributed to training on large-scale datasets of perspec-tive images. However, when transferred to wide field-of-view (FoV) im-ages, such as those captured by fisheye cameras, they return erroneousoutputs due to a covariate shift stemming from the radial distortion onthe image pixels. We propose a method to generalize vision foundationmodels to fisheye cameras. The crux of our method lies in a set of learn-able parameters, termed Distortion Extenders (DEX), that model thefisheye distortion coefficients and the distributional shift between fish-eye and perspective images encoded in the latent space. By minimizing aself-supervised alignment loss, DEX transforms the latent embeddings offisheye images to resemble those of perspective images to recover high-fidelity estimates. DEX is architecture- and task-agnostic: We demon-strate DEX on monocular depth estimation and open-vocabulary seg-mentation for convolution- and Transformer-based architectures, wherewe consistently improve over baselines across indoor and outdoor fisheyedatasets. As a byproduct, the activations of DEX can also be decoded todistortion coefficients to support camera calibration. Code available at:https://github.com/Suchisrit/DEX.

Poster