ECCV 2026 论文列表

Unified Multimodal Models (UMMs) integrate image under-standing and generation within a single architecture, yet how the twotasks interact remains understudied. We investigate transferability inUMMs: whether training a capability on one task improves the samecapability on the other without explicit supervision. Through controlledexperiments, we empirically find that transferability depends on archi-tecture—models with fully shared transformer backbone and a unifiedvisual encoder exhibit consistent cross-task transfer, while loosely cou-pled designs show little or none. Leveraging this transferability, we pro-pose a practical training strategy. The most straightforward way to im-prove a target generative capability (e.g., counting) is to fine-tune gen-eration directly, but this can degrade visual quality due to distributionshift. Instead, we train the corresponding understanding task and letit transfer into generation, which improves capability-specific genera-tive performance while minimizing distribution shift. We validate thisacross three capabilities—counting, spatial relation, and text recogni-tion/generation—showing that cross-task transferability can be system-atically exploited in UMMs.

Accurate trajectory forecasting of surrounding traffic par-ticipants is a core capability for autonomous driving, enabling vehiclesto anticipate behavior and plan safe maneuvers. We observe that cur-rent state-of-the-art forecasting models on Argoverse 2 and the WaymoOpen Motion Dataset tailor their training objectives to the differentbenchmark metrics. Because these metrics encourage conflicting behav-ior, we propose a paradigm change for trajectory forecasting: trainingmodels with metric-agnostic probabilistic objectives and treating metricoptimization as a downstream task applied to the predictive distribution.Concretely, we introduce Trajectory Distribution Evaluation (TraDiE)policies, metric-specific policies that map a predictive distribution to theset of K trajectories and confidences required by trajectory forecast-ing metrics. We evaluate this framework by introducing DONUT-NLL,which adapts the training objective of the state-of-the-art trajectory fore-casting model DONUT to directly optimize the predictive distribution.Using our policies, DONUT-NLL achieves state-of-the-art results on allmetrics of the Waymo motion prediction benchmark.

Character swapping aims to faithfully transfer a target iden-tity while preserving the source scene layout and pose, yielding visuallyseamless composites. However, existing general image editing methodsoften struggle to simultaneously achieve identity consistency, backgroundpreservation, and pose or structural stability. This challenge is further ex-acerbated by the lack of high-quality character swapping datasets. To ad-dress these challenges, we propose Spatio-Temporal Identity PreferenceAlignment (SIPA) framework, that systematizes diffusion-based pref-erence optimization from both spatial and temporal perspectives. Spa-tially, we introduce Region-Weighted Preference (RWP), which increasesthe weighting of critical regions (e.g., the face or body) during DPOtraining, focusing preference signals on the desired local differences andthereby substantially reducing data dependence while improving train-ing stability. Temporally, we propose Time-Injected Identity Preference(TIP): during low-noise steps, we explicitly inject face-difference cues intothe DPO preference term to further enhance facial identity consistency.In addition, we build and plan to release a two-stage dataset tailoredfor character swapping. Extensive quantitative and qualitative exper-iments demonstrate that SIPA consistently outperforms prior methodsacross identity fidelity, structure preservation, and visual quality, achiev-ing state-of-the-art performance.

Differentially private training adds isotropic Gaussian noiseto clipped gradients, corrupting every singular direction equally. In vi-sion models, where spatial correlation concentrates gradient energy intoa low-rank subspace, most of this noise falls in directions that carrylittle signal. Spectral gradient orthogonalization via polar decomposi-tion is introduced as a post-processing step that recovers directionalsignal from the noisy gradient’s low-rank structure at zero additionalprivacy cost. A phase transition governs the utility of this approach:orthogonalization improves accuracy only when the per-direction spec-tral signal-to-noise ratio (SNR) suffices for singular vector recovery; inlow-SNR regimes, the directional bias of the gradient is replaced by anearly random orthogonal update, and the transformation is harmful.The recovery threshold is determined by the spectral gap of the gradientand is surpassed at large batch sizes. Empirically, the benefit scales withmodel capacity: spectral orthogonalization achieves a +20.9% improve-ment over DP-SGD on WRN-28-10 (B = 4096) and +14.9% on ResNet-18, while reducing inter-run variance by a factor of two to three. In thefine-tuning regime, spectral orthogonalization matches the stability ofDP-Adam while maintaining a first-order memory footprint. Combiningspectral with temporal denoising yields 50.3% on CIFAR-10 (ε = 4), thehighest accuracy in any tested configuration. These gains are specific tomoderate-to-high-SNR regimes such as large-batch training of higher-capacity models. Small-batch or low-SNR settings are better served byDP-SGD or temporal denoising.

Video lane detection requires predictions that remain sta-ble across frames, yet severe vehicle occlusions can break temporal cues.In streaming recurrent models, corrupted observations may enter thehidden state and produce errors that persist into later frames. Existingocclusion-aware refinements usually provide obstacle masks as auxiliaryinputs, so the state-update path is only indirectly protected. We proposeSIGMA-Lane, which treats this failure mode as state contamination inState Space Model (SSM)-based temporal modeling. SIGMA-Lane placesocclusion-aware gates on the SSM write and residual-fusion paths, con-trolling how current observations enter temporal memory and are fusedback after temporal propagation. After coordinate-consistent affine align-ment, the model combines two complementary paths: SSM-consistentdual-gating for temporal filtering and Structural Spatial Retrieval (SSR)for recovering missing lane structure from aligned historical priors. Ex-periments on VIL-100 and OpenLane-V show improved temporal stabil-ity under heavy occlusion, with competitive F1 and mIoU scores.

Computer-use agents that combine GUI interaction withstructured API calls via the Model Context Protocol (MCP) show promisefor automating software tasks. However, existing approaches lack a prin-cipled understanding of how agents should balance these two modalitiesand how to enable iterative self-improvement across diverse applica-tions. We formulate MCP-GUI interplay as a unified hybrid policylearning problem where the agent learns when each modality providescomplementary advantages, and show that distillation and experienceaugmentation target fundamentally different failure modes—requiringapplication-aware mechanism selection. Built on this formulation, wepropose a self-evolving framework with a fully automatic pipelinethat orchestrates automatic environment generation and validation, tra-jectory collection, gap-driven task synthesis, and quality-filtered training—all without manual intervention. A key innovation is our experiencebank, which accumulates LLM-learned rules from trajectory comparison,enabling inference-time improvement without fine-tuning. Systematiccross-application analysis across three desktop applications revealsthat the optimal strategy depends on MCP–GUI composition and tool-chain complexity: distillation achieves 77.8% pass rate on MCP-dominanttasks (+17.8pp), while the experience bank excels on GUI-intensive tasks(+10.0pp). Code available at: https://github.com/Tiantian-H/EE-MCP

Existing safety mechanisms for multimodal large languagemodels (MLLMs) face a fundamental trade-off between safety and util-ity. Model fine-tuning achieves robust safety but compromises generalutility. Input-side safety guardrails offer a lightweight alternative, yetthey suffer from severe over-refusal, indiscriminately blocking benignqueries or those the model could have safely answered through refusalor advisory responses. We identify that the root cause of over-refusallies in the input-aware paradigm: safety guardrails make safety decisionswithout considering whether the model itself is capable of generatingsafe responses. Usually, MLLMs already possess intrinsic safety mech-anisms that can transform harmful inputs into harmless outputs, butinput-side safety guardrails override this capability, degrading user ex-perience. Motivated by this insight, we propose a paradigm shift towardoutput-aware safety guardrails. Our method operates within the model’shidden state space to predict whether the forthcoming generation willbe unsafe before it is fully produced. By training a lightweight classifiervia multi-instance contrastive learning on hidden state representations,our approach distinguishes between inputs that will lead to unsafe out-puts and those that will not, even when the inputs themselves containrisky elements. This enables precise intervention only when the model’sactual response would be harmful. Extensive experiments demonstratethat our output-aware safety guardrail matches the safety performance ofexisting methods while drastically reducing over-refusal, preserving themodel’s utility and built-in safety capabilities. Our code is available athttps://github.com/kunzhan/OutGuard.

Infrared small target detection (IRSTD) aims to localizesmall targets in cluttered infrared scenes. Extensive research follows thepixel-level supervision-guided ‘encoder–decoder’ segmentation paradigm.Although these methods have achieved promising performance, they of-ten overlook that infrared small targets occupy only a few pixels andare usually surrounded by blurred, low-contrast boundaries caused bycluttered backgrounds. Based on this observation, we argue that thefirst principle of IRSTD should be target localization rather than densereconstruction of target regions entangled with indistinguishable back-ground noise. In this paper, we reformulate IRSTD as a centroid re-gression task and propose a novel Single-Point Supervision guided In-frared Probabilistic Response Encoding method (namely, SPIRE), whichis non-trivial because point-level supervision must produce detectionoutputs comparable to dense supervision. Specifically, we design Point-Response Prior Supervision (PRPS) to transform single-point annota-tions into probabilistic response maps consistent with infrared point-target response characteristics, and combine it with a High-ResolutionProbabilistic Encoder (HRPE) that performs encoder-only probabilis-tic regression followed by lightweight peak-based inference, without de-coder reconstruction. By preserving high-resolution features and increas-ing effective supervision density, SPIRE alleviates optimization instabil-ity under sparse target distributions. Finally, extensive experiments ontwo public IRSTD benchmarks, SIRST-UAVB and SIRST4, demonstratethat SPIRE achieves competitive target-level detection performance withconsistently low false alarm rate (Fa) and significantly reduced compu-tational cost.

Instruction-based image editing enables natural-language con-trol over visual modifications, yet existing models falter under Instruc-tion–Visual Complexity (IV-Complexity), where intricate instructionsmeet cluttered or ambiguous scenes. We introduce RePlan (Region-alignedPlanning), a plan-then-execute framework that couples a vision–languageplanner with a diffusion editor. The planner decomposes instructions viastep-by-step reasoning and explicitly grounds them to target regions;the editor then applies changes using a training-free attention-region in-jection mechanism, enabling precise, parallel multi-region edits withoutiterative inpainting. To strengthen planning, we apply GRPO-based rein-forcement learning using 1K instruction-only examples, yielding substan-tial gains in reasoning fidelity and format reliability. We further presentIV-Edit, a benchmark focused on fine-grained grounding and knowledge-intensive edits. Across IV-Complex settings, RePlan consistently outper-forms strong baselines trained on far larger datasets, improving regionalprecision and overall consistency. Project page: https://replan- iv-edit.github.io/

Reference-to-video (R2V) generation is a controllable videosynthesis paradigm that constrains the generation process using bothtext prompts and reference images, enabling applications such as per-sonalized advertising and virtual try-on. In practice, existing R2V meth-ods typically introduce additional high-level semantic or cross-modal fea-tures alongside the VAE latent representation of the reference image andjointly feed them into the diffusion Transformer (DiT). These auxiliaryrepresentations provide semantic guidance and act as implicit alignmentsignals, which can partially alleviate pixel-level information leakage inthe VAE latent space. However, they may still struggle to address copy–paste artifacts and multi-subject confusion caused by modality mismatchacross heterogeneous encoder features. In this paper, we propose Re-fAlign, a representation alignment framework that explicitly aligns DiTreference-branch features to the semantic space of a visual foundationmodel (VFM). The core of RefAlign is a reference alignment loss thatpulls the reference features and VFM features of the same subject closerto improve identity consistency, while pushing apart the correspond-ing features of different subjects to enhance semantic discriminability.This simple yet effective strategy is applied only during training, incur-ring no inference-time overhead, and achieves a better balance betweentext controllability and reference fidelity. Extensive experiments on theOpenS2V-Eval benchmark demonstrate that RefAlign outperforms cur-rent state-of-the-art methods in TotalScore, validating the effectivenessof explicit reference alignment for R2V tasks.

We present RadarGen, a diffusion model for synthesizing re-alistic automotive radar point clouds from multi-view camera imagery.RadarGen adapts efficient image-latent diffusion to the radar domain byrepresenting radar measurements in bird’s-eye-view form that encodesspatial structure together with radar cross section (RCS) and Dopplerattributes. A lightweight recovery step reconstructs point clouds fromthe generated maps. To better align generation with the visual scene,RadarGen incorporates BEV-aligned depth, semantic, and motion cuesextracted from pretrained foundation models, which guide the stochasticgeneration process toward physically plausible radar patterns. Condition-ing on images makes the approach broadly compatible, in principle, withexisting visual datasets and simulation frameworks, offering a scalable di-rection for multimodal generative simulation. Evaluations on large-scaledriving data show that RadarGen captures characteristic radar measure-ment distributions and reduces the gap to perception models trained onreal data, marking a step toward unified generative simulation acrosssensing modalities.

We propose a multimodal, physically grounded approach formetric-scale amodal object reconstruction and pose estimation undersevere hand occlusion. Unlike prior occlusion-aware 3D generation meth-ods that rely only on vision, we leverage physical interaction signals:proprioception provides the posed hand geometry, and multi-contacttouch constrains where the object surface must lie, reducing ambigu-ity in occluded regions. We represent object structure as a pose-aware,camera-aligned signed distance field (SDF) and learn a compact latentspace with a Structure-VAE. In this latent space, we train a condi-tional flow-matching diffusion model, pretraining on vision-only imagesand finetuning on occluded manipulation scenes while conditioning onvisible RGB evidence, occluder/visibility masks, the hand latent rep-resentation, and tactile information. Crucially, we incorporate physics-based objectives and differentiable decoder-guidance during finetuningand inference to reduce hand–object interpenetration and to align thereconstructed surface with contact observations. Because our methodproduces a metric, physically consistent structure estimate, it integratesnaturally into existing two-stage reconstruction pipelines, where a down-stream module refines geometry and predicts appearance. Simulation ex-periments show that adding proprioception and touch substantially im-proves completion under occlusion and yields physically plausible recon-structions at correct real-world scale compared to vision-only baselines;we further validate transfer by deploying the model on a real humanoidrobot with an end-effector different from those used during training. Seehttps://github.com/hsp-iit/physical-generative-reconstruction

Underwater object detection is fundamentally ill-posed dueto severe light attenuation and scattering. These physical degradationsinduce highly non-linear geometric distortions in the visual feature space,deviating significantly from the canonical manifold of clear images andrendering standard Euclidean convolutions suboptimal. To address this,we propose PhysFlowNet, that reformulates representation rectificationas a physics-guided geometric evolution on a Riemannian manifold. Phys-FlowNet first extracts a joint spatial-frequency physical prior to encap-sulate macroscopic scattering mechanics. Conditioned on this prior, weintroduce the Parallel Physics-Residual Bottleneck (PPRB), which exe-cutes a metric-preconditioned Riemannian feature retraction to transportdistorted features back to their canonical states. Moreover, to counter-act cascaded smoothing during scale transitions, we introduce a Physics-Guided Manifold Downsampling (PMD) strategy to strictly preserve finedetail textures and structural boundaries. Ultimately, the network is op-timized via a Unified Evidential-Contrastive Objective (UECO). Com-prising a Supervised Contrastive Manifold Loss (SCML) and an Eviden-tial Uncertainty Regulation (EUR), UECO jointly ensures latent geo-metric alignment and reliable uncertainty calibration. Extensive evalua-tions across challenging underwater benchmarks demonstrate that Phys-FlowNet achieves state-of-the-art performance, establishing a principledgeometric paradigm for physically degraded vision tasks.

Despite the rapid progress of Multimodal Large LanguageModels (MLLMs) in 2D vision-language tasks, robust multi-view spatialreasoning remains a fundamental bottleneck due to the lack of struc-tured 3D cognitive pathways in existing datasets. To address this, weintroduce MV-STRIDE, a Multi-View hierarchical SpaTial Reasoningdataset with Interdependent and Decomposed capabilitiEs. Moving be-yond _x001D_at data structures, MV-STRIDE explicitly models the dependencyrelationships between foundational perception, scene understanding, andcomplex contextual reasoning, providing a coherent learning pathwayaligned with human spatial cognition. We develop a systematic QA gen-eration pipeline leveraging diverse 3D scene sources that enforces cross-view dependency constraints to prevent single-view solvability, generat-ing multi-level spatial reasoning tasks supported by cognitively groundedchain-of-thought supervision for complex inference. Extensive evalua-tions demonstrate that our multi-stage training framework based on ourhierarchical dataset achieves state-of-the-art performance across multiplespatial reasoning benchmarks, notably the multi-view oriented MMSI-Bench. Our approach enables MLLMs to maintain robust, 3D-consistentspatial reasoning across diverse viewpoints. The code and dataset areavailable at https://co1dspring.github.io/MV-STRIDE/.

Reinforcement Learning from Human Feedback has emergedas a standard for aligning di!usion models. However, we identify a fun-damental limitation of the standard DPO formulation when optimizingmulti-dimensional preferences: by relying on the Bradley-Terry model,it aggregates heterogeneous evaluation axes (e.g., aesthetic quality andsemantic alignment) into a single scalar reward. This aggregation createsa reward conflict where the model is forced to unlearn desirable featuresof a specific dimension if they appear in a globally non-preferred sam-ple. To address this issue, we propose Multi Reward Conditional DPO(MCDPO) which resolves reward conflicts by introducing a disentangledBradley-Terry objective. MCDPO explicitly injects a preference outcomevector as a condition during training, which allows the model to learnthe correct optimization direction for each reward axis independentlywithin a single network. Extensive experiments on Stable Di!usion 1.5and SDXL demonstrate that MCDPO achieves superior performance onbenchmarks even with 18% and 3% training data respectively. Notably,our conditional framework enables dynamic and multiple-axis control atinference time using Classifier Free Guidance to amplify specific rewarddimensions without additional training or external reward models.

Infrared and visible video fusion combines the object saliencyfrom infrared images with the texture details from visible images toproduce semantically rich fusion results. However, most existing meth-ods are designed for static image fusion and cannot effectively han-dle frame-to-frame motion in videos. Current video fusion methods im-prove temporal consistency by introducing interactions across frames,but they often require high computational cost. To mitigate these chal-lenges, we propose MAVFusion, an end-to-end video fusion frameworkfeaturing a motion-aware sparse interaction mechanism that enhancesefficiency while maintaining superior fusion quality. Specifically, we lever-age optical flow to identify dynamic regions in multi-modal sequences,adaptively allocating computationally intensive cross-modal attentionto these sparse areas to capture salient transitions and facilitate inter-modal information exchange. For static background regions, a lightweightweak interaction module is employed to maintain structural and appear-ance integrity. By decoupling the processing of dynamic and static re-gions, MAVFusion simultaneously preserves temporal consistency andfine-grained details while significantly accelerating inference. Extensiveexperiments demonstrate that MAVFusion achieves state-of-the-art per-formance on multiple infrared and visible video benchmarks, achieving aspeed of 14.16 FPS at 640 × 480 resolution. The source code is availableat https://github.com/ixilai/MAVFusion.

Recent years have witnessed remarkable progress in autonomousdriving, yet generalization to long-tail and open-world scenarios remainsthe primary bottleneck for large-scale deployment. To address this, oneline of research explores LLMs and VLMs for their vision-language un-derstanding and reasoning capabilities, equipping AVs with the abilityto interpret rare and safety-critical situations when generating drivingactions. In parallel, another line investigates generative world models tocapture the spatio-temporal evolution of driving scenes, enabling agentsto imagine and evaluate possible futures before acting. Inspired by hu-man intelligence, which seamlessly unites understanding and imaginationas a hallmark of AGI, this work explores a unified model that bringsthese two capabilities together for autonomous driving. We present LM-GenDrive, the first framework that unifies LLM-based multimodal un-derstanding with generative world models for end-to-end closed-loop au-tonomous driving. Given multi-view camera inputs and natural-languageinstructions, our model generates both realistic future driving videosand corresponding control signals. By coupling an LLM with genera-tive video capabilities, LMGenDrive gains complementary benefits: fu-ture video prediction enhances spatio-temporal scene modeling, while theLLM provides strong semantic priors and instruction grounding learnedfrom large-scale pretraining. A progressive three-stage training strat-egy—ranging from vision pretraining to multi-step long-horizon driv-ing—is proposed to further improve stability and performance. The re-sulting model can also operate in two complementary modes: low-latencyonline planning and autoregressive offline video generation. Experimentsshow that LMGenDrive significantly outperforms state-of-the-art meth-ods on challenging closed-loop driving benchmarks, improving instruc-tion following, spatio-temporal understanding, and robustness to rarescenarios. Our work not only sets a new state-of-the-art in autonomousdriving, but also demonstrates that unifying multimodal understandingand generation provides a promising direction for building more gener-alizable and robust embodied decision-making systems.

Streaming video editing has made rapid progress, yet practi-cal deployment is still limited by two core issues: maintaining stable back-grounds and non-edited regions over time, and achieving the low latencyrequired for real-time interactive scenarios. Meanwhile, recent streamingvideo generation methods are mostly developed for synthesis and cannotbe directly applied to editing due to the strict preservation requirementand region-specific control. In this work, we present a novel streamingvideo editing framework that performs causal, frame-by-frame editingwith strong content preservation and real-time responsiveness. Our keydesign is a three-stage distillation pipeline that progressively transfersediting capability from a powerful bidirectional foundation model to anefficient unidirectional streaming editor, enabling stable long-horizon ed-its without sacrificing visual fidelity. To further support real-time deploy-ment, we introduce an AR-oriented mask cache that reuses region-relatedcomputation across frames, substantially reducing redundant processingand accelerating inference. Finally, we establish a dedicated benchmarkfor streaming video editing. Extensive evaluations demonstrate that ourmethod achieves state-of-the-art visual quality among streaming base-lines while drastically boosting inference speed to 12.66 FPS, making itsuitable for interactive and augmented reality applications.

Change detection aims to identify semantic changes betweenremote sensing images. However, features from models are easily dis-turbed by non-semantic variations, such as illumination, shadows, andatmospheric changes, leading to false alarms and limited generaliza-tion in real-world scenarios. In this paper, we propose SCDistill, aframework for learning semantic-robust change detection via semantic-invariant self-distillation. First, to strengthen semantic consistency, weintroduce a semantic-invariant self-distillation strategy that learns se-mantic robustness from perturbed yet semantically consistent data, em-powering the change detector to extract disturbance-resistant featuresand achieve more reliable and accurate semantic change identification.Second, to expand paired data with non-semantic variations, we designa diffusion-based perturbation simulation pipeline that synthesizes com-plex environmental changes, enabling the model to explicitly learn todistinguish semantic changes from appearance-level fluctuations and re-duce false alarms caused by non-semantic disturbances. These compo-nents promote robustness from data and representation perspectives,leading to synergistic performance gains. Extensive experiments demon-strate that SCDistill achieves state-of-the-art performance on multiplesemantic change detection benchmarks and exhibits strong generaliza-tion to binary change detection and change captioning tasks. Code isaccessible at https://github.com/elecreak/SCDistill.