ECCV 2026 Papers

Multimodal Large Language Models (MLLMs) have demon-strated strong image-level visual understanding and reasoning, yet theirpixel-level perception across both images and videos remains limited.Foundation segmentation models such as the SAM series produce high-quality masks, but they rely on low-level visual prompts and cannotnatively interpret complex conversational instructions. Existing segmen-tation MLLMs narrow this gap, but are usually specialized for eitherimages or videos and rarely support both textual and visual promptsin one interface. We introduce X2SAM, a unified segmentation MLLMthat extends any-segmentation capabilities from images to videos. Givenconversational instructions and visual prompts, X2SAM couples an LLMwith a Mask Memory module that stores guided vision features for tem-porally consistent video mask generation. The same formulation sup-ports generic, open-vocabulary, referring, reasoning, grounded conversa-tion generation, interactive, and visual grounded segmentation across im-age and video inputs. We further introduce the Video Visual Grounded(V-VGD) segmentation benchmark, which evaluates whether a modelcan segment object tracks in videos from interactive visual prompts.With a unified joint training strategy over heterogeneous image and videodatasets, X2SAM delivers strong video segmentation performance, re-mains competitive on image segmentation benchmarks, and preservesgeneral image and video chat ability. Code is available at https://github.com/wanghao9610/X2SAM.

Generating realistic cast shadows for inserted foreground ob-jects is a crucial yet challenging problem in image composition, wheremaintaining geometric consistency between objects and their shadows incomplex scenes remains difficult due to the ill-posed nature of shadowformation. To address this challenge, we propose VSDiffusion, a visibility-constrained two-stage framework that narrows the solution space by in-corporating visibility priors. In Stage I, a coarse shadow mask is pre-dicted to localize plausible shadow regions. In Stage II, conditional dif-fusion guided by lighting and depth cues estimated from the compositeimage is used to generate accurate shadows. Within VSDiffusion, visibil-ity priors are injected through two complementary pathways: (1) a vis-ibility control branch with shadow-gated cross-attention that providesmulti-scale structural guidance, and (2) a learned soft prior map thatreweights the training loss in error-prone regions to encourage geometriccorrection. In addition, we introduce a high-frequency guided enhance-ment module to sharpen shadow boundaries and improve texture inter-action with the background. Extensive experiments on the widely usedDESOBAv2 benchmark demonstrate that VSDiffusion produces geomet-rically consistent shadows in complex scenes and establishes new state-of-the-art results across most evaluation metrics. The code is availableat https://github.com/Jadelingli/VSDiffusion.

Vision-language models (VLMs) have demonstrated remark-able capabilities in understanding and reasoning about visual content,but significant challenges persist in tasks requiring cross-viewpoint under-standing and spatial reasoning. We identify a critical limitation: currentVLMs excel primarily at egocentric spatial reasoning (from the cam-era’s perspective) but fail to generalize to allocentric viewpoints whenrequired to adopt another entity’s spatial frame of reference. We introduceViewSpatial-Bench, a dedicated multi-perspective spatial localizationbenchmark covering both camera-perspective and human perspective rea-soning, supported by an automated 3D annotation pipeline that generatesprecise directional labels. Comprehensive evaluation of diverse VLMs onViewSpatial-Bench reveals a significant performance disparity: modelsdemonstrate reasonable performance on camera-perspective tasks butexhibit reduced accuracy when reasoning from a human viewpoint. Byfine-tuning VLMs on our multi-perspective spatial dataset, we achievean overall performance improvement of 46.24% across tasks, confirmingthe strong latent spatial reasoning capacity of VLMs under targetedsupervision. Our work establishes a crucial benchmark for spatial intel-ligence in embodied AI systems and provides empirical evidence thatmodeling 3D spatial relationships enhances VLMs’ corresponding spatialcomprehension capabilities.Our benchmark is available at HuggingFace.

Motion, speech, and sound effects are fundamental elementsof human-centric videos, yet their heterogeneous temporal characteris-tics make joint generation highly challenging. Existing audio-video gen-eration models often fail to maintain consistent alignment across thesemodalities, leading to noticeable mismatches between motion, speech,and environmental sounds. We present Unison, a unified framework thatexplicitly promotes coherence across the motion, speech, and sound modal-ities. Within the audio stream, Unison employs a semantic-guided har-monization strategy that decouples the generation of speech and sound-effect components. Leveraging bidirectional audio cross-attention andsemantic-conditioned gating for semantic-driven adaptive recomposition,this approach effectively mitigates speech dominance and enhances acous-tic clarity. For audio–motion synchronization, we propose a bidirectionalcross-modal forcing strategy where the cleaner modality guides the nois-ier one through decoupled denoising schedules, reinforced by a progres-sive stabilization strategy. Extensive experiments demonstrate that Uni-son achieves state-of-the-art performance in both audio perceptual qual-ity and cross-modal synchronization, highlighting the importance of ex-plicit multimodal harmonization in human-centric video generation.

Traditional Shot Boundary Detection (SBD) inherently strug-gles with complex transitions by formulating the task around isolated cutpoints, frequently yielding corrupted video shots. We address this fun-damental limitation by formalizing the Shot Transition Detection (STD)task. Rather than searching for ambiguous points, STD explicitly detectsthe continuous temporal segments of transitions. To tackle this, we pro-pose TransVLM, a Vision-Language Model (VLM) framework for STD.Unlike regular VLMs that predominantly rely on spatial semantics andstruggle with fine-grained inter-shot dynamics, our method explicitly in-jects optical flow as a critical motion prior at the input stage. Through asimple yet effective feature-fusion strategy, TransVLM directly processesconcatenated color and motion representations, significantly enhancingits temporal awareness without incurring any additional visual tokenoverhead on the language backbone. To overcome the severe class im-balance in public data, we design a scalable data engine to synthesizediverse transition videos for robust training, alongside a comprehensivebenchmark for STD. Extensive experiments demonstrate that TransVLMachieves superior overall performance, outperforming traditional heuris-tic methods, specialized spatiotemporal networks, and top-tier VLMs.

Reconstructing 3D scene structures from sparse, low-overlapobservations remains a fundamental challenge in autonomous driving.Recent state-of-the-art frameworks achieve promising results by incor-porating voxel-based Gaussians, but incur substantial computational re-dundancy due to a uniform volumetric processing strategy. To bridgethe gap between the efficiency of pixel-based Gaussian methods and thestructural completeness of voxel-based Gaussian approaches, we proposeFocusGS, a simple yet effective framework that shifts the paradigmfrom global densification to targeted structural completion. Our centralinsight is that structural completion should be decoupled from determin-istic regions, with computation concentrated exclusively on areas exhibit-ing geometric ambiguity. Specifically, FocusGS addresses the localizationchallenge by deriving a 3D Geometric Ambiguity Manifold to accuratelyisolate localized areas prone to occlusion and high geometric uncertainty.To overcome the subsequent manifold completion challenge, we design alightweight targeted structure completion module that selectively instan-tiates and optimizes continuous Gaussian queries strictly within this un-structured, sparse topological subspace. Extensive experiments demon-strate that FocusGS achieves a superior efficiency-quality trade-off, ad-vancing state-of-the-art performance on driving-centric benchmarks whilenaturally reducing the total number of Gaussians by ∼74% and decreas-ing rendering time by ∼34%.

Simulation provides a low-cost, scalable pathway to largescale robotic manipulation data collection. However, existing 3D scene generation methods can rarely be applied directly to manipulation data synthesis, as their generated scenes often lack instance-level interactivity and physical plausibility. Focusing on tabletop manipulation, we propose TabletopGen, a training-free and automated tabletop scene generation and interactive simulation engine. Starting from text or a single image, we first obtain independent 3D object models via generative instance extraction. Second, we introduce a novel pose and scale alignment approach that recovers a collision-free scene layout using a Differentiable Rotation Optimizer and a Top-View Spatial Alignment mechanism. Finally, we assemble the generated scene in a physics simulator with collision geometry, yielding a stable, interactable environment for synthesizing multimodal manipulation data. Extensive experiments and user studies demonstrate that TabletopGen achieves state-of-the-art performance in visual fidelity, layout accuracy, and physical plausibility. Furthermore, we validate the executability of the collected trajectories on a real robotic arm via zeroshot real-to-sim-to-real policy transfer, indicating that TabletopGen can serve as a reliable data engine for robotic manipulation data synthesis.

Diffusion transformer (DiT) architectures show great poten-tial for real-world image super-resolution (Real-ISR). However, theircomputationally expensive iterative sampling necessitates one-step dis-tillation. Existing one-step distillation methods struggle with Real-ISRon DiT. They suffer from fundamental trajectory mismatch and gen-erate severe grid-like periodic artifacts. To tackle these challenges, wepropose StrSR, a novel one-step adversarial distillation framework fea-turing spectral and trajectory regularization. Specifically, we proposean asymmetric discriminative distillation architecture to bridge the tra-jectory gap. Additionally, we design a frequency distribution matchingstrategy to effectively suppress DiT-specific periodic artifacts caused byhigh-frequency spectral leakage. Extensive experiments demonstrate thatStrSR achieves state-of-the-art performance in Real-ISR, across bothquantitative metrics and visual perception. The code and models will bereleased at https://github.com/jkwang28/StrSR.

Semi-supervised few-shot learning (SSFSL) resembles real-world applications such as “auto-annotation”, as it aims to learn a modelfrom a few labeled and abundant unlabeled task-specific examples to an-notate the unlabeled ones. Despite the availability of powerful open-sourceVision-Language Models (VLMs) and open-world data, existing SSFSLliterature largely neglects these resources. In contrast, the related areafew-shot learning (FSL) has already exploited them to boost performance.Arguably, to solve real-world auto-annotation, SSFSL should leverage suchopen resources. To bridge this gap, we explore established SSL methodsto finetune a VLM. Unexpectedly, they significantly underperform FSLbaselines that do not use unlabeled data. Our in-depth analysis revealsthe root cause of failure: VLMs produce “flat” distributions of softmaxprobabilities, resulting in zero utilization of unlabeled data and weak su-pervision signals. To address this challenge, we propose an embarrassinglysimple solution that uses temperatures to sharpen the softmax output,which not only increases the confidence scores of pseudo-labels to improvethe utilization of unlabeled data, but also strengthens training supervisionfor effective finetuning. Furthermore, we exploit task-relevant open data,e.g., those retrieved from VLMs’ publicly available pretraining set. Tomitigate the imbalance and domain gaps in retrieved data, we employa stage-wise training strategy. Building on the successful finetuning ofVLMs and the exploitation of open data, we present a simple yet effectiveSSFSL method, Stage-Wise Finetuning with Temperatures (SWIFT).Across five benchmarks, SWIFT outperforms recent FSL and SSL meth-ods by ∼5 accuracy points. SWIFT even rivals supervised learning, whichfinetunes a VLM assuming unlabeled data having ground-truth labels!

Resting-state functional connectivity (FC) is widely used tomodel brain organization and predict phenotypes, yet short fMRI acqui-sitions often yield unreliable estimates. We propose a Structure-AwareLatent Diffusion framework, SALD, to synthesize long-scan-like FC fromlimited observations. Our model conditions the generation process onshort-scan FC, structural connectivity (SC) as a lightweight anatomi-cal prior, and subject-level covariates. Crucially, in the Adolescent BrainCognitive Development Study (ABCD) cohort, our results suggest a fi-delity–utility tension in FC synthesis: likelihood-based diffusion trainingmay improve distributional fidelity while attenuating subtle inter-subjectvariation relevant to phenotype prediction. To address this, we introducea reward-guided Low-Rank Adaptation (LoRA) strategy that distills aguidance signal isolating the FC-congruent portion of phenotype vari-ance, steering generation toward preserving this signal while maintainingsample realism. Experiments on the ABCD cohort show that our methodbetter preserves phenotype-relevant signal in short-to-long FC synthesis,with the most pronounced improvements over short-scan baselines occur-ring in the low-scan-time regime across the evaluated phenotypes. Thesefindings suggest a promising direction for phenotype-aware short-to-longFC synthesis.

Vision–Language Models (VLMs) provide rich semantic pri-ors but are underexplored in Semi-supervised Semantic Segmentation.Recent attempts to integrate VLMs to inject high-level semantics over-look the semantic misalignment between visual and textual representa-tions that arises from using domain-invariant text embeddings withoutadapting them to dataset- and image-specific contexts. This lack of do-main awareness, coupled with limited annotations, weakens the model’ssemantic understanding by preventing effective vision-language align-ment. As a result, the model struggles with contextual reasoning, showsweak intra-class discrimination, and confuses similar classes. To addressthese challenges, we propose the Hierarchical Vision–Language Trans-former (HVLFormer), which achieves domain-aware and domain-robustalignment between visual and textual representations within a query-driven mask-transformer architecture. Firstly, we transform text embed-dings from a pre-trained VLM into multi-scale, dataset-aware textualobject queries that capture class semantics from coarse to fine gran-ularity and enhance the model’s semantic understanding. Next, thesequeries are refined using image-specific visual context, aligning globaltextual semantics with local scene structures and improving class dis-crimination. Finally, to achieve domain-robustness, we introduce cross-view and modal consistency regularization, which enforces predictionconsistency within the mask-transformer architecture across augmentedviews. It ensures that language queries remain robust to intra-class vari-ations and diverse visual scenes without overfitting to the small labeledset while maintaining stable vision–language alignment under pertur-bations during decoding. With limited training data, HVLFormer out-performs state-of-the-art methods on public benchmarks. Project page:numnz.github.io/HVLFormer.

Spatial reasoning remains a persistent challenge for manyvision–language models (VLMs), and improving it typically requires fine-tuning with substantial additional parameters. Our preliminary analysisreveals that rescaling activations in selected transformer layers—withoutmodifying pretrained weights—can significantly influence downstreamperformance. Motivated by this observation, we propose ScAle, an ultra-lightweight adaptation method that learns a small set of scalar coeffi-cients to modulate last-token attention and MLP activations in a fullyfrozen backbone. We evaluate our method on the synthetic spatial rea-soning benchmark SpatialEval and on real-world VQA datasets (CO-COQA and VGQA) across multiple model families. Our method, ScAle,achieves up to 134.1% relative accuracy gains using only 1K train-able parameters without requiring millions of trainable parameters as instandard LoRA. Despite its extreme compactness, our approach recov-ers a substantial fraction of standard PEFT performance while preserv-ing strong non-spatial VQA accuracy. These results demonstrate thatbounded activation reweighting provides a simple, architecture-agnostic,and highly parameter-efficient alternative for adapting pretrained VLMs.Code is available at https://github.com/rchowdhubnor/ScAle.git.

We present ReinDriveGen, a framework that enables full con-trollability over dynamic driving scenes, allowing users to freely edit actortrajectories to simulate safety-critical corner cases such as front-vehiclecollisions, drifting cars, vehicles spinning out of control, pedestrians jay-walking, and cyclists cutting across lanes. Our approach constructs adynamic 3D point cloud scene from multi-frame LiDAR data, introducesa vehicle completion module to reconstruct full 360° geometry from par-tial observations, and renders the edited scene into 2D condition imagesthat guide a video diffusion model to synthesize realistic driving videos.Since such edited scenarios inevitably fall outside the training distri-bution, we further propose an RL-based post-training strategy with apairwise preference model and a pairwise reward mechanism, enablingrobust quality improvement under out-of-distribution conditions with-out ground-truth supervision. Extensive experiments demonstrate thatReinDriveGen outperforms existing approaches on edited driving scenar-ios and achieves state-of-the-art results on novel ego viewpoint synthesis.

Recent advances in generative super-resolution (SR) havegreatly improved visual realism, yet existing evaluation and optimizationframeworks remain misaligned with human perception. Full-Referenceand No-Reference metrics often fail to reflect perceptual preference, ei-ther penalizing semantically plausible details due to pixel misalignmentor favoring visually sharp but inconsistent artifacts. Moreover, most SRmethods rely on ground-truth (GT)–dependent distribution matching,which does not necessarily correspond to human judgments. In this work,we propose RefReward-SR, a low-resolution (LR) reference-aware rewardmodel for preference-aligned SR. Instead of relying on GT supervisionor NR evaluation, RefReward-SR assesses high-resolution (HR) recon-structions conditioned on their LR inputs, treating the LR image asa semantic anchor. Leveraging the visual–linguistic priors of a Multi-modal Large Language Model (MLLM), it evaluates semantic consistencyand plausibility in a reasoning-aware manner. To support this paradigm,we construct RefSR-18K, the first large-scale LR-conditioned preferencedataset for SR, providing pairwise rankings based on LR–HR consistencyand HR naturalness. We fine-tune the MLLM with Group Relative Pol-icy Optimization (GRPO) using LR-conditioned ranking rewards, andfurther integrate GRPO into SR model training with RefReward-SR asthe core reward signal for preference-aligned generation. Extensive ex-periments show that our framework achieves substantially better align-ment with human judgments, producing reconstructions that preservesemantic consistency while enhancing perceptual plausibility and visualnaturalness.

Vision-Language-Action (VLA) models provide a unifiedparadigm for robotic manipulation, yet their real-world deploymentremains limited by execution efficiency. While existing efforts predomi-nantly focus on compute-centric efficiency to reduce per-step inferencelatency, the intrinsic policy efficiency of these models remains largelyunexplored. Policy efficiency is fundamentally affected by two factors,namely the effective executable length of predicted action chunks and thetotal physical steps required to complete a task. These two factors jointlydetermine the number of forward inference calls during execution. Weobserve that current VLA policies struggle with planning unreliabilityand action redundancy, suffering from severe tail degradation in actionchunks and tending to generate redundant physical steps. To address this,we propose PolicyTrim, a reinforcement learning-based post-trainingframework that extends reliable action chunk length and reduces redun-dant physical steps. For reliable chunk extension, we employ a dynamicexploration strategy that rewards the successful completion of longer exe-cutable lengths, progressively pushing the trustworthy prediction horizonto its empirical limit. For step efficiency, we design a redundancy-awarereward that favors successful task completion with fewer steps whilepenalizing unreproducible shortcuts, effectively eliminating redundantphysical actions. Experiments on three benchmarks and three VLA mod-els show that PolicyTrim improves action chunk utilization by 3×, re-duces physical execution steps by 51.4%, and delivers up to a 5.83× end-to-end deployment speedup without compromising task success rates.Project Page: https://inceptionwang.github.io/PolicyTrim/

3D Gaussian Splatting (3DGS) enables efficient novel-viewsynthesis, yet its extension to simultaneous RGB and thermal infrared(TIR) reconstruction is hindered by a fundamental spectral mismatch:RGB relies on reflected illumination while TIR captures emitted radi-ance. This disparity in frequency characteristics and physical sensingmechanisms causes gradient interference when naively sharing a unifiedrepresentation. Common methods typically treat TIR as an auxiliary in-tensity channel, ignoring the physical radiative transfer governing ther-mal imaging. Critically, enforcing a single opacity field across modalitiesis physically invalid due to cross-spectral visibility disparities, constrain-ing representational capacity and yielding thermally implausible recon-structions. To address these limitations, we propose Physically GroundedDual-Opacity Gaussian Splatting, a framework that unifies RGB andTIR reconstruction under a shared geometric scaffold with modality-aware visibility modeling. We introduce Radiative Attribute Parameter-ization, explicitly modeling each Gaussian’s thermal response throughemissivity, temperature, and reflectance. Dual-Opacity Rendering en-ables spectrally aware visibility handling by assigning modality-specificopacities to shared location parameters, resolving cross-modal occlusionconflicts. Training jointly optimizes photometric reconstruction, physicalpriors, and RGB-guided geometric regularization to resolve parameterambiguities. Experiments demonstrate competitive performance in bothradiometric accuracy and photorealistic novel view synthesis, highlight-ing the potential of 3DGS for joint thermal field with high-fidelity RGBreconstruction.

Recent advances in text-to-video (T2V) generation have achieved good visual quality, yet synthesizing videos that faithfully followphysical laws remains an open challenge. Existing methods mainly basedon graphics or prompt extension struggle to generalize beyond simplesimulated environments or learn implicit physics reasoning. The scarcityof training data with rich physics interactions and phenomena is alsoa problem. In this paper, we first introduce a Physics-Augmented videodata construction Pipeline, PhyAugPipe, that leverages a vision–languagemodel (VLM) with chain-of-thought reasoning to collect a training dataset,PhyVidGen-135K. Then we formulate a principled Physics-aware Group-wise Direct Preference Optimization, PhyGDPO, framework that usesreal-world video as winning case to guarantee correct physics learningand builds upon the groupwise Plackett–Luce probabilistic model tocapture holistic preferences beyond pairwise comparisons. We design aPhysics-Guided Rewarding (PGR) scheme that uses VLM-based physicalrewards to direct the optimization to focus on challenging physics. Plus,we propose a LoRA-Switch Reference (LoRA-SR) scheme that avoidsfull-model duplication as reference for efficient training. Experimentsshow that our method outperforms state-of-the-art methods. Our codeand data is at https://github.com/caiyuanhao1998/Open-PhyGDPO

Recent advances in 3D feedforward reconstruction neural net-works have achieved remarkable success in dense reconstruction from imageswithout any camera parameters. Yet, equipping these models with robustsemantic understanding remains an open problem. Here we introduce anapproach that performs 3D reconstruction and 3D panoptic segmentationin a unified framework. We build on existing 3D reconstruction models andaugment them with a set-based mask decoder. The approach is jointly trainedwith a geometric and semantic loss, which are shown to be mutually beneficial.More precisely, the features are initialized from the geometric informationand then finetuned to capture jointly geometry and semantics. We demon-strate the generality of our approach by successfully applying our frameworkboth to online and all-to-all attention reconstruction backbones. Our methodachieves state-of-the-art performance in 3D panoptic segmentation acrossScanNet, ScanNet200, and ScanNet++ datasets. Ablation studies show thatsuch joint training of a unified model equips 3D feedforward reconstructionneural networks with panoptic segmentation and yields mutually beneficialimprovements.

Existing diffusion-based 3D scene generation methods pri-marily operate in 2D image/video latent spaces, which makes maintain-ing cross-view appearance and geometric consistency inherently chal-lenging. To bridge this gap, we present OneWorld, a framework thatperforms diffusion directly within a coherent 3D representation space.Central to our approach is the 3D Unified Representation Autoencoder(3D-URAE); it leverages pretrained 3D foundation models and augmentstheir geometry-centric nature by injecting appearance and distilling se-mantics into a unified 3D latent space. Furthermore, we introduce token-level Cross-View-Correspondence (CVC) consistency loss to explicitly en-force structural alignment across views, and propose Manifold-Drift Forc-ing (MDF) to mitigate train–inference exposure bias and shape a robust3D manifold by mixing drifted and original representations. Comprehen-sive experiments demonstrate that OneWorld generates high-quality 3Dscenes with superior cross-view consistency compared to state-of-the-art2D-based methods.