paperlist.ai

AI conference papers

AI conference paper library

Search papers from 30+ top conferences, including NeurIPS, ICLR, ICML, CVPR, ACL and AAAI. Filter by keyword, then open any paper for its abstract, authors and links.

ECCV 2026

Stitched Embeddings: A Unified Latent Space for 3D Garments and 2D Patterns

Andrea Sanchietti, Riccardo Marin, Bharat Bhatnagar, Yuanlu Xu, Gerard Pons-Moll

While garments are essential for realistic digital humans,their topological variety makes them much harder to model than para-metric bodies. Traditional tailoring relies on 2D sewing patterns, yetbridging these patterns to 3D geometry currently requires physical simu-lations. We present Stitched Embeddings, the first simulation-free frame-work to unify 3D garment reconstruction and sewing pattern inferencewithin a single bidirectional latent space. By leveraging the geometricpriors of a pretrained 3D foundation model, our approach overcomesthe data scarcity typically associated with high-quality garment mod-eling. We propose to use the BoxMesh as a critical intermediate repre-sentation to align 2D panels into 3D configurations without the com-putational overhead of a simulator. This architecture achieves state-of-the-art accuracy in pattern reconstruction while significantly improvingefficiency. Furthermore, our differentiable pipeline enables novel applica-tions, including pattern recovery from meshes and 3D editing from 2Dpatterns. Finally, this work provides a scalable link between neural 3Dvision and the physical garment manufacturing pipeline. Project page:

ECCV 2026

Starve to Perceive: Taming Lazy Perception in VLMs with Constrained Visual Bandwidth

Yuhuan Wu, Haozhe Wang, Cong Wei, Chong Peng, Fangzhen Lin, Wenhu Chen

Vision-Language Models (VLMs) deployed as situated agentsin high-resolution visual environments require active perception — theability to dynamically decide where to look through operations like zoom-ing, cropping, and panning. However, current training paradigms pro-duce models that mimic the surface form of such operations withoutfunctionally depending on their outputs, a phenomenon we term lazyperception. We trace this to a fundamental learning asymmetry: whencoarse global views combined with language priors suffice for moderateaccuracy, the model has no incentive to learn harder multi-step visualsearch. If a model can succeed without actively looking, it will neverlearn to look. This motivates Starve to Perceive, a training paradigm thatconstrains visual bandwidth — restricting each observation to a tight to-ken budget so that no single view suffices for task completion, therebystrongly incentivizing strategic and efficient active perception. Despiterequiring no auxiliary losses, reward shaping, or architectural changes— serving as a minimal, plug-in modification to standard post-trainingpipelines — models trained under perceptual starvation achieve substan-tial gains of 5% average relative improvement across diverse benchmarks.Our codes and data will be publicly available at https://github.com/WhuanY/Starve2Perceive.

ECCV 2026

Stabilizing Deep Reconstruction Operators with Contractive Anchoring

Arghya Sinha, Trishit Mukherjee, Kunal Chaudhury

Pretrained deep denoisers can be used to solve a wide rangeof model-based image reconstruction tasks via Plug-and-Play (PnP) andRegularization-by-Denoising (RED) algorithms, without retraining pertask. These denoisers are trained only for single-step denoising. Usingthem as Image Reconstruction (IR) regularizers in an iterative processcan destabilize reconstruction. A common failure mode is the peak-and-collapse behaviour: metrics such as PSNR improve for early iterations andthen abruptly degrade, making these algorithms unreliable in practice.We propose a data-driven stabilization framework that (i) formalizes thisinstability of any IR operator through a local quantity and (ii) preventscollapse by regularizing this quantity adaptively, requiring no retrainingor modification of the given pretrained network. Our key idea is tocontrol the potentially unstable IR operator with a contractive operatorwhose stable iterates act as an anchor and prevent collapse. We furtherintroduce an efficient family of trainable contractive operators that serve asstrong anchors while remaining lightweight. Extensive experiments acrossproximal algorithms, denoiser architectures, noise levels, and imagingtasks show consistent, collapse-free performance and improved reliabilityof PnP and RED reconstruction.

ECCV 2026

Spectral Evolution-Guided Token Pruning in Large Multimodal Models

Bin CHEN, Yuxiang Cai, Yadan Luo, Yi Zhang, Jianwei Yin, Zhi Chen

Reducing visual token redundancy is critical for acceleratingMultimodal Large Language Models (MLLMs) without degrading cross-modal reasoning performance. Existing token pruning methods typicallyrely on single-layer signals, such as attention scores or token similari-ties, which overlook the cross-layer transformation of visual representa-tions and may exhibit positional bias in multimodal token sequences.To address this limitation, we propose a training-free token pruningframework based on Cross-Layer Spectral Evolution (CLSE). Instead ofmeasuring token importance from single-layer feature magnitudes, CLSEquantifies how token representations evolve across Transformer layers inthe frequency domain. This evolution reflects the transition from high-frequency structural details to low-frequency semantic abstractions. Weobserve that tokens with stronger spectral redistribution across layers aremore likely to be semantically active and should therefore be preserved.By modeling cross-layer token dynamics, CLSE provides a stable im-portance criterion that mitigates positional bias. Extensive experimentson both image and video benchmarks demonstrate that CLSE achievesa superior trade-off between efficiency and accuracy under aggressivetoken reduction. Across multiple MLLMs, CLSE reduces FLOPs, KVcache memory, and latency while maintaining competitive or improvedperformance.

ECCV 2026

Decoding Children’s Gait Behavior

Yifan Shen, Boyi Li, Meihuan Huang, Yuanzhe Liu, Xu Cao, Jinyang Jin, Zhengyuan Li, Anglin Liu, Junho Kim, JingYuan Zhu, Lan Fangzhou, Jianguo Cao, Jintai Chen, Ismini Lourentzou, James Rehg

We introduce a new problem domain for human action recog-nition: the fine-grained analysis of children’s gait behaviors from stan-dard RGB video. We specifically target the ambulatory patterns of chil-dren aged 3–17 years. Such behaviors arise naturally in the diagnosis andtreatment of several critical developmental and neuromuscular disorders,such as cerebral palsy and hemiplegia. Despite their clinical value, cur-rent 3D sensor-based gait analysis systems are expensive, intrusive, andoften impractical for young subjects. To address this, we introduce anew dataset comprising over 1,100 high-frame-rate (60 FPS) video se-quences from 110 subjects, accompanied by synchronized, anonymizedpose sequences. In each session, the child performs a 5-second "walk-around" task, capturing the gait cycle from multiple viewpoints. Cru-cially, we demonstrate that current state-of-the-art approaches, includ-ing gait foundation models and Multimodal Large Language Models(MLLMs), fail to effectively resolve these clinical nuances. We identifythe key technical challenges in analyzing these erratic and subtle mo-tor patterns and describe a unified end-to-end framework for decodingfundamental components of pediatric gait. Through comprehensive ex-perimental results, we demonstrate the potential of this dataset to drivenovel research questions and establish a rigorous baseline for automatedchild gait assessment. Project page: pediamedai.com/ChildrenGait

ECCV 2026

SMART: When is it Actually Worth Expanding a Speculative Tree?

Lifu Wang, Pan Zhou

Tree-based speculative decoding accelerates autoregressive generation by verifying a branching tree of draft tokens in a single targetmodel forward pass. However, existing methods prioritize maximizing token-level likelihood or the number of accepted tokens while ignoring a critical “efficiency paradox”: the computational overhead of drafting and verifying big trees can grow super-linearly, particularly at scale. This often leads to negative wall-clock speedup when batch sizes increase or hardware saturation limits are reached. To address this, we propose SMART, a System-aware Marginal Analysis framework for Runtime Tree construction. SMART reformulates tree expansion as a hardware-aware optimization problem that directly maximizes end-toend speedup. By applying a principled marginal benefit–cost rule at inference time, SMART expands a node only when its marginal benefit– cost ratio exceeds the tree-level speedup. SMART is training-free and serves as a plug-and-play controller for existing frameworks like MSD and EAGLE. Extensive evaluations across three MLLMs (e.g., LLaVA, Qwen2-VL) and four LLMs (e.g., Llama-3.1, DeepSeek-R1) demonstrate that SMART consistently outperforms state-of-the-art baselines. It delivers an average additional speedup of 20.0% for MLLMs and 15.4% for LLMs across compute-bound batching regimes and diverse GPU architectures without performance loss.

ECCV 2026

Self-Evolving Just-In-Time Memory for Proactive Embodied Safety

Bingrui Sima, Lizhong Wang, Xiaoya Lu, Kun He, Xiao Yang

While Vision-Language Models (VLMs) have empowered em-bodied agents to execute complex household tasks, they struggle toproactively handle dynamically emerging hazards during closed-loop in-teractions. Existing safety approaches often rely on runtime guardrailsto block unsafe actions or induce excessive caution, which severely stallstask progress instead of actively resolving the underlying risks. To breakthis safety–progress trade-off, we introduce the Self-Evolving Just-In-Time Memory framework, which reframes embodied safety from progress-stalling guardrails to proactive hazard mitigation. The framework con-sists of a Risk-Sufficient Topological Belief Graph (RSG) for persistentsafety-relevant state tracking under partial observability, an Agency-Grounded Factual Memory for precise hazard anticipation, and an Expe-rience Memory that injects procedural Meta-Skills to guide executable,progress-preserving mitigation. Furthermore, we propose an automatedTest-Verify-Write loop, allowing agents to continually refine their mit-igation Meta-Skills from execution traces at test time. Experiments onIS-Bench demonstrate that our framework substantially boosts the Safe-Success rate across multiple VLM backbones (e.g., +30.3% on Qwen3-VL-8B), enabling agents to proactively mitigate hazards without stallingtask progress. Code is available at https://github.com/DyMessi/JIT-Memory.

ECCV 2026

SeekFlow: Synergizing Radiology and Pathology Foundation Models for Precision Oncology via Knowledge-Guided Evidence Flow

Peixiang Huang, Yanyan Huang, Yihang Chen, Maximus Yeung, Yuming Jiang, Lequan Yu

Integrating macro-scale radiology and micro-scale pathologyprovides complementary insights crucial for precision oncology. How-ever, this integration is often hindered by modality discrepancy. The fea-ture embeddings from heterogeneous foundation models reside on poorlyaligned latent manifolds, and naive static fusion may therefore mix unre-lated semantic neighborhoods, leading to topological degeneracy and se-mantic entanglement. To bridge this gap, we propose a novel multimodallearning framework for Synthesizing radiology and pathology EvidencEvia Knowledge-guided FLOW matching. Specifically, we construct ashared feature manifold anchored by learnable clinical prototypes, en-abling the disentanglement and synthesis of shared and modality-specificfeature evidence. We then reformulate multimodal integration as a trans-port problem via knowledge-guided flow matching and seek continuousflows that smoothly transport features onto this semantic substrate. Thisparadigm preserves the intrinsic topology of the feature spaces and re-duces feature collapse. Furthermore, an uncertainty-aware evidential ag-gregator is incorporated to synthesize cross-modal feature evidence forreliable decision-making. Extensive experiments on three cancer datasets(i.e., TCGA Glioma, Gastric Cancer, and Chondrosarcoma) demonstratethe effectiveness of SeekFlow. Notably, SeekFlow consistently achievesstate-of-the-art performance across all benchmarks, outperforming ex-isting multimodal fusion methods in both diagnosis and prognosis tasks.The code is available at https://github.com/HKU-MedAI/SeekFlow.

ECCV 2026

ScenarioControl: Vision-language Controllable Vectorized Latent Scenario Generation

Lili Gao, Yanbo Xu, William Koch, Samuele Ruffino, Luke Rowe, Behdad Chalaki, Dmitriy Rivkin, Julian Ost, Roger Girgis, Mario Bijelic, Felix Heide

We introduce ScenarioControl, the first vision-language con-trol mechanism for learned driving scenario generation. Given a textprompt or an input image, ScenarioControl synthesizes diverse, realistic3D scenario rollouts – including map, 3D boxes of reactive actors overtime, pedestrians, driving infrastructure, and ego camera observations.The method generates scenes in a vectorized latent space that representsroad structure and dynamic agents jointly. To connect multimodal con-trol with sparse vectorized scene elements, we propose a cross-global con-trol mechanism that integrates cross-attention with a lightweight global-context branch, enabling fine-grained control over road layout and traf-fic conditions while preserving realism. The method produces tempo-rally consistent scenario rollouts from the perspectives different actorsin the scene, supporting long-horizon continuation of driving scenarios.Extensive experiments validate that the control adherence and fidelityof ScenarioControl compare favorable to all tested methods across allexperiments. To facilitate training and evaluation, we release a datasetwith text annotations aligned to vectorized map structures at the projectwebpage: https://light.princeton.edu/ScenarioControl

ECCV 2026

SAMPLe: A Sharpness Aware Minimization based Optimizer for Prompt Learning in Vision-Language Models

Hossein Rajoli nowdeh, Fatemeh Lotfi, Niloufar Alipour Talemi, Hossein Kashiani, Xiaolong Ma, Fatemeh Afghah

Pre-trained Vision-Language Models (VLMs) like CLIP haveproven highly effective as foundation models for various downstream ap-plications. However, prompt learning in VLMs encounters a performance-generalization dilemma: while prompts can be tuned to achieve highaccuracy on seen distributions, this tuning process often undermines theirgeneralizability to unseen data. The limited set of learnable prompts,which contextualize and condition the input to steer it toward the taskwithin the pretrained VLM, tends to overfit the training data, leading toa trade-off between task-specific performance and preserving generaliza-tion. To address this dilemma, we introduce SAMPLe (Sharpness-AwareMinimization Prompt Learning), a plug-in sharpness-aware optimizerthat enhances prompt generalizability by accounting for loss landscapesharpness. Unlike conventional methods, SAMPLe balances explorationand exploitation by satisfying objective function constraints at each step,dynamically adapting to the current optimization state based on the localcurvature and gradient properties. This approach reduces overfitting onseen distributions and improves adaptability to unseen data, preservingthe generalization potential of pre-trained VLM models. We integrateSAMPLe into multiple prompt learning frameworks, including CoOp,CoCoOp, MaPLe, TCP, and Co-Prompt, demonstrating its effectivenessacross diverse methods. Experiments show that SAMPLe elevates promptlearning frameworks and consistently outperforms existing optimizersacross diverse settings, establishing itself as a robust, model-agnosticsolution for prompt learning.

ECCV 2026

OpenSubject: Leveraging Video-Derived Identity and Diversity Priors for Subject-driven Image Generation and Manipulation

Yexin Liu, Manyuan Zhang, Yueze Wang, Hongyu Li, Dian Zheng, Weiming Zhang, Changsheng Lu, Harry Yang

Subject-driven image generation and manipulation are fun-damental for personalized content creation, such as identity-preservingportrait synthesis, multi-character storytelling, and controllable photoediting. However, existing models still struggle with multi-reference set-tings, often showing identity drift and unstable editing quality. A keybottleneck is data: current resources are often limited in scale, diversity,and consistency of subject-level details, especially for unified support ofboth generation and manipulation. To address this gap, we introduceOpenSubject, a video-derived large-scale corpus for subject-driven gen-eration and manipulation. Our pipeline leverages cross-frame identitypriors through four key stages: (i) Video Curation. We apply resolutionand aesthetic filtering to obtain high-quality clips. (ii) Cross-FrameSubject Mining and Pairing. We utilize vision language model (VLM)-based category consensus, local grounding, and diversity-aware pairing toselect image pairs. (iii) Identity-Preserving Reference Image Syn-thesis. We introduce segmentation map-guided outpainting to synthesizeinput images for subject-driven generation and box-guided inpaintingto generate input images for subject-driven manipulation, together withgeometry-aware augmentations and irregular boundary erosion. (iv) Ver-ification and Captioning. We utilize a VLM to validate synthesizedsamples, re-synthesize failed samples based on stage (iii), and then con-struct short and long captions. Furthermore, we introduce a benchmarkcovering both subject-driven generation and subject-driven manipulation,evaluated with a VLM-based judge on identity fidelity, prompt adherence,and consistency. Extensive experiments demonstrate that training onOpenSubject significantly enhances performance in complex scenes.

ECCV 2026

SAMA: Factorized Semantic Anchoring and Motion Alignment for Instruction-Guided Video

Xinyao Zhang, Wenkai Dong, YuXin Song, Bo Fang, Qi Zhang, Jing Wang, Fan Chen, Hui Zhang, Haocheng Feng, Yu Lu, Hang Zhou, Chun Yuan, Jingdong Wang

Current instruction-guided video editing models struggle tosimultaneously balance precise semantic modifications with faithful mo-tion preservation. While existing approaches rely on injecting explicitexternal priors (e.g., VLM features or structural conditions) to mitigatethese issues, this reliance severely bottlenecks model robustness and gen-eralization. To overcome this limitation, we present SAMA (factorizedSemantic Anchoring and Motion Alignment), a framework that factor-ize video editing into semantic anchoring and motion modeling. First, weintroduce Semantic Anchoring which establish a reliable visual anchorby jointly predicting semantic tokens and video latents at sparse an-chor frames, enabling purely instruction-aware structural planning. Sec-ond, Motion Alignment pre-trains the same backbone on motion-centricvideo restoration pretext tasks (cube inpainting, speed perturbation, andtube shuffle), enabling the model to internalize temporal dynamics di-rectly from raw videos. SAMA is optimized with a two-stage pipeline: afactorized pre-training stage that learns inherent semantic-motion rep-resentations without paired video-instruction editing data, followed bysupervised fine-tuning on paired editing data. Remarkably, the factorizedpre-training alone already yields strong zero-shot video editing ability,validating the proposed factorization. SAMA achieves state-of-the-artperformance among open-source models and is competitive with leadingcommercial systems (e.g. Kling-Omni). Code, models, and datasets arereleased in https://cynthiazxy123.github.io/SAMA.

ECCV 2026

Rethinking Token Reduction for Diffusion Models via Output-Similarity-Awareness

Hangyeol Lee, Hyojeong Lee, Joo-Young Kim

Diffusion Transformers (DiTs) achieve superior image gener-ation quality but suffer from quadratic computational complexity rela-tive to token count. While various token reduction (TR) methods havebeen proposed to mitigate this cost, they overlook the primary objectiveof generative models: minimizing recovery error, which requires reflect-ing output token similarity. Specifically, they rely solely on input tokensimilarity inherited from reduction-only ViT paradigms, leading to a fun-damental misalignment with this objective.To bridge this gap, we propose DiTo, a novel TR paradigm that shiftsthe focus toward output-centric token reduction. Based on the observa-tion that output token similarity is consistently preserved across adjacenttimesteps, DiTo utilizes prior-step similarities as an effective proxy toestablish token correspondences at a Matching timestep, and reuses theresulting correspondences across multiple subsequent Reduction steps.To optimize this interleaved scheduling, we propose Pair Match Ra-tio (PMR)-guided Interval Scheduling to determine the optimal match-ing frequency. Furthermore, to mitigate localized approximation errorsand resulting blocking artifacts caused by repeated reuse, we proposeFrequency-aware Token Matching by incorporating a selection-frequencypenalty. Extensive experiments demonstrate that DiTo consistently out-performs existing TR methods with 1.6–3.9 dB higher PSNR at compa-rable speedups, achieving a superior Pareto frontier.

ECCV 2026

RegRet: Enhancing Region-Level Retrieval in Large Multimodal Models

Xun Liang, Honghui Yang, Weihang Pan, Boyuan Pan, Yao Hu, Binbin Lin, Deng Cai, Ruisi Zhao, Wenxiao Wang

Region-level retrieval aims to align user-specified image regions with relevantregions or textual descriptions, playing a crucial role in realworld applications such as e-commerce product search and RAG. Although recent Large Multimodal Models (LMMs)have made significant strides in multimodal retrieval, they primarily focus on global-level tasksand struggle to capture effective region-level representations. To bridge this gap, we presentRegRet, an LMM-based Region-level Retrieval framework that enhances the regional rep-resentations without compromising overall global retrieval performance. At its core, RegRetintegrates a Region-Aware Encoder to capture detailed regional features while balancing themwith the global background context. To further enhance the fine-grained understanding anddiscriminability of representations, we design a multi-stage training pipeline that includesdetailed localized captioning and regional contrastive learning tasks. In addition, consideringthe absence of region-level contrastive training data and the limited diversity of evaluationtasks in current benchmarks, we introduce the REGMB benchmark. It comprises 225k con-trastive pairs, covering four multimodal retrieval tasks. Extensive experiments validate theeffectiveness of our approach. RegRet outperforms strong baselines in the zero-shot setting.Further training with contrastive learning leads to an average improvement of more than 20%on both REGMB and public benchmarks, while achieving comparable or better results onglobal-level retrieval tasks. The code and data will be released for future research.

ECCV 2026

Proteus: Model Leakage-Induced Adversarial Attack in Federated Learning

Junjie Shan, Yue Zhang, Ziqi Zhao, Ka-Ho Chow

The massive volume and privacy-sensitive nature of visualdata have made federated learning (FL) a preferred paradigm for train-ing vision models across distributed data sources. However, during train-ing, FL repeatedly shares the in-progress model with randomly selectedparticipants. This paper investigates an overlooked yet practical threatarising from this sharing process: leaked intermediate models can be ex-ploited by adversaries to craft adversarial examples that compromise thefinal deployed model. Although directly using an intermediate model,especially one leaked early in training, as a surrogate yields only mod-erate attack gains, it can serve as an anchor for anticipating subsequenttraining dynamics. Based on this insight, we propose Proteus, a modelleakage-induced adversarial attack that leverages a leaked model to iden-tify vulnerabilities that persist throughout training, thereby generatingadversarial examples that remain effective against the final deployedmodel. For the first time, we show that models exposed well before con-vergence can already pose substantial risks to the final model, even if itundergoes hundreds of additional training rounds after leakage. Exten-sive experiments across diverse datasets, neural architectures, and FLconfigurations confirm the severity of this threat. Proteus exploits in-herent model leakage in FL and improves the attack success rate from59.75% when directly using the leaked model to 85.40%, even when leak-age occurs after only 30% of the total training process.

ECCV 2026

Posterior Samplings are Missing Modalities Generators for Medical Image Translation

Jonghun Kim

Magnetic resonance imaging comes in various modality con-trasts that provide complementary anatomical and pathological informa-tion. Complete multimodal acquisitions are often unavailable due to timeand protocol constraints. This leads to real-world datasets with missingmodalities, where conventional medical image translation methods aretypically limited to fixed source-target settings or require retraining foreach observed source-target pair. We propose a unified framework thatformulates missing-modality generation as a linear inverse problem un-der a joint distribution and solves it via posterior sampling with a flowmatching model. By learning a joint prior over the complete modalityset, our method can reconstruct arbitrary missing modalities at inferencetime by guiding the sampling trajectory to enforce measurement consis-tency with observed modalities. We further mitigate inter-modality er-ror propagation in multi-target generation by adopting a many-to-onesampling strategy. Experiments on BraTS and IXI datasets show thatour method achieves the best performance over baselines across mostmissing-modality scenarios. In downstream tumor segmentation, synthe-sized images from our method result in higher segmentation performance,indicating better preservation of clinically relevant structures. Our codeis available at github.com/jongdory/PS-MIT.

ECCV 2026

Posterior Augmented Flow Matching

George Stoica, Sayak Paul, Matthew Wallingford, Abhay Nori, Vivek Ramanujan, Winson Han, Ali Farhadi, Ranjay Krishna, Judy Hoffman

Flow matching (FM) trains a time-dependent vector field that transports samples from a simple prior to a complex data distribution. However, for high-dimensional images, each training sample supervises only a single trajectory and intermediate point, yielding an extremely sparse and high-variance training signal. This under-constrained supervision can cause flow collapse, where the learned dynamics memorize specific source–target pairings, mapping diverse inputs to overly similar outputs, failing to generalize. We introduce Posterior Augmented Flow Matching (PAFM), a theoretically grounded generalization of FM that replaces single-target supervision with an expectation over an approximate posterior of valid target completions for a given intermediate state and condition. PAFM factorizes this intractable posterior into (i) the likelihood of the intermediate under a hypothesized endpoint and (ii) the prior probability of that endpoint under the condition, and uses an importance sampling scheme to construct a mixture over multiple candidate targets. We prove that PAFM yields an unbiased estimator of the original FM objective while substantially reducing gradient variance during training by aggregating information from many plausible continuation trajectories per intermediate. Finally, we show that PAFM improves over FM by up to 3.4 FID50K across different model scales (SiT-B/2 and SiT-XL/2), different architectures (SiT and MMDiT), and in both class and text conditioned benchmarks (ImageNet and CC12M), with a negligible increase in the compute overhead. Code: https://github.com/gstoica27/PAFM.git.

ECCV 2026

Vinci2: Providing Proactive Assistance in Continuous Egocentric Videos

Sitong Gong, Tianyu Yan, Caixin Kang, Bo Zheng, Xiang Ruan, Huchuan Lu, Kaipeng Zhang, Yoichi Sato, Yifei Huang

When should an intelligent assistant speak up without beingasked? Continuous egocentric video offers rich, evolving context that en-ables a new form of assistance: one that is proactive rather than merelyreactive. Yet existing approaches either wait passively for user queries ortreat every detected event as requiring a response, without consideringthe user’s history, current activity, or whether assistance would actuallybe welcome. We reframe proactive assistance as a context-dependent de-cision problem: the agent must not only perceive what is happening,but reason over accumulated temporal context to determine when andwhether to intervene. To this end, we present Vinci2, a proactive ego-centric assistance system that advances the on-device assistant Vincifrom reactive response toward proactivity. On the evaluation side, wepresent EgoServe, the first large-scale benchmark for proactive assistancein continuous egocentric video. EgoServe comprises over 3,000 serviceinstances organized along 4 temporal memory horizons, ranging fromimmediate safety alerts to long-term habit coaching, across 10 servicecategories. On the modeling side, we propose EgoMemo, a training-free,memory-augmented agent that maintains three complementary mem-ory representations: multi-scale temporal summaries, a semantic knowl-edge graph, and visual embedding archives. At each timestep, EgoMemoperforms retrieval-augmented reasoning to determine whether assistanceis warranted and, if so, produces contextually grounded responses. Ex-periments demonstrate that EgoMemo establishes strong baselines onEgoServe while remaining competitive on existing egocentric benchmarks.Our benchmark and code are publicly available at Vinci2.

ECCV 2026

Optimization-Guided Diffusion for Interactive Scene Generation

Shihao Li, Naisheng Ye, Tianyu Li, Kashyap Chitta, Tuo An, Peng Su, Boyang Wang, Haiou Liu, Chen Lv, Hongyang Li

Realistic and diverse multi-agent driving scenes are crucial for evalu-ating autonomous vehicles, but safety-critical events which are essential for thistask are rare and underrepresented in driving datasets. Data-driven scene genera-tion offers a low-cost alternative by synthesizing complex traffic behaviors fromexisting driving logs. However, existing models often lack controllability or yieldsamples that violate physical or social constraints, limiting their usability. Wepresent OMEGA, an optimization-guided, training-free framework that enforcesstructural consistency and interaction awareness during diffusion-based samplingfrom a scene generation model. OMEGA re-anchors each reverse diffusion stepvia constrained optimization, steering the generation towards physically plausibleand behaviorally coherent trajectories. Building on this framework, we formu-late ego–attacker interactions as a game-theoretic optimization in the distributionspace, approximating Nash equilibria to generate realistic, safety-critical adver-sarial scenarios. Experiments on nuPlan and Waymo show that OMEGA improvesgeneration realism, consistency, and controllability, increasing the ratio of physi-cally and behaviorally valid scenes from 32.35% to 72.27% for free explorationcapabilities, and from 11% to 80% for controllability-focused generation. Our ap-proach can also generate 5× more near-collision frames with a time-to-collisionunder three seconds while maintaining the overall scene realism.

ECCV 2026

Objects as Audio-Visual Modal Sound Fields

Zisen Shao, Zihao Wei, Derong Jin, Ruohan Gao

While modern 3D reconstruction excels at modeling objectgeometry and appearance, it largely ignores the rich acoustic cues re-vealed through physical interaction. Object impact sounds convey mate-rial, stiffness, and structural properties that complement vision, yet exist-ing impact sound modeling approaches either rely on expensive physics-based simulation or require large datasets to generalize in a purely data-driven manner. We introduce Audio-Visual Modal Sound Field (AV-MSF), a novel object-level acoustic representation reconstructed frommulti-view images and only a few impact sound recordings. AV-MSFbuilds on 3D Gaussian Splatting integrated with dense 3D visual featureto provide a strong geometry-aware prior, and represents the impactsound field using compact, physically meaningful modal parameters, en-abling robust few-shot reconstruction. Experiments on two real-worlddatasets show that AV-MSF achieves state-of-the-art impact sound ren-dering, outperforming both physics-based and data-driven baselines. Fur-thermore, we demonstrate downstream applications enabled by our rep-resentation, including contact localization and object sound editing.

Keep browsing

More papers from top AI conferences

30+ top conferences, by conference, year, and topic.