paperlist.ai

AI conference papers

AI conference paper library

Search papers from 30+ top conferences, including NeurIPS, ICLR, ICML, CVPR, ACL and AAAI. Filter by keyword, then open any paper for its abstract, authors and links.

ECCV 2026

ARVAR: Accelerating Visual Autoregressive Model via Attention Retrospect

Jiedong Zhuang, Lu Lu, Ming Dai, Jian Chen, Qiang Liu, Haoji Hu

Visual autoregressive (VAR) models synthesize images in acoarse-to-fine manner by predicting multi-scale token maps, achievingcompetitive text-to-image quality with fewer decoding iterations thanraster-scan autoregression. However, late-scale refinement dominates in-ference time because computation grows quadratically with resolutionand attention becomes increasingly costly. Prior accelerations either fil-ter tokens heuristically, risking severe artifacts, or rely on extra deci-sion models that require per-model fine-tuning with reduced flexibility.We present the first in-depth study of attention behavior across VARgeneration steps and uncover two key properties: (1) attention patternscorresponding layers remain consistent across steps; (2) conditional andunconditional branches progressively converge at later scales. Leveragingthese insights, we introduce a novel training-free, flashattention-friendlyacceleration recipe ARVAR with two primary components: (1) Atten-tion Retrospect (AR) trims tokens at high resolution phase by to-ken saliency from early stage; (2) Branch Halving (BH) drops oneof the conditional/unconditional branches in the late steps. Combined,AR and BH deliver faster and higher-quality autoregressive image gen-eration, achieving 1.56× and 3.01× speedups on HART and Infinityrespectively with negligible performance drop. Compared with existingacceleration methods, ARVAR substantially improves image fidelity.

ECCV 2026

Are Video Reasoning Models Ready to Go Outside?

Yangfan He, Changgyu Boo, Jaehong Yoon

In real-world deployment, vision-language models often en-counter disturbances such as weather, occlusion, and camera motion.Under such conditions, their understanding and reasoning degrade sub-stantially, revealing a gap between clean, controlled (i.e., unperturbed)evaluation settings and real-world robustness. To address this limitation,we propose ROVA, a novel training framework that improves robustnessby modeling a robustness-aware consistency reward under spatio-temporalcorruptions. ROVA introduces a difficulty-aware online training strategythat prioritizes informative samples based on the model’s evolving capa-bility. Specifically, it continuously re-estimates sample difficulty via self-reflective evaluation, enabling adaptive training with a robustness-awareconsistency reward. We also introduce PVRBench, a new benchmark thatinjects real-world perturbations into embodied video datasets to assessboth accuracy and reasoning quality under realistic disturbances. Weevaluate ROVA and baselines on PVRBench, UrbanVideo, and VisBench,where open-source and proprietary models suffer up to 35% and 28%drops in accuracy and reasoning under realistic perturbations. ROVAeffectively mitigates performance degradation, boosting relative accuracyby at least 24% and reasoning by over 9% compared with baseline models(QWen2.5/3-VL, InternVL2.5, Embodied-R). These gains transfer to cleanstandard benchmarks, yielding consistent improvements.

ECCV 2026

A4-Agent: An Agentic Framework for Zero-Shot Affordance Reasoning

Zixin Zhang, Kanghao Chen, Hanqing Wang, Hongfei Zhang, Harold Haodong Chen, Chenfei Liao, Litao Guo, Yinchuan Li, Yingcong Chen

Affordance prediction, which identifies interaction regionson objects based on language instructions, is critical for embodied AI.Prevailing end-to-end models couple high-level reasoning and low-levelgrounding into a single monolithic pipeline and rely on training over an-notated datasets, which leads to poor generalization on novel objects andunseen environments. In this paper, we move beyond this paradigm byproposing A4-Agent, a training-free agentic framework that decouplesaffordance prediction into a three-stage pipeline. Our framework coordi-nates specialized foundation models at test time: (1) a Dreamer thatemploys generative models to visualize how an interaction would look;(2) a Thinker that utilizes large vision-language models to decide whatobject part to interact with; and (3) a Spotter that orchestrates vi-sion foundation models to precisely locate where the interaction area is.By leveraging the complementary strengths of pre-trained models with-out any task-specific fine-tuning, our zero-shot framework significantlyoutperforms state-of-the-art supervised methods across multiple bench-marks and demonstrates robust generalization to real-world settings.

ECCV 2026

2K Retrofit: Entropy-Guided Efficient Sparse Refinement for High-Resolution 3D Geometry Prediction

Tianbao Zhang, Zhenyu Liang, Zhenbo Song, Nana Wang, Xiaomei Zhang, Xudong Cai, Zheng Zhu, Kejian Wu, Gang Wang, Zhaoxin Fan

VLM-Parser object features purpose •••3-View VLM Parse ManualImages Stool AssemblyGeneralizingAssembly VLM-PlannersequenceInferVLMUpdateAssembly LampAttempts sequence Assembly(a) Digital Twins System (b) Parser and Planner (c) Zero-shot AssemblyFig. 1: Illustrating the core concept of ZAP: a VLM-guided agent transforms multi-view RGB-D observations into an assembly plan through internal reasoning, while adigital twin is used for geometry-based feasibility verification.

ECCV 2026

Minute4D: Training High-Fidelity 4D Gaussian Splatting in One Minute

Binjian Xie, Chenhui Shi, Pengju Zhang, Yihong Wu

Dynamic view synthesis has seen significant advances, yet re-constructing scenes from uncalibrated, casual videos remains challengingdue to ambiguity between camera and object motion, inadequate viewcoverage and slow optimization. In this work, we present Minute4D,a novel and efficient approach for high-fidelity 4D scene reconstructionfrom monocular videos using Gaussian Splatting. Our Gaussian initial-ization begins with an efficient geometric recovery leveraging pre-trainedvisual foundation models. To improve reconstruction fidelity, we intro-duce a Segmentation-Tracking Enhancement module that leverages 2Dsemantic priors and 3D point tracking to jointly enhance the geometricaccuracy and completeness of the initial Gaussians. To reduce redun-dancy and accelerate Gaussian optimization, we propose a Loss-GuidedDensity Control strategy that adaptively densifies and prunes Gaussiansbased on multi-view photometric loss. Our method reduces the optimiza-tion time to approximately 40 seconds for a typical 400-frame video,yielding a speed-up of at least 5×. Extensive experiments show thatour method not only enables substantially faster optimization but alsoachieves superior performance across several benchmarks.*: Equal contribution.†: Co-corresponding authors.

ECCV 2026

Your Data Manifold is Secretly a Reward Model: Shell-LCC for Text-to-Video Generation

Shihao Zhang, Yunzhi Li, Yuguang Yan, Junzhe Zhang, Wei Zhao, Bohan Wang, Hanwang Zhang

Recent text-to-video (T2V) diffusion models rely heavily on auxiliary rewardsignals (e.g., via reward models or DPO) to align generated content with human aesthetics andimprove realism. These signals, however, incur substantial computational overhead, requirecostly human annotations, and often yield limited improvement in fine-grained local details. Inthis paper, we argue that your data manifold is secretly a reward model. By explicitlymodeling the manifold structure of high-quality Supervised Fine-Tuning (SFT) data andencouraging video latents to lie on this manifold, we derive dense, differentiable, and nearlycost-free reward signals that significantly improve video quality, particularly in mitigating low-level distortions. Our modeling builds upon Local Coordinate Coding (LCC), which capturesthe ‘skeleton’ of the manifold. However, directly applying LCC suffers from mean regression,pulling latents toward the geometric mean and losing high-frequency details. We thereforeextend it to Shell Local Coordinate Coding (Shell-LCC), which models the manifold ‘surface’as an isotropic shell to align with the true high-density region. Experiments demonstratethat our approach improves realism, enhances high-frequency details, reduces over-smoothingartifacts, and alleviates motion blur.

ECCV 2026

VoCa: Unified Autoregressive Modeling for Talking Audio-Video Generation

Zhuofan Zong, Jiale Yuan, Yufei Liu, Dongzhi Jiang, Hao Shao, Zimu Lu, Ke Wang, Yunqiao Yang, Mingjie Zhan, Hongsheng Li

Talking-head generation from text typically relies on cas-caded pipelines that synthesize audio before video, leading to error ac-cumulation, latency, and audiovisual misalignment. Existing end-to-endmodels address some of these issues but often lack autoregressive capabil-ities. In this paper, we present VoCa, a unified autoregressive frameworkthat jointly generates speech and talking-head video from text transcriptsand a reference image. VoCa employs a large language model (LLM) tomap text into shared representations, which are concurrently processedby dedicated speech and video decoders to ensure strict synchronization.To bridge the semantic gap between audio-optimized features and hu-man motion, we introduce a speech refiner with a tailored multi-stagetraining strategy. Furthermore, to enable step-by-step autoregressive co-generation, we propose a unified window partitioning mechanism andan associated improved rolling-window denoising strategy. This adaptsa bidirectional diffusion transformer into a causal architecture synchro-nized with the LLM’s autoregressive generation windows. Extensive ex-periments show that VoCa matches or surpasses state-of-the-art baselinesin visual quality, improves audio–lip synchronization, and identity preser-vation. Code and models will be released upon acceptance.

ECCV 2026

ViTAL‑X: Video-Text Alignment with Cross‑Modal Temporal Edits

Sethuraman T V, Savya Khosla, Onkar Susladkar, Aditi Tiwari, Seoung Wug Oh, Kushal Kafle, Joon-Young Lee, Derek Hoiem, Simon Jenni

Video-text models adapted from image-text architectures(e.g., CLIP) frequently exhibit temporal blindness—the inability to per-ceive fundamental cues like order, direction, and motion dynamics. Stan-dard datasets mask this limitation by enabling models to exploit staticspatial shortcuts. To systematically evaluate this, we introduce XTE-Bench, a diagnostic probe revealing that even large-scale video-languagemodels struggle with basic temporal reasoning, indicating that param-eter scaling alone is insufficient to resolve this flaw. To address this,we propose Cross-Modal Temporal Edits (XTE), a self-supervisedframework that injects precise temporal supervision. By performing syn-chronized video-text transformations, XTE generates hard temporal neg-atives without manual annotation. We instantiate this with ViTAL-X,a lightweight model that equips frozen image-text backbones with tem-poral awareness while preserving their foundational spatial knowledge.Across six temporal benchmarks, ViTAL-X achieves state-of-the-art per-formance. Utilizing only 0.4B parameters and 1M training clips, ViTAL-X outperforms 7B-parameter models and surpasses baselines trained on600× more data. These results demonstrate that targeted, high-qualitytemporal alignment provides a highly efficient alternative to pure scaling.

ECCV 2026

ViewFusion: Structured Spatial Thinking Chains for Multi-View Reasoning

Xingjian Tao, Yiwei Wang, Yujun Cai, Yifan Song, Jing Tang

Multi-view spatial reasoning remains difficult for currentvision-language models. Even when multiple viewpoints are available,models often underutilize cross-view relations and instead rely on single-image shortcuts, leading to fragile performance on viewpoint transforma-tion and occlusion-sensitive cases. We present ViewFusion, a two-stageframework that explicitly separates cross-view spatial pre-alignment fromquestion answering. In the first stage, the model performs deliberatespatial pre-thinking to infer viewpoint relations and spatial transfor-mations across views, forming an intermediate workspace that goes be-yond a simple re-description. In the second stage, the model conductsquestion-driven reasoning conditioned on this workspace to producethe final prediction. We train ViewFusion with synthetic reasoningsupervision followed by reinforcement learning using GRPO, which im-proves answer correctness while stabilizing the intended two-stage gen-eration behavior. On MMSI-Bench, ViewFusion improves accuracyby 5.3% over Qwen3-VL-4B-Instruct, with the largest gains on exam-ples that require genuine cross-view alignment. Our code is available athttps://github.com/taoxj2001/ViewFusion.

ECCV 2026

VesselTok: Tokenizing Vessel-like 3D Biomedical Graph Representations for Reconstruction and Generation

Chinmay Prabhakar, Bastian Wittmann, Tamaz Amiranashvili, Paul Büschl, Ezequiel de la Rosa, Julian McGinnis, Benedikt Wiestler, Bjoern Menze, Suprosanna Shit

Spatial graphs provide a lightweight and elegant represen-tation of curvilinear anatomical structures such as blood vessels, lungairways, and neuronal networks. Accurately modeling these graphs iscrucial in clinical and (bio-)medical research. However, the high spatialresolution of large networks drastically increases their complexity, result-ing in significant computational challenges. In this work, we aim to tacklethese challenges by proposing VesselTok, a framework that approachesspatially dense graphs from a parametric shape perspective to learn la-tent representations (tokens). VesselTok leverages centerline points witha pseudo radius to effectively encode tubular geometry. Specifically, welearn a novel latent representation conditioned on centerline points to en-code neural implicit representations of vessel-like, tubular structures. Wedemonstrate VesselTok’s performance across diverse anatomies, includ-ing lung airways, lung vessels, and brain vessels, highlighting its ability torobustly encode complex topologies. To prove the effectiveness of Vessel-Tok’s learned latent representations, we show that they (i) generalize tounseen anatomies, (ii) support generative modeling of plausible anatom-ical graphs, and (iii) transfer effectively to downstream inverse problems,such as link prediction.

ECCV 2026

Unsafe by Reciprocity: How Generation–Understanding Coupling Undermines Safety in Unified Multimodal Models

Kaishen Wang, Heng Huang

Recent advances in Large Language Models (LLMs) and Text-to-Image (T2I) models have led to the emergence of Unified Multimodal Models (UMMs), where multimodal understanding and image generation are tightly integrated within a shared architecture. Prior studies suggest that such reciprocity enhances cross-functionality performance through shared representations and joint optimization. However, the safety implications of this tight coupling remain largely unexplored, as existing safety research predominantly analyzes understanding and generation functionality in isolation. In this work, we investigate whether cross-functionality reciprocity itself constitutes a structural source of vulnerability in UMMs. We propose RICE: Reciprocal Interaction–based Cross-functionality Exploitation, a novel attack paradigm that explicitly exploits bidirectional interactions between understanding and generation. Using this framework, we systematically evaluate Generation-toUnderstanding (G→U) and Understanding-to-Generation (U→G) attack pathways, demonstrating that unsafe intermediate signals can propagate across modalities and amplify safety risks. Extensive experiments show high Attack Success Rates (ASR) in both directions, revealing previously overlooked safety weaknesses inherent to UMMs. The code is available at https://github.com/tunantu/UMM-Safety.

ECCV 2026

SyntheticDoc: A Large Synthetic Dataset for Document Unwarping and Illumination Correction

Daniel Woortmann, Tanguy Magne, Olga Sorkine-Hornung

Deep learning models have become the standard tool for doc-ument rectification and illumination correction, yet their performance isfundamentally bound by their training data. For nearly a decade, the com-munity has heavily relied on Doc3D, a pioneering but increasingly limiteddocument unwarping dataset in terms of scale and quality. To address thisbottleneck, we introduce SyntheticDoc, a massive, high-quality datasetdesigned to push the boundaries of document unwarping. SyntheticDocis composed of 1,000,000 high-resolution procedurally generated trainingsamples, alongside extensive validation and test sets. Each sample ispaired with rich, pixel-perfect annotations, including UV maps, normalmaps, albedo and shading. To ensure physical accuracy and photorealism,the paper geometries are generated via a physics-based simulator andrendered using a path tracer. To demonstrate the benefit of our dataset,we train a simple baseline model on SyntheticDoc and report on itsperformance in comparison to state-of-the-art methods on both documentunwarping and illumination correction tasks. Our dataset is available athttps://igl.ethz.ch/projects/SyntheticDoc/ and the code used to generateit at https://github.com/tanguymagne/SyntheticDoc.

ECCV 2026

Stream3D-VLM: Online 3D Spatial Understanding with Incremental Geometry Priors

Hanxun Yu, Xuan Qu, Lei Ke, Boqiang Zhang, Yuxin Wang, Jianke Zhu, Dong Yu

Despite advances in 3D scene understanding, existing 3DLarge Multimodal Models operate in offline settings, requiring completescene observations or predefined video clips. In this paper, we presentan online 3D vision-language model that enables real-time spatial un-derstanding from streaming video. Our approach adopts an autoregres-sive streaming control modeling based on the LLM’s next-token predic-tion objective to learn when to respond, and employs a lightweight Vi-sual–Spatial Feature Integration (VSFI) module to incrementally injecttemporally aligned geometry priors into the visual stream. To alleviatelong-context decoding overhead, we propose a plug-and-play Geometry-Adaptive Voxel Compression (GAVC) module for efficient visual tokencompression. To address the scarcity of streaming 3D–language data,we further develop a scalable data generation pipeline that curates over1M online spatio-temporal 3D QA pairs and establishes a comprehensivebenchmark spanning 29 tasks. Extensive experiments show that our ap-proach significantly outperforms both proprietary and open-source mod-els across online and offline 3D spatial understanding, reasoning, andgrounding tasks.

ECCV 2026

Straight-Path Flow Matching for Incomplete Multi-View Clustering

Yiteng Yuan, Junyan Wang, Zheyuan Liu, Hong Jia, Lei Fan, Zhulin Tao, Lianbo Guo

Incomplete Multi-View Clustering addresses the problem ofclustering multi-modal data when certain views are missing. Recent end-to-end generative approaches leverage diffusion models to recover miss-ing views via stochastic noise-to-data trajectories. While expressive, suchmechanisms are not explicitly designed for clustering, as they initializefrom cluster-agnostic noise and rely on stochastic denoising dynamics.In this work, we revisit probability path design in end-to-end generativeIMVC. We introduce a flow-matching framework with a linear interpo-lation path between paired view representations, that replaces diffusionwith probability flows between observed and missing views. We providea formal analysis showing that deterministic ODE flows are inherentlybetter aligned with clustering objectives than diffusion-based stochastictrajectories, especially in terms of transport mechanisms that respectclass-conditional data distributions and maintain cluster consistency infinite-step regimes. Building upon this insight, we develop an end-to-end IMVC architecture that integrates straight-path flow-matching viewcompletion with cluster-level and entropy-based alignment to enforcecross-view clustering consistency. Extensive experiments on standardIMVC benchmarks demonstrate that the proposed framework establishesnew state-of-the-art performance.

ECCV 2026

StoryBlender: Inter-Shot Consistent and Editable 3D Storyboard with Spatial-temporal Dynamics

Bingliang Li, Zhenhong Sun, Jiaming Bian, Yuehao Wu, Yifu Wang, Hongdong Li, Yatao Bian, Huadong Mo, Daoyi Dong

Storyboarding is a core skill in visual storytelling for film,animation, and games. However, automating this process requires a sys-tem to achieve two properties that current approaches rarely satisfy si-multaneously: inter-shot consistency and explicit editability. While 2Ddiffusion-based generators produce vivid imagery, they often suffer fromidentity drift along with limited geometric control; conversely, traditional3D animation workflows are consistent and editable but require expert-heavy, labor-intensive authoring. We present StoryBlender, a grounded3D storyboard generation framework governed by a Story-centric Re-flection Scheme. At its core, we propose StoryBlender system, whichis built on a three-stage pipeline: (1) Semantic-Spatial Grounding, toconstruct a continuity memory graph to decouple global assets fromshot-specific variables for long-horizon consistency; (2) Canonical AssetMaterialization, to instantiate entities in a unified coordinate space tomaintain visual identity; and (3) Spatial-Temporal Dynamics, to achievelayout design and cinematic evolution through visual metrics. By or-chestrating multiple agents in a hierarchical manner within a verifica-tion loop, StoryBlender iteratively self-corrects spatial hallucinations viaengine-verified feedback. The resulting native 3D scenes support direct,precise editing of cameras and visual assets while preserving unwaver-ing multi-shot continuity. Experiments demonstrate that StoryBlendersignificantly improves consistency and editability over both diffusion-based and 3D-grounded baselines. Code and dataset are available athttps://engineeringai-lab.github.io/StoryBlender.

ECCV 2026

SpatiO: Adaptive Test-Time Orchestration of Vision-Language Agents for Spatial Reasoning

ChanYeong Hwang, Miso Choi, Sunghyun On, Jinkyu Kim, Jungbeom Lee

Understanding visual scenes requires not only recognizing objects but also reasoning about their spatial relationships. Unlike general vision-language tasks, spatial reasoning requires integrating multiple inductive biases, such as 2D appearance cues, depth signals, and geometric constraints, whose reliability varies across contexts. This motivates spatial adaptability: the ability to flexibly coordinate reasoning strategies depending on the input. However, most existing approaches rely on a single reasoning pipeline that implicitly learns a fixed spatial prior, limiting their ability to adapt under distribution changes. Multi-agent systems offer a promising alternative by aggregating diverse reasoning trajectories, but prior spatial reasoning attempts primarily employ homogeneous agents, restricting the inductive-bias diversity they can leverage. In this work, we introduce SpatiO, a heterogeneous multi-agent framework for spatial reasoning that coordinates multiple vision-language specialists with complementary inductive biases. To enable effective collaboration, we propose Test-Time Orchestration (TTO), a calibration mechanism that dynamically evaluates and reweights agents based on their observed reliability during inference, without modifying model parameters. Extensive experiments on diverse spatial reasoning benchmarks, including 3DSRBench, STVQA-7k, CV-Bench, and Omni3D-Bench, demonstrate that SpatiO consistently improves spatial reasoning performance over both closed-source and open-source baselines. The project page is available at https://cy-h1329.github.io/spatio/.

ECCV 2026

Small Vision-Language Models are Smart Compressors for Long Video Understanding

Junjie Fei, Jun Chen, Zechun Liu, Yunyang Xiong, Chong Zhou, Wei Wen, Junlin Han, Mingchen Zhuge, Saksham Suri, Qi Qian, Shuming Liu, Lemeng Wu, Raghuraman Krishnamoorthi, Vikas Chandra, Mohamed Elhoseiny, Chenchen Zhu

Adapting Multimodal Large Language Models (MLLMs) forhour-long videos is bottlenecked by context window limits. Existing effi-ciency heuristics blindly sacrifice fidelity. They frequently discard tran-sient decisive moments, blur fine-grained evidence, and waste represen-tational bandwidth on irrelevant backgrounds. We propose Tempo, anefficient, query-aware framework that compresses long videos for down-stream understanding. Tempo unifies a Small Vision-Language Model(SVLM) to act as a local temporal compressor. It casts visual token re-duction as an early cross-modal distillation process, generating compact,intent-aligned video representations in a single forward pass. To enforcestrict inference budgets without breaking temporal causality, we intro-duce Adaptive Token Allocation (ATA). Exploiting the SVLM’s inher-ent zero-shot relevance prior and empirical semantic front-loading, ATAacts as a training-free, O(1) dynamic router. It allocates dense band-width to query-critical segments while compressing redundancies downto minimal temporal anchors to maintain the global storyline. Exten-sive experiments demonstrate that our compact 6B architecture achieveshighly competitive performance with aggressive dynamic compression(0.5–16 tokens/frame). On the extreme-long LVBench (4101s), Temposcores 52.3 under a strict 8K visual budget, outperforming proprietarymodels like GPT-4o and Gemini 1.5 Pro. Crucially, empirical profiling re-veals that Tempo frequently compresses hour-long videos to token countsbelow theoretical limits, proving that true long-form understanding relieson intent-driven efficiency rather than greedily padded context windows.

ECCV 2026

RubricRL: Simple Generalizable Rewards for Text-to-Image Generation

Xuelu Feng, Yunsheng Li, Ziyu Wan, Zixuan Gao, Junsong Yuan, Dongdong Chen, Chunming Qiao

Reinforcement learning (RL) has recently emerged as a promis-ing approach for aligning text-to-image generative models with humanpreferences. A key challenge, however, lies in designing effective and in-terpretable rewards. Existing methods often rely on either compositemetrics (e.g., CLIP, OCR, and realism scores) with fixed weights or asingle scalar reward distilled from human preference models, which canlimit interpretability and flexibility. We propose RubricRL, a simple andgeneral framework for rubric-based reward design that offers greater in-terpretability, composability, and user control. Instead of using a black-box scalar signal, RubricRL dynamically constructs a structured rubricfor each prompt—a decomposable checklist of fine-grained visual cri-teria such as object correctness, attribute accuracy, OCR fidelity, andrealism—tailored to the input text. Each criterion is independently eval-uated by a multimodal judge (e.g., o4-mini), and a prompt-adaptiveweighting mechanism emphasizes the most relevant dimensions. This de-sign not only produces interpretable and modular supervision signalsfor policy optimization (e.g., GRPO or DiffusionNFT), but also enablesusers to directly adjust which aspects to reward or penalize. Experimentswith autoregressive and diffusion text-to-image models demonstrate thatRubricRL improves prompt faithfulness, visual detail, and generalizabil-ity, while offering a flexible and extensible foundation for interpretableRL alignment across text-to-image architectures.

ECCV 2026

RoMan-4D: Learning Robot Arm Manipulation from 4D World Models

Kaichen Zhou, Yuzhen Chen, Fangneng Zhan, Hang Hua, Grace Chen, Xinhai Chang, Ao Qu, Yilun Du, Zhuang Liu, Paul Pu Liang, Mengyu Wang

Video world models can generate realistic futures from a sin-gle instruction, but they often fail to track the same physical pointsconsistently across time. As a result, the generated videos appear plausi-ble, yet lack the physical grounding required for reliable action execution,such as robot manipulation. We present GEM-4D, a geometry-groundedvideo world model that resolves this limitation by injecting dense 4Dcorrespondence supervision distilled from a pretrained geometry foun-dation model into the video generative backbone during training. Thissupervision enables the video world model to jointly capture appearanceand geometric structure while retaining a single-stream architecture withno additional inference cost. We further introduce an inverse dynam-ics module that converts correspondence-consistent video rollouts intoexecutable robot trajectories, enabling direct deployment in both real-world and simulated manipulation. GEM-4D achieves state-of-the-artperformance on both video prediction and geometric consistency acrossboth simulation and realistic scenarios and improves real-world manipu-lation success from 61% to 81%. Additional results are available at thehttps://gem-4d.github.io/.

ECCV 2026

RoboGesture: Real-Time Semantic-aligned Co-Speech Gestures Generation for Humanoid Interaction

Zifan Wang, Ziang Ren, Pengyang Shi, Zirui Wang, Chenghuai Lin, Tianze Wang, Zekun Qi, Liangliang Zhao, He Wang, Li Yi

Enabling humanoid robots to respond to human speech withsynchronized and semantically meaningful gestures is fundamental tonatural human-robot interaction. However, this task faces three criticalbarriers: the scarcity of semantically rich datasets, the "modality eclipse"where models ignore audio cues in favor of kinematic inertia, and the sim-to-real gap regarding physical safety. We propose RoboGesture, a robot-centric framework that co-designs data, modeling, and control to powera complete interactive human–humanoid system in which the robot lis-tens, responds, and gestures in real time. We first establish the Robo-Gesture dataset featuring over 300 gesture categories and develop anautomated pipeline to synthesize large-scale collision-free, robot-specificaudio-motion pairs. Our architecture features a Hierarchical Semantic-Acoustic Aligner that extracts multi-granular prosodic and semantic cuesdirectly from raw audio tokens. These cues drive a Streaming ConditionalMotion Generator based on a diffusion transformer with conditional flowmatching. To ensure high responsiveness, we introduce Anti-Inertia CFGMasking, which prevents the model from collapsing into repetitive his-torical patterns by compelling it to proactively mine control signals fromthe audio modality. Finally, an MPC-based safety filter ensures real-time, collision-free execution on physical hardware. Experiments on aUnitree G1 humanoid demonstrate that RoboGesture generates safer,more rhythmic, and more semantically appropriate responses comparedto state-of-the-art baselines.

Keep browsing

More papers from top AI conferences

30+ top conferences, by conference, year, and topic.