paperlist.ai

AI conference papers

AI conference paper library

Search papers from 30+ top conferences, including NeurIPS, ICLR, ICML, CVPR, ACL and AAAI. Filter by keyword, then open any paper for its abstract, authors and links.

ECCV 2026

Ray-Path-Aware Virtual Point Removal on 2D Layer-Wise Nearest Point Map

DAEHYEON JEON, Kyungdon Joo, Jae-Young Sim

When LiDAR scans a scene with reflective materials, it of-ten produces erroneous virtual points, which resemble real structuresand degrade downstream 3D perception. Conventional methods rely onmulti-stage pipelines that combine 3D surface reconstruction with heuris-tic symmetry detection, making them highly sensitive to surface noise,reflection irregularities, and accumulated estimation errors across stages.To address this, we propose an end-to-end framework that reformu-lates virtual point removal as ray-aligned geometric reasoning in the 2Dequirectangular projection domain, eliminating the dependency on ex-plicit surface modeling. Specifically, we introduce a Layer-Wise NearestPoint Map, which organizes multi-return LiDAR points by layer corre-spondence rather than fixed distance intervals, preserving geometric con-tinuity between real and reflected surfaces. Building on this representa-tion, we propose a Ray-Path-Aware Attention mechanism that constrainsfeature aggregation along physically plausible reflection trajectories de-rived from surface normals and incident ray directions. This physicallyguided attention enables the network to learn reflection-induced symme-try between real and reflected regions without stage-wise surface estima-tion. Extensive experiments on real-world LiDAR datasets demonstratethat our method achieves more accurate and stable virtual point removalthan prior symmetry-based approaches.

ECCV 2026

Prune Once: Retraining-Free Task-Agnostic Pruning for Vision-Language Models

Kang MinSeok, Hyunwoo Kim, Chanyoung Kim, Minwoo Kim, Jaekoo Lee, Dahuin Jung

Vision-language models (VLMs) have achieved remarkable generalization across diverse multimodal tasks through large-scale pretraining, yet their rapidly increasing computational and memory requirements pose significant challenges for deployment in constrained environments. Existing pruning strategies often depend on task-specific criteria or LLM-oriented importance measures, making them unsuitable for task-agnostic pruning, where no task-specific samples are available at pruning time and the pruned model remains broadly applicable. We introduce a retraining-free VLM pruning framework called PORTA that derives a taskand modality-agnostic importance formulation based on activation variation, estimated from generic calibration data, which reliably captures feature-level representation utility across modalities. PORTA further incorporates an adaptive sparsity allocation mechanism that assigns layer-wise pruning ratios based on output feature variability, avoiding the limitations of uniform sparsity and reducing performance degradation at high compression levels. Extensive experiments across VLM architectures, such as CLIP, BLIP, and Qwen2-VL, demonstrate that PORTA achieves competitive downstream performance under high sparsity without requiring any retraining, supporting efficient VLM compression. Code is available at https://github.com/cau-hailab/PORTA.git.

ECCV 2026

In-Context Sync-LoRA for Portrait Video Editing

Sagi Polaczek, Or Patashnik, Ali Mahdavi-Amiri, Danny Cohen-Or

Editing portrait videos is a challenging task that requires flexible yet precise control over a wide range of modifications, such as appearance changes, expression edits, or the addition of objects. The key difficulty lies in preserving the subject’s original temporal behavior, demanding that every edited frame remains precisely synchronized with the corresponding source frame. We present Sync-LoRA, a method for editing portrait videos that achieves high-quality visual modifications while maintaining frame-accurate synchronization and identity consistency. Our approach uses an image-to-video diffusion model, where the edit is defined by modifying the first frame and then propagated to the entire sequence. To enable accurate synchronization, we train an in-context LoRA using paired videos that depict identical motion trajectories but differ in appearance. These pairs are automatically generated and curated through a synchronization-based filtering process that selects only the most temporally aligned examples for training. This training setup teaches the model to combine motion cues from the source video with the visual changes introduced in the edited first frame. Trained on a compact, highly curated set of synchronized human portraits, SyncLoRA generalizes to unseen identities and diverse edits (e.g., modifying appearance, adding objects, or changing backgrounds), robustly handling variations in pose and expression. Our results demonstrate high visual fidelity and strong temporal coherence, achieving a robust balance between edit fidelity and precise motion preservation. The project page is available at http://sagipolaczek.github.io/Sync-LoRA.

ECCV 2026

OmniCamera: A Unified Framework for Multi-task Video Generation with Arbitrary Camera Control

Yukun Wang, Ruihuang Li, Jiale Tao, Shiyuan Yang, Liyi Chen, Zhantao Yang, Handz Handz, Yulan Guo, Shuai Shao, Qinglin Lu

Video fundamentally intertwines two crucial axes: the dynamic content of a scene and the camera motion through which it is observed. However, existing generation models often entangle these factors, limiting independent control. In this work, we introduce OmniCamera, a unified framework designed to explicitly disentangle and command these two dimensions. This compositional approach enables flexible video generation by allowing arbitrary pairings of camera and content conditions, unlocking unprecedented creative control. To overcome the fundamental challenges of modality conflict and data scarcity inherent in such a system, we present two key innovations. First, we construct OmniCAM, a novel hybrid dataset combining curated real-world videos with synthetic data that provides diverse paired examples for robust multi-task learning. Second, we propose a Dual-level Curriculum Co-Training strategy that mitigates modality interference and synergistically learns from diverse data sources. This strategy operates on two levels: first, it progressively introduces control modalities by difficulties (condition-level), and second, trains for precise control on synthetic data before adapting to real data for photorealism (data-level). As a result, OmniCamera achieves state-of-the-art performance, enabling flexible control for complex camera movements while maintaining superior visual quality.

ECCV 2026

Where to Look Matters: Learning Influential Views for VLM-based 3D Visual Grounding

Tsung-Chih Chiang, Hsuan-Kung Yang, Jou-Min Liu, Ting-Ru Liu*, Chun-Wei Huang, Quan Kong, Chun-Yi Lee

Recent zero-shot 3D visual grounding methods leverage vision–language models (VLMs) to localize objects in 3D scenes from natural language queries. However, these methods typically rely on heuristic rules to select which camera views are provided to the VLM, often prioritizing object visibility rather than grounding relevance. We present IVSGround, a framework that learns Influential View Selection for VLM-based 3D visual grounding. Instead of using fixed heuristics, a lightweight view selector is trained to identify views that provide discriminative evidence for grounding. To obtain supervision signals, we generate training signals using feedback from a reasoning VLM through a two-stage rejection sampling process. During inference, the learned selector predicts query-conditioned influential views for each candidate object, which are then evaluated by a frozen reasoning VLM through comparative grounding. Experiments on ScanRefer and NR3D show that IVSGround consistently improves grounding accuracy over existing zeroshot pipelines, demonstrating that selecting where to look is crucial for effective 3D visual grounding.

ECCV 2026

Multi-Modal Controlled Coherent Motion Generation

Yifei Liu, Qiong Cao, Hongwei Yi, Huaiguang Jiang, Changxing Ding

It is natural for us to walk and talk simultaneously. This paper tackles the challenge of replicating such natural behaviors in 3D avatar motion generation driven by concurrent multi-modal inputs, e.g., a text description “a man is walking” alongside a speech audio. Existing methods, constrained by the scarcity of aligned multi-modal data, typically combine motions from individual modalities sequentially or through weighted sum. However, they often result in mismatched or unrealistic movements. To overcome these limitations, we propose MOCO, a novel diffusion-based framework capable of processing multiple simultaneous inputs—including speech audio, text descriptions, and trajectory data—to generate coherent and lifelike motions without requiring aligned multimodal data. Our key innovation lies in decoupling the motion generation process. During each denoising step, the diffusion model independently generates motions for each modality from the input noise and assembles the body parts according to predefined spatial rules. The resulting combined motion is then diffused and serves as the input noise for the subsequent denoising step. This iterative approach enables each modality to refine its contribution within the context of the overall motion, progressively harmonizing movements across modalities. Consequently, the generated motions become increasingly natural and fluid with each iteration, achieving coherent and synchronized behaviors. We evaluate our approach using a purpose-built multi-modal benchmark. Experimental results demonstrate that MOCO outperforms existing baselines, advancing the field of multi-modal motion generation for 3D avatars. The code will be released on https://feifeifeiliu.github.io/MOCO.

ECCV 2026

Learning Zero-Shot Subject-Driven Video Generation Using 1% Compute

Daneul Kim, Jingxu Zhang, Wonjoon Jin, Sunghyun Cho, Qi Dai, Jaesik Park, Chong Luo

Subject-driven video generation (SDV-Gen) aims to produce videos of a specific subject by adapting a pretrained video model, enabling personalized and application-driven content creation. To achieve this goal, per-subject tuning methods require approximately 200 A100 GPU hours to generate a customized video, whereas zero-shot methods avoid per-subject tuning but typically rely on millions of subjectvideo pairs for the supervision, incurring massive network fine-tuning costs (10K–200K A100 GPU hours). We propose a dataand computeefficient zero-shot SDV-Gen framework that avoids test-time per-subject tuning and the use of large-scale subject-video pairs. Our key idea decomposes SDV-Gen into (i) identity injection learned from subject-image pairs and (ii) motion-awareness preservation maintained by a small set of arbitrary videos. We optimize the two tasks with stochastic switching, using random reference-frame sampling and image-token dropout to prevent trivial first-frame copying. Our gradient analysis shows that the two objectives rapidly evolve toward nearly orthogonal update subspaces, explaining the stable optimization. Using CogVideoX-5B, we adapt a single model with 200K subject-image pairs and 4,000 arbitrary videos in 288 A100 GPU hours. This yields about 1% of compute compared to prior zero-shot baselines (i.e., 0.4% of VACE and 2.8% of Phantom) while using no subject-video pairs, yet remaining competitive in subject fidelity and motion quality. We show that the same recipe transfers to Wan 2.1-1.3B and Wan 2.2-5B.

ECCV 2026

Accelerating Text-to-Video Generation with Calibrated Sparse Attention

Shai Yehezkel, Shahar Yadin, Noam Elata, Yaron Ostrovsky-Berman, Bahjat Kawar

Recent diffusion models enable high-quality video generation, but suffer from slow runtimes. The large transformer-based backbones used in these models are bottlenecked by spatiotemporal attention. In this paper, we identify that a significant fraction of token-totoken connections consistently yield negligible scores across various inputs, and their patterns often repeat across queries. Thus, the attention computation in these cases can be skipped with little to no effect on the result. This observation continues to hold for connections among local token blocks. Motivated by this, we introduce CalibAtt, a training-free method that accelerates video generation via calibrated sparse attention. CalibAtt performs an offline calibration pass that identifies block-level sparsity and repetition patterns that are stable across inputs, and compiles these patterns into optimized attention operations for each layer, head, and diffusion timestep. At inference time, we compute the selected input-dependent connections densely, and skip the unselected ones in a hardware-efficient manner. Extensive experiments on Wan 2.1 14B, Mochi 1, and few-step distilled models at various resolutions show that CalibAtt achieves up to 1.58× end-to-end speedup, outperforming existing training-free methods while maintaining video generation quality and text-video alignment.

ECCV 2026

CritiqueDriveVLM: From Verifier-Guided Reinforcement Learning to Latent Thought Distillation for Autonomous Driving

Zhaohong Liu, Hao Ye, Xianlin Zhang, Mengshi Qi

End-to-end Vision-Language Models (VLMs) show immense potential in autonomous driving. However, standard Supervised FineTuning (SFT) often suffers from reasoning hallucinations and conservative biases. While traditional tool-augmented frameworks and Chain-ofThought (CoT) approaches mitigate these issues, they incur exorbitant token consumption and unacceptable latency, rendering real-time deployment impractical. To resolve this reliability-efficiency trade-off, we propose CritiqueDriveVLM, a novel unified three-stage framework internalizing reasoning directly into the VLM. First, we introduce Critique-Driven Multi-Turn Reinforcement Learning (RL) guided by a multi-dimensional verifier. By providing granular scalar feedback and a multi-turn penalty, we force the policy to internalize logical deduction, cultivating a robust System-2 Teacher that achieves high accuracy without fragile external tools. Subsequently, we propose Latent Thought Distillation to overcome the latency bottleneck. By aligning the Student’s latent representations with the Teacher’s fully converged reasoning states, we compress deep logical capabilities into a fast, CoT-free System-1 Student. Extensive experiments on the widely-used DriveLMM-o1 benchmark demonstrate remarkable improvements. Compared to the base model, our tool-free Teacher significantly boosts Multiple Choice Quality (MCQ) from 55.54% to a state-of-the-art 76.54%. Crucially, our distilled Student preserves competitive reasoning depth while drastically minimizing generation length to an average of merely 28 tokens. This slashes inference latency by 88% (from 3482 ms to 416 ms), paving a highly robust pathway for low-latency autonomous driving. Our source code is available at https://github.com/MICLAB-BUPT/CritiqueDriveVLM.

ECCV 2026

SCALE: Semantic-Calibrated Guidance Enhancement for Prompt-Faithful Diffusion

Tianhang Lu, Sudong Cai, Bingzhi Chen, Shao-Dong Shen, Chunting Liu, Longguang Wang, Bing Wang

Ensuring prompt faithfulness remains a central challenge for text-to-image diffusion models. Classifier-Free Guidance (CFG) improves prompt adherence but exhibits an inherent quality–alignment tension: increasing the guidance scale to strengthen conditioning on the prompt often degrades visual quality and introduces artifacts. To probe the limit of greedy alignment maximization, we first introduce SAP (Semantic Alignment Projection), a greedy update rule that projects each sampling update onto the guidance direction to maximize per-step alignment progress. We then show that SAP can fail due to the loss of orthogonal corrective freedom, discarding high-dimensional components that are crucial for rectifying accumulated trajectory drift. Based on this diagnosis, we propose SCALE (Semantic-CALibrated Guidance Enhancement), a drop-in, training-free guidance mechanism. SCALE selectively amplifies the semantic component along the guidance direction while preserving the orthogonal corrective component, improving semantic alignment without compromising structural fidelity. Across multiple text-to-image benchmarks, SCALE delivers substantial and consistent gains in prompt adherence and compositional alignment with negligible overhead over standard sampling. Project page: https://github.com/SudongCAI/SCALE.

ECCV 2026

Rolling Shutter Relative Pose Estimation Made Practical

Daniel Barath

Rolling shutter (RS) cameras equip virtually all consumerdevices, yet RS-aware relative pose estimation has remained impracti-cal: the state-of-the-art solver requires a minimum of 20 point corre-spondences, making RANSAC-based robust estimation prohibitively ex-pensive due to the exponential dependence of the iteration count onthe sample size. We make RS relative pose estimation practical by in-troducing affine correspondences (ACs) into the RS two-view geome-try. We derive novel RS-corrected affine constraints that account forthe coupling between point perturbations and the row-dependent es-sential matrix, providing two equations per correspondence beyond thestandard epipolar constraint. Building on these constraints, we developa linearized algebraic solver that estimates pose and RS motion fromonly 7 ACs. The solver exploits the physical smallness of RS parametersto linearize the constraints, eliminates the 12 RS unknowns via null-space projection, and solves the remaining degree-20 system via actionmatrices in 1.2 ms. On the TUM RS benchmark, our method achievesthe best pose and RS parameter accuracy among all tested methodsand, uniquely among RS solvers, provides accurate translational veloc-ity estimates – which are poorly conditioned from point correspondencesalone due to a v-t coupling. On the global-shutter EuRoC MAV dataset,the solver achieves comparable accuracy to the standard 5-point algo-rithm, demonstrating that it generalizes well to the GS setting. Code isat https://github.com/danini/rolling_shutter_made_practical.

ECCV 2026

VC-VAE: Leveraging Video Codecs for Training-Efficient and High-Fidelity Video VAE

Xinxu Ge, Shang Chai, Litong Gong, Zitong Yu, Xin Liu, Tiezheng Ge

Video Variational Auto-Encoders (Video VAEs) compress video data from the highly redundant pixel space into a compact latent representation, playing an important role in state-of-the-art video generation models. However, existing methods typically learn inter-frame correlations implicitly, overlooking the potential of breaking down video compression into two separate parts: keyframe encoding and inter-frame dynamic encoding, which is a fundamental design of traditional video codecs. To address this, we incorporate traditional video codec standard design into the Video VAE and introduce VC-VAE, a model that explicitly separates keyframe and inter-frame dynamic compression. We start by establishing a high-fidelity static keyframe anchor through initialization from a powerful pre-trained image VAE. Then, to explicitly model dynamic relative to this anchor, we introduce the Temporal Dynamic Difference Convolution (TDC), an operator designed to learn sparse motion residuals from inter-frame differences while maintaining a separate pathway for static content. Qualitative and quantitative experiments show that our proposed VC-VAE significantly outperforms baseline models in reconstruction quality, dynamic modelling, and training efficiency.

ECCV 2026

Intrinsically Stable Spiking Neural Networks: Overcoming the Performance Barrier in the Absence of Batch Normalization

Ruichen Ma, Xiaoyang Zhang, Jian Bai, Guanchao Qiao, Liwei Meng, Ning Ning, Yang Liu, Shaogang Hu

The performance of deep spiking neural networks (SNNs) often relies on batch normalization (BN). However, the advanced dynamic BN variants used in state-of-the-art models introduce runtime multiplications, which weaken the hardware-efficiency motivation of SNNs. To address this tension, we identify catastrophic firing-rate decay as a primary cause of severe performance degradation in normalization-free SNNs. Guided by this insight, this work proposes the Intrinsically Stable SNN (IS-SNN) architecture, which removes activation-normalization layers by enforcing signal homeostasis through topology-aware weight standardization and modified residual connections. By folding the standardization operations into static weights offline, IS-SNN removes the runtime statistics tracking and multiplications introduced by activation normalization, restoring an accumulation-oriented inference datapath. Comprehensive experiments show that IS-SNN achieves performance competitive with or superior to computationally expensive dynamic BN techniques across VGG, ResNet, and Transformer-based models. Notably, it achieves a competitive accuracy of 68.05% on ImageNet and overcomes the severe depth limitations of prior BN-free attempts. Together with a 96.4% reduction in FPGA lookup table resource consumption for neuron implementations, these results support IS-SNN as a practical framework for building accurate and hardware-friendly deep neuromorphic systems.

ECCV 2026

BATQuant: Outlier-Resilient MXFP4 Quantization via Learnable Block-wise Optimization

Jifu Li, Manyi Zhang, Xiaobo Xia, Han Bao, Haoli Bai, Zhenhua Dong, Xianzhi Yu

Microscaling floating-point (MXFP) formats have emerged as a promising standard for deploying Multi-modal Large Language Models (MLLMs) and Large Language Models (LLMs) on modern accelerator architectures. However, existing Post-Training Quantization (PTQ) methods, particularly rotation-based techniques designed for integer formats, suffer from severe performance collapse when applied to MXFP4. Recent studies attribute this failure to a fundamental format mismatch: global orthogonal rotations inadvertently transfer outlier energy across quantization blocks, inducing new outliers that disrupt local block-wise scaling, while often creating bimodal activation distributions that underutilize the limited quantization range. To address these issues, we propose BATQuant (Block-wise Affine Transformation), which restricts transformations to align with MXFP granularity to prevent cross-block outlier propagation, while relaxing orthogonality constraints to optimize distribution shaping. To ensure parameter efficiency, we introduce Global and Private Kronecker (GPK) decomposition to effectively reduces storage and runtime overhead and incorporate Block-wise Learnable Clipping to suppress residual outliers. Extensive experiments on both MLLMs and LLMs demonstrate that BATQuant establishes new state-of-theart results under aggressive W4A4KV16 configurations, recovering up to 96.43% of full-precision performance on multimodal benchmarks and clearly outperforming existing methods across diverse tasks.

ECCV 2026

A²-Edit: Precise Reference-Guided Image Editing of Arbitrary Objects and Ambiguous Masks

Guangzhao Li, Baixuan Zhao, Siqi Luo, Hantao Jiang, Guangtao Zhai, Xiaohong Liu

We propose A2-Edit, a unified inpainting framework for arbitrary object categories, which allows users to replace any target region with a reference object using only a coarse mask. To address the issues of severe homogenization and limited category coverage in existing datasets, we construct a large-scale, multi-category dataset UniEdit-500K, which includes 8 major categories, 209 fine-grained subcategories, and a total of 500,104 image pairs. Such rich category diversity poses new challenges for the model, requiring it to automatically learn semantic relationships and distinctions across categories. To this end, we introduce the Mixture of Transformer module, which performs differentiated modeling of various object categories through dynamic expert selection, and further enhances cross-category semantic transfer and generalization through collaboration among experts. In addition, we propose a Mask Annealing Training Strategy (MATS) that progressively relaxes mask precision during training, reducing the model’s reliance on accurate masks and improving robustness across diverse editing tasks. Extensive experiments on benchmarks such as VITON-HD and AnyInsertion demonstrate that A2-Edit consistently outperforms existing approaches across all metrics, providing a new and efficient solution for arbitrary object editing. The code is released at https://github.com/huayu-zheng/A2Edit.

ECCV 2026

MonoSR: Open-Vocabulary Spatial Reasoning on Monocular Images

Qirui Wang, Jingyi He, Yining Pan, Si Yong Yeo, Xulei Yang, Shijie Li

Spatial reasoning from a single RGB image, including the inference of distances, sizes, and inter-object relationships without depth sensors or multiple views, remains a critical blind spot for current Vision Language Models (VLMs). Existing benchmarks either rely on multiview video sequences that expose explicit geometric cues, or are confined to indoor environments too small for model training. To close this gap, we introduce MonoSR, a large-scale dataset for open-world monocular spatial reasoning comprising over 1M QA pairs from 230K images spanning indoor, outdoor, and object-centric domains across 98 semantic categories. Every QA pair is guaranteed answerable from a single RGB image via a four-stage observability filter validated by human audit. Tasks are organized into three cognitive levels: foundational perception, perspective-aware imagination, and situational reasoning, supporting both large-scale training and fine-grained evaluation. Comprehensive benchmarking of state-of-the-art openand closed-source VLMs reveals consistent limitations across all three domains, with object-centric metric estimation emerging as the most challenging frontier. We further conduct a systematic auxiliary information study by injecting scene context, 2D visual prompts, and 3D bounding boxes, allowing us to quantify the geometric gap that future monocular perception modules must close and provide actionable design guidelines. Our project page is available at this link.

ECCV 2026

IDeaL: Data-Free Multi-Teacher Distillation via Improved Dead Leaves

Feyza Yavuz, Mert Bulent Sariyildiz, Diane Larlus

Multi-teacher distillation has emerged as a way to combine complementary teacher models into a single student model that exhibits the strengths of all its teachers. The student is trained to mimic the output of the teachers on a set of images, typically the union of the individual teacher’s training sets, assuming this data is available. In this paper, we question that assumption and explore alternative options. We first study how far one can go when distilling from teachers fed with different types of noise. Then, we show that information contained in the teachers can be leveraged to tailor the noise for multi-teacher distillation: we propose a method that, thanks to decorrelation losses at both patch and image levels, generates teacher-specific, improved samples optimized for data-free distillation. Experiments show that our most effective samples, IDeaL, lead to strong students that successfully capture complementary information from the teachers, yielding surprisingly competitive results that substantially narrow the gap with students distilled from real images. Moreover, given a limited budget of 1K images for distillation, students distilled using our IDeaL samples match or surpass the performance of those distilled using a 1K-image subset of ImageNet.

ECCV 2026

YesTrack: Referring Multi-Object Tracking via MLLM-based Yes/No Reasoning

Quansheng Hu, Qin Sun, Qiansen Dai, Jin Ding, Wan Zhang, Xue Zhou, Jianxiao Zou

Referring multi-object tracking (RMOT) aims to track every instance in a video that matches a given language expression. Despite the recent integration of multimodal large language models (MLLMs) to enhance generalization, existing methods predominantly relegate them to the role of caption generators, necessitating external modules for final decision-making. This paradigm not only introduces extra latency but also severely underutilizes the inherent vision–language alignment capabilities of MLLMs. To address these limitations, we propose YesTrack, a novel two-stage RMOT method that reformulates referring as a discriminative task, directly leveraging MLLMs for Yes/No verification without explicit text generation. To further enhance the reliability and efficiency of this MLLM-based verification, we introduce two lightweight temporal consistency constraints: Temporal Confidence Prior (TCP) and Temporal Reference Propagation (TRP). We further validate the generality of this discriminative paradigm by proposing YesTrack-MOT, a straightforward yet highly effective instantiation for generic multiobject tracking (MOT). Experiments on Refer-KITTI and Refer-KITTIV2 show that YesTrack significantly outperforms existing state-of-theart methods while maintaining high efficiency, even when implemented with the smallest variant of Qwen3-VL. Code is released at https: //github.com/ggbondrighthere24/YesTrack.

ECCV 2026

Generative Action Tell-Tales: Assessing Human Motion in Synthesized Videos

Xavier Thomas, Youngsun Lim, Ananya Srinivasan, Audrey Zheng, Deepti Ghadiyaram

Despite rapid advances in video generative models, robust metrics for evaluating visual and temporal correctness of complex human actions remain elusive. Critically, existing pure-vision encoders and Multimodal Large Language Models (MLLMs) are strongly appearancebiased, lack temporal understanding, and thus struggle to discern intricate motion dynamics and anatomical implausibilities in generated videos. We tackle this gap by introducing a novel evaluation metric derived from a learned latent space of real-world human actions. Our method first captures the nuances, constraints, and temporal smoothness of real-world motion by fusing appearance-agnostic human skeletal geometry features with appearance-based features. We posit that this combined feature space provides a robust representation of action plausibility. Given a generated video, our metric quantifies its action quality by measuring the distance between its underlying representations and this learned real-world action distribution. For rigorous validation, we develop a new multi-faceted benchmark specifically designed to probe temporally challenging aspects of human action fidelity. Through extensive experiments, we show that our metric achieves substantial improvement of more than 68% compared to existing state-of-the-art methods on our benchmark, performs competitively on established external benchmarks, and has a stronger correlation with human perception. Our in-depth analysis reveals critical limitations in current video generative models and establishes a new standard for advanced research in video generation. Code is available at https://xthomasbu.github.io/video-gen-evals/

ECCV 2026

Spatiotemporal Flux Probing for Single-Photon Videography

Jerry Yan, Matteo Forlivesi, Bowen Tan, Andrew Xie, Siddharth Somasundaram, Sotiris Nousias

We address the problem of recovering high-speed videos from dynamic scenes under extreme photon sparsity. Existing methods rely on aggregating photon detections in local spatiotemporal windows to improve signal-to-noise ratio; however, this local grouping discards global structure and fails in low-light regimes where photon detections are sparse in space and time. In this work, we show that the information needed to recover both motion and illumination is encoded in correlations over the full space-time pattern of photon arrivals. Building on this insight, we develop a spatiotemporal flux probing theory and an algorithm that estimates the Fourier coefficients of the underlying intensity directly from the photon stream. We demonstrate that our approach (1) recovers fast motion and temporal illumination dynamics with substantially fewer photons than prior methods, (2) enables velocity-selective videography that automatically refocuses video onto specific detected motions, and (3) generalizes across sensing modalities including singlephoton, event, and spike cameras.

Keep browsing

More papers from top AI conferences

30+ top conferences, by conference, year, and topic.