paperlist.ai

AI conference papers

AI conference paper library

Search papers from 30+ top conferences, including NeurIPS, ICLR, ICML, CVPR, ACL and AAAI. Filter by keyword, then open any paper for its abstract, authors and links.

ECCV 2026

Taming Camera-Controlled Video Generation with Verifiable Geometry Reward

Zhaoqing Wang, Xiaobo Xia, Zhuolin Bie, Jinlin Liu, Dongdong Yu, JiaWang Bian, Changhu Wang

Recent advances in video diffusion models have remarkably improved camera-controlled video generation, but most methods rely solely on supervised fine-tuning (SFT), leaving online reinforcement learning (RL) post-training largely underexplored. In this work, we introduce an online RL post-training framework that optimizes a pretrained video generator for precise camera control. To make RL effective in this setting, we design a verifiable geometry reward that delivers dense segment-level feedback to guide model optimization. Specifically, we estimate the 3D camera trajectories for both generated and reference videos, divide each trajectory into short segments, and compute segment-wise relative poses. The reward function then compares each generated-reference segment pair and assigns an alignment score as the reward signal, which helps alleviate reward sparsity and improve optimization efficiency. Moreover, we construct a comprehensive dataset featuring diverse large-amplitude camera motions and scenes with varied subject dynamics. Extensive experiments show that our online RL post-training clearly outperforms SFT baselines across multiple aspects, including camera-control accuracy, geometric consistency, and visual quality, demonstrating its superiority in advancing camera-controlled video generation.

ECCV 2026

SPARC: Single-Pass Scaling for Motion Forecasting with Conformal Bayesian Last Layers

Sakif Hossain, Julian Teusch, Jörg Müller

Human motion forecasters are increasingly accurate and fast, but reliable deployment requires uncertainty estimates that are structured, calibrated, and efficient. Bayesian and ensemble-based uncertainty estimates often require repeated stochastic inference [15,26], while conformal calibration alone does not provide an epistemic signal or preserve trajectory covariance structure [14,50]. We introduce SPARC (SinglePass Adaptive Risk Calibration), a Bayesian–conformal uncertainty layer for motion forecasting. A deterministic MLP backbone predicts the future mean, and a conjugate Bayesian last layer converts time-domain feature leverage into an analytic horizon-wise epistemic scale κ (x). This t scale inflates a graph-temporal Gaussian covariance without changing its correlation structure, and split conformal calibration produces 95% marginal prediction tubes with finite-sample validity under exchangeability. The key interface is the structured factorization κ (x)Σ (x), t str,t which injects feature-space epistemic uncertainty into trajectory densities without Monte Carlo sampling. Across nine dataset/protocol blocks and deterministic, multimodal, and calibration baselines, SPARC ranks first on NLL and on the combined MPJPE+NLL criterion while retaining competitive point accuracy and efficient calibrated tubes. Ranking windows by κ separates high-error cases, making the scale usable as a lightweight risk monitor.

ECCV 2026

Warp-free Cross-view Geo-localization via Feature-space Consensus Mining

Zhuo Song, Lian Xu, Runqing Jiang, Kunhong Li, Yongjian Zhang, Ye Zhang, Yulan Guo

Cross-view geo-localization is challenging due to drastic viewpoint changes and large appearance discrepancies between street-level and satellite imagery. Although existing methods often use geometric warping to expose co-visible cues, such transformations rely on restrictive spatial assumptions and inevitably introduce severe visual distortions under view-dependent visibility, yielding noisy supervision and fragile correspondences. To overcome this, we propose a novel joint-view consensus-guided learning framework that entirely bypasses explicit geometric warping. Instead of forcing rigid spatial alignment, we dynamically mine and adaptively strengthen a semantic consensus directly within the feature space. Specifically, an auxiliary joint-view pathway during training enables direct cross-view interaction, allowing each view to selectively aggregate corroborative evidence into a unified consensus representation. To resolve feature heterogeneity among the singleand joint-view streams, we introduce global pattern probes acting as a semantic dictionary to project divergent modalities into a strictly aligned metric space. Guided by a consensus-mediated contrastive objective, single-view embeddings are explicitly pulled toward the joint-view anchor during training, distilling this consensus-mining capability into the single-view encoders for robust retrieval at inference. Extensive experiments demonstrate that our method achieves state-of-the-art performance across four standard benchmarks, underscoring the importance of discovering cross-view semantic consensus for reliable geo-localization.

ECCV 2026

3D FaceShell: Attribute Transfer in 3D Face Avatars as a VLM Defense Mechanism

Weston Bondurant, Srijan Das, Hieu Le, Stephanie Schuckers

Photorealistic 3D face avatars are increasingly deployed asreusable digital assets across applications such as telepresence, anima-tion, and personalized media. At the same time, vision–language models(VLMs) can infer sensitive attributes from rendered images with open-ended semantic reasoning without any fine-tuning. This creates a newprivacy challenge: once a 3D face avatar is shared, any of its renderingscan be analyzed to extract high-level facial attributes. Existing defenseslargely operate in 2D image space and do not address identity-preservingsemantic manipulation of 3D facial representations.We propose 3D FaceShell, a framework for steering VLM interpreta-tions of faces rendered from 3D models while preserving geometric fidelityand facial identity. 3D FaceShell augments the original 3D representa-tion with a learnable Gaussian shell that produces subtle, spatially dis-tributed perturbations optimized through multi-view embedding align-ment. The perturbations are designed to be visually inconspicuous yetsufficient to redirect VLM-based attribute inference in a view-consistentExtensive experiments on reconstructed celebrity face avatars and mul-tiple black-box VLMs demonstrate that 3D FaceShell significantly in-creases attribute injection and mismatch rates while maintaining highperceptual similarity and identity consistency. Our results show that itis possible to manipulate VLM-level semantic interpretation of 3D faceswithout compromising their human-recognizable appearance.

ECCV 2026

What VGGT Knows About Overlap: Probing Geometric Foundation Models for Co-Visibility

Filippo Ziliotto, Luciano Serafini, Lamberto Ballan, Tommaso Campari

A fundamental challenge in 3D reconstruction and robotic localization is co-visibility: determining which image pairs share overlapping visible surfaces, particularly in scenarios with minimal overlap. We demonstrate that VGGT implicitly encodes co-visibility as an emergent behavior: without any supervision for this task, its internal representations exhibit a clear hierarchical structure mirroring that of large language models — early layers build a 3D-aware scene representation, while late layers act as dedicated co-visibility reasoners. In particular, we identify layer L17 as a negative anchor that consistently routes nonco-visible pairs for this backbone, regardless of the evaluation setting, providing task-grounded evidence of layer specialization in a geometrygrounded foundation model. Building on this, we introduce Co-VGGT, which freezes VGGT and trains only a lightweight layer-wise mixtureof-experts head (∼7.5M parameters) to classify co-visibility from RGB alone, treating each layer as a specialized expert whose geometric abstraction is adaptively weighted per input pair. On the Co-VisiON benchmark, Co-VGGT surpasses the human annotation baseline and improves over prior work by more than 25% pairwise and 10% multiview. Pairwise predictions are well-calibrated (ECE=0.030), enabling direct use as edge weights in visibility graphs for downstream SfM and SLAM pipelines without post-hoc correction. Code and data available1.

ECCV 2026

ID-PreFeR: ID-Preserving Face Restoration with Mixed Data Quality

Chengxuan Zhu, Yuchen Hong, Qingnan Fan, Qi Zhang, Bingtao Fu, Jinxiu Liang, Jinwei Chen, Huaqi Zhang, Chao Xu, Boxin Shi

This paper introduces ID-PreFeR, a robust identity-preservingface restoration method that tackles the ill-posed face restoration prob-lem by incorporating personalized identity information. Existing approachesoften suffer from high training and storage costs and are sensitive tothe quality of reference images. To address these issues, we propose alightweight personalization injector that enables efficient personalizationwithout the need for regularization data. We also introduce an iden-tity–quality disentanglement training strategy to ensure robust identitylearning, even when some reference images are of poor quality. Further-more, we propose an identity-preserving sampling strategy to enhanceidentity fidelity during inference. Extensive experiments on both syn-thetic data and a newly collected real-world mobile-phone dataset verifythe effectiveness and practicality of the proposed method.

ECCV 2026

Prefill-Time Interventions against Adversarial Attacks on Large Vision-Language Models

Zhewen Yao, Yao Zhu, Shiliang Zhang

Recent studies have revealed critical vulnerabilities in Large Vision-Language Models (LVLMs) to sophisticated adversarial attacks that induce the model into generating adversary-specific content. However, existing LVLM-oriented detectors largely rely on external judging pipelines that incur substantial computational overhead, precluding their use in real-time scenarios. Towards this end, we propose PTI (PrefillTime Intervention), a method that probes an LVLM’s internal states to detect adversarial images before any response token is generated. PTI combines two complementary modules trained only on benign data: (i) a masked reconstruction module in the vision tower that identifies visual anomalies through token-wise reconstruction errors, and (ii) a dynamic modeling module in the language model backbone that tracks layer-wise trajectories of salient tokens to detect semantic inconsistencies. We establish a comprehensive evaluation framework covering both black-box and white-box attacks, including L -bounded, unrestricted, p and patch-based perturbations. Extensive experiments demonstrate that PTI achieves state-of-the-art detection performance and robust generalization across diverse threat models. Crucially, its minimal inference overhead paves the way for secure, real-time LVLM applications.

ECCV 2026

Poppy: Polarization-Based Plug-and-Play Guidance for Enhancing Surface Normal Estimation

Irene Kim, Sai Tanmay Reddy Chakkera, Alexandros Graikos, Dimitris Samaras, Akshat Dave

Monocular surface normal estimators trained on large-scale RGB-normal data often perform poorly in the edge cases of reflective, textureless, and dark surfaces. Polarization encodes surface orientation independently of texture and albedo, offering a physics-based complement for these cases. Existing polarization methods, however, require multi-view capture or specialized training data, limiting generalization. We introduce Poppy, a training-free framework that refines normals from any frozen RGB backbone using single-shot polarization measurements at test time. Keeping backbone weights frozen, Poppy optimizes perpixel offsets to the input RGB and output normal along with a learned reflectance decomposition. A differentiable rendering layer converts the refined normals into polarization predictions and penalizes mismatches with the observed signal. Across seven benchmarks and three backbone architectures (diffusion, flow, and feed-forward), Poppy reduces mean angular error by 23–26% on synthetic data and 6–16% on real data. These results show that guiding learned RGB-based normal estimators with polarization cues at test time refines normals on challenging surfaces without retraining. An interactive demo and code are available at https://irnkim.github.io/poppy/.

ECCV 2026

Discrete Noise Inversion for Next-scale Autoregressive Text-based Image Editing

Minh-Quan Dao, Xiaoxiao He, Ligong Han, Ngan Nguyen, Amin Nobari, Han Zhang, Faez Ahmed, Viet Anh Nguyen, Dimitris N. Metaxas

Visual autoregressive (VAR) models have recently emerged as a promising alternative to diffusion models, achieving competitive performance in text-to-image generation while offering substantially faster inference. Although conditional image generation has been widely studied, training-free prompt-guided image editing remains largely unexplored despite its importance for practical applications. In this paper, we present Visual AutoRegressive Inverse Noise (VARIN), the first noise inversion framework for training-free text-based image editing in visual autoregressive models. VARIN introduces Location-aware Argmax Inversion (LAI), a pseudo-inverse function for the argmax operator that reconstructs the inverse Gumbel noise used during discrete autoregressive sampling. The recovered inverse noise enables accurate reconstruction of the source image while providing controllable guidance for promptdriven image editing. Extensive experiments demonstrate that VARIN produces edits that closely follow the target prompts while effectively preserving the background and structural details of the source image. These results establish VARIN as a simple, efficient, and practical image editing framework for visual autoregressive models.

ECCV 2026

Semantic-Aware, Physics-Informed, Geometry-Grounded Weather Synthesis

Chenghao Qian, Nedko Savov, Lingdong Kong, Yeying Jin, Rui Song, Wenjing Li, Zhun Zhong, Jiaqi Ma, Gustav Markkula, Luc Van Gool

Weather synthesis aims to add weather effects to input videos while preserving scene identity, structure, and motion. The key limitation of existing methods is the lack of diversity in weather appearance and effective control over weather dynamics (e.g., temporal evolution and particle motion). Most approaches rely on text prompts, which are inherently underspecified and often fail to produce detailed weather characteristics. Additionally, general-purpose video editors optimized for clean and aesthetic outputs tend to suppress heavy weather phenomena, making dense particle effects difficult to generate. To address these, we propose a Semantic-Aware, Physics-Informed, and Geometry-Grounded framework that steers an off-the-shelf video editor to synthesize diverse global appearances and detailed particle dynamics. We factorize the synthesis into three conditional signals, so that each provides a distinct and stable source of guidance: semantics specifies what the weather should look like, dynamics governs how it evolves over time, and geometry determines where it should appear in the scene. Specifically, we introduce (1) semantic-aware appearance anchoring to establish the target appearance from scene semantics and user input; (2) physicsinformed dynamic simulation to generate particle effects by simulating a Gaussian-represented particle field under gravity, wind, and turbulence; and (3) geometry-grounded video synthesis to align the simulated particles with target scene geometry and synthesize the final video. Experiments demonstrate that our method produces diverse, physically and visually realistic weather effects. Furthermore, we show that our synthesized data significantly improves the robustness of autonomous driving semantic segmentation under adverse weather conditions. See our project page at https://jumponthemoon.github.io/w-crafter/.

ECCV 2026

Real5-OmniDocBench: A Full-Scale Physical Reconstruction Benchmark for Robust Document Parsing in the Wild

Cheng Cui, Changda Zhou, Tingquan Gao, Xueqing Wang, Ziyue Gao, Jing Tang, Yi Liu

While Vision-Language Models (VLMs) achieve near-perfect scores on digital document benchmarks like OmniDocBench, their performance in the unpredictable physical world remains largely unknown due to the lack of controlled yet realistic evaluations. We introduce Real5- OmniDocBench, the first benchmark to provide a full-scale, one-to-one physical reconstruction of the complete OmniDocBench v1.5 test set (1,355 images) across five critical real-world scenarios: Scanning, Warping, Screen-Photography, Illumination, and Skew. Unlike prior benchmarks that either lack digital correspondence or employ partial sampling, our complete ground-truth mapping enables, for the first time, rigorous factor-wise attribution of performance degradation, allowing us to pinpoint whether failures stem from geometric distortions, optical artifacts, or model limitations. Our benchmark establishes a challenging new standard for the community, demonstrating that the “reality gap” in document parsing is far from closed, and provides a diagnostic tool to guide the development of truly resilient document intelligence. The Real5-OmniDocBench dataset is publicly available at https:// huggingface.co/datasets/PaddlePaddle/Real5-OmniDocBench.

ECCV 2026

RAG-3DSG: Enhancing 3D Scene Graphs with Re-Shot Guided Retrieval-Augmented Generation

Yue Chang, Rufeng Chen, Zhaofan Zhang, Yi Chen, Yifan Tian, Sihong Xie

Open-vocabulary 3D Scene Graph (3DSG) can enhance various downstream tasks in robotics by leveraging structured semantic representations, yet current 3DSG construction methods suffer from semantic inconsistencies caused by noisy cross-image aggregation under occlusions and constrained viewpoints. To mitigate the impact of such inconsistency, we propose RAG-3DSG, which introduces re-shot guided uncertainty estimation. By measuring the semantic consistency between original limited viewpoints and re-shot optimal viewpoints, this method quantifies the underlying semantic ambiguity of each graph object. Based on this quantification, we devise an Object-level Retrieval-Augmented Generation (RAG) that leverages low-uncertainty objects as semantic anchors to retrieve more reliable contextual knowledge, enabling a VisionLanguage Model to rectify the predictions of uncertain objects and optimize the final 3DSG. Extensive evaluations across three challenging benchmarks and real-world robot trials demonstrate that RAG-3DSG achieves superior recall and precision, effectively mitigating semantic noise to provide more reliable scene representations for robotics tasks.

ECCV 2026

Edit in 2D, Verify in 3D: Reinforcement Learning for Multi-view Consistent Scene Editing

JiYuan Wang, Chunyu Lin, Lei Sun, Zhi Cao, Yuyang Yin, Lang Nie, Zhenlong Yuan, Xiangxiang Chu, Yunchao Wei, Kang Liao, Guosheng Lin

Leveraging the priors of 2D diffusion models for 3D editing has emerged as a promising paradigm. However, maintaining multiview consistency in edited results remains challenging, and the extreme scarcity of 3D-consistent editing paired data renders supervised finetuning (SFT), the most effective training strategy for editing tasks, infeasible. In this paper, we observe that, while generating multi-view consistent 3D content is highly challenging, verifying 3D consistency is tractable, naturally positioning reinforcement learning (RL) as a feasible solution. Motivated by this, we propose RL3DEdit, a single-pass framework driven by RL optimization with novel rewards derived from the 3D foundation model, VGGT. Specifically, we leverage VGGT’s robust priors learned from massive real-world data, feed the edited images, and utilize the output confidence maps and pose estimation errors as reward signals, effectively anchoring the 2D editing priors onto a 3D-consistent manifold via RL. Extensive experiments demonstrate that RL3DEdit achieves stable multi-view consistency and outperforms state-of-the-art methods in editing quality with high efficiency. To promote the development of 3D editing, we will release the code and model.

ECCV 2026

ECHO: Efficient Chest X-ray Report Generation with One-step Block Diffusion

lifeng chen, tianqi you, Hao Liu, Zhimin Bao, Jile Jiao, Xiao Han, Zhicai Ou, Tao Sun, Mou XiaoFeng, Xiaojie Jin, Yi Xu

Chest X-ray report generation (CXR-RG) has the potential to substantially alleviate radiologists’ workload. However, conventional autoregressive vision–language models (VLMs) suffer from high inference latency due to sequential token decoding. Diffusion-based models offer a promising alternative through parallel generation, but they still require multiple denoising iterations. Compressing multi-step denoising to a single step could further reduce latency, but often degrades textual coherence due to the mean-field bias introduced by token-factorized denoisers. To address this challenge, we propose ECHO, an efficient diffusion-based VLM (dVLM) for chest X-ray report generation. ECHO enables stable one-step-per-block inference via a novel Direct Conditional Distillation (DCD) framework, which mitigates the mean-field limitation by constructing unfactorized supervision from on-policy diffusion trajectories to encode joint token dependencies. In addition, we introduce a Response-Asymmetric Diffusion (RAD) training strategy that further improves training efficiency while maintaining model effectiveness. Extensive experiments demonstrate that ECHO surpasses state-of-the-art autoregressive methods, improving RaTE and SemScore by 64.33% and 60.58%, respectively, while achieving up to an 8× inference speedup with negligible degradation in clinical accuracy. Code, data, and models are publicly available at https://echo-midea-airc.github.io/.

ECCV 2026

From Hallucination to Grounding: Diagnosing Visual Spatial Intelligence via CRISP

Zhixing Li, Yinan Yu

Current VLM evaluations often conflate language priors with genuine spatial reasoning. To address this, we introduce CRISP, a novel structural-diagnostic evaluation paradigm that assesses visual spatial intelligence through consistency, the alignment between implicit perception and explicit reasoning. Unlike traditional black-box QA, CRISP utilizes metric 3D Scene Graphs and an oracle intervention protocol to decouple latent reasoning capabilities from perceptual bottlenecks. This granular diagnosis uncovers a systematic perception-reasoning disconnect. Crucially, we reveal that while proprietary models possess robust latent reasoning engines, they suffer from inaccurate metric estimation and a critical failure to leverage their implicit structural representations. Conversely, open-source models remain fundamentally bottlenecked by their lack of multi-hop compositional reasoning. By shifting the focus from merely “guessing correctly” via language priors to genuinely “perceiving, verifying, and reasoning,” CRISP offers a rigorous roadmap for multimodal alignment beyond end-to-end post-training. The code and dataset are available at https://github.com/iiyamayuki/CRISP-Bench.

ECCV 2026

Blind to Position, Biased in Language: Probing Mid-Layer Representational Bias in Vision-Language Encoders for Zero-Shot Language-Grounded Spatial Understanding

Na Min An, Inha Kang, Minhyun Lee, Hyunjung Shim

Vision–Language Encoders (VLEs) are widely adopted as the backbone of zero-shot referring image segmentation (RIS), enabling textguided localization without task-specific training. However, prior works underexplored the underlying biases within mid-layer representations that preserve positional and language-specific information. Through layerwise investigation, we reveal that the conventionally used final-layer multimodal embeddings prioritize global semantic alignment, leading to two coupled consequences. First, vision embeddings exhibit weak sensitivity to positional cues. Second, multilingual text embeddings form language-dependent geometric shifts within the shared space. Motivated by these findings, we identify an underexplored pathway within VLE mid-layers to construct a spatial map, applicable for improving zero-shot RIS by 1–7 mIoU on nine RefCOCO benchmarks. Furthermore, leveraging mixed-language mid-layer embeddings yields enhanced spatial grounding accuracy (+7–8 mIoU and IoU@50), albeit with increased inference cost, and also improves performance on the zero-shot text-to-image retrieval task. Our work opens up the discussion about the effects of effective representational bias probing of VLEs for enhanced spatial grounding. The code is available at https://github.com/kaist-cvml/Biased2Grounded.

ECCV 2026

Don’t Settle at the Mode! Mitigating Diversity Collapse in Pretrained Flow Models via Feature Self-Guidance

Pradhaan S Bhat, Rishubh Parihar, Abhijnya Bhat, Venkatesh Babu Radhakrishnan

State-of-the-art flow models generate stunning images from text or image prompts. However, they suffer from diversity collapse when generating multiple samples under the same conditioning. Existing methods address this issue via either latent guidance, which has limited effectiveness, or sample selection, which relies on external reward models that incur significant inference-time overhead. In this work, we introduce an efficient, training-free self-guidance mechanism to mitigate diversity collapse without requiring additional reward models. Specifically, we disperse the internal features of the flow model during batch generation with feature self-guidance. Further, to keep the features close to the manifold, we introduce a manifold regularization step that projects these dispersed features back onto the data manifold, ensuring diverse generation without sacrificing alignment with the input conditions. Our method integrates seamlessly as a plug-and-play module into pretrained flow models, adding only a marginal inference cost. Experiments demonstrate significant improvements in diversity while preserving fidelity across several conditional flow models, including multi-step and few-step text-to-image, depth-to-image, and reference image generation.

ECCV 2026

Physics-Grounded Disentangled Flow Modeling for Brain Disease Progression Trajectory

Jun Wang, Peirong Liu

Forecasting longitudinal brain lesion evolution is critical for disease monitoring and treatment planning. Existing approaches typically learn a direct mapping from a baseline image to a future observation, without explicitly modeling the physical mechanisms underlying the lesion progression. Such an entangled modeling of structural deformation and image intensity variation limits physical plausibility, model generalization, and interpretability. To address this, we propose PDF, a Physics-grounded Disentangled Flow matching framework for longitudinal brain disease forecasting. We explicitly decompose the longitudinal modeling of lesion growth into two processes, each learned by a dedicated flow matching network: morphology evolution, which captures lesion growth and structural deformation; and intensity evolution, which models signal changes driven by variations in lesion concentration. To enforce physics-grounded constraints, we introduce a PDE-regularized loss based on lesion growth dynamics, that enforces a diffusion-reactionadvection formulation for morphological evolution. Experiments on three public longitudinal datasets spanning diverse brain diseases demonstrate state-of-the-art performance, validating the effectiveness of the disentangled modeling framework and physics-grounded learning design. Code is publicly available at https://github.com/jhuldr/PDF.

ECCV 2026

Generating Multi-view Adversarial Examples for Visual Geometry Grounded Transformer

Qi Song, Ziyuan Luo, Haoliang Han, Renjie Wan

The Visual Geometry Grounded Transformer (VGGT) enables unified feed-forward 3D reconstruction from multi-view images. However, deploying such a high-performance model may expose critical security vulnerabilities. Traditional adversarial perturbations require costly per-scene optimization, while Universal Adversarial Perturbations (UAPs) rely on a single static pattern and fail to effectively attack VGGT. To address these limitations, we propose MVAP-G, a multiview adversarial perturbation generator that produces imperceptible consistent perturbations across multiple views in a single feed-forward pass. To ensure perturbation consistency across diverse scenes, we design a cross-view adversarial alignment mechanism to process multi-view images. Experiments demonstrate that MVAP-G significantly degrades VGGT performance without iterative optimization during inference. This work pioneers multi-view adversarial attacks on 3D foundation models, uncovering severe vulnerabilities and underscoring the urgent need for robust 3D vision systems. The code is available at https://github.com/ qsong2001/mvap-g.

ECCV 2026

Drop-In Perceptual Optimization for 3D Gaussian Splatting

Ezgi Ozyilkan, Zhiqi Chen, Oren Rippel, Jona Ballé, Kedar Tatwawadi

Despite their output being ultimately consumed by human viewers, 3D Gaussian Splatting (3DGS) methods often rely on ad-hoc combinations of pixel-level losses, resulting in blurry renderings. To address this, we systematically explore perceptual optimization strategies for 3DGS by searching over a diverse set of distortion losses. We conduct the first-of-its-kind large-scale human subjective study on 3DGS, involving 39,320 pairwise ratings across several datasets and 3DGS frameworks. A regularized version of Wasserstein Distortion, which we call WD-R, emerges as the clear winner, excelling at recovering fine textures without incurring a higher splat count. WD-R is preferred by raters more than 2.3× over the original 3DGS loss, and 1.5× over the current best method Perceptual-GS. WD-R also consistently achieves state-of-the-art LPIPS, DISTS, and FID scores across various datasets, and generalizes across recent frameworks, such as Mip-Splatting and Scaffold-GS, where replacing the original loss with WD-R consistently enhances perceptual quality within a similar resource budget (number of splats for Mip-Splatting, model size for Scaffold-GS), and leads to reconstructions being preferred by human raters 1.8× and 3.6×, respectively. We also find that this carries over to the task of 3DGS scene compression, with ≈ 50% bitrate savings for comparable perceptual metric performance.

Keep browsing

More papers from top AI conferences

30+ top conferences, by conference, year, and topic.