AI conference papers

AI conference paper library

Search papers from 30+ top conferences, including NeurIPS, ICLR, ICML, CVPR, ACL and AAAI. Filter by keyword, then open any paper for its abstract, authors and links.

arXiv

MVGD-Net: A Novel Motion-aware Video Glass Surface Detection Method

Yiwei Lu, Hao Huang, Tao Yan

Glass surface ubiquitous in both daily life and professional environments presents a potential threat to vision-based systems, such as robot and drone navigation. To solve this challenge, most recent studies have shown significant interest in Video Glass Surface Detection (VGSD). We observe that objects in the reflection (or transmission) layer appear farther from the glass surfaces. Consequently, in video motion scenarios, the notable reflected (or transmitted) objects on the glass surface move slower than objects in non-glass regions within the same spatial plane, and this motion inconsistency can effectively reveal the presence of glass surfaces. Based on this observation, we propose a novel network, named MVGD-Net, for detecting glass surfaces in videos by leveraging motion inconsistency cues. Our MVGD-Net features three novel modules: the Cross-scale Multimodal Fusion Module (CMFM) that integrates extracted spatial features and estimated optical flow maps, the History Guided Attention Module (HGAM) and Temporal Cross Attention Module (TCAM), both of which further enhances temporal features. A Temporal-Spatial Decoder (TSD) is also introduced to fuse the spatial and temporal features for generating the glass region mask. Furthermore, for learning our network, we also propose a large-scale dataset, which comprises 312 diverse glass scenarios with a total of 19,268 frames. Extensive experiments demonstrate that our MVGD-Net outperforms relevant state-of-the-art methods.

arXiv

WHU-PCPR: A cross-platform heterogeneous point cloud dataset for place recognition in complex urban scenes

Xianghong Zou, Jianping Li, Yandi Yang, Weitong Wu, Yuan Wang, Qiegen Liu, Zhen Dong

Point Cloud-based Place Recognition (PCPR) demonstrates considerable potential in applications such as autonomous driving, robot localization and navigation, and map update. In practical applications, point clouds used for place recognition are often acquired from different platforms and LiDARs across varying scene. However, existing PCPR datasets lack diversity in scenes, platforms, and sensors, which limits the effective development of related research. To address this gap, we establish WHU-PCPR, a cross-platform heterogeneous point cloud dataset designed for place recognition. The dataset differentiates itself from existing datasets through its distinctive characteristics: 1) cross-platform heterogeneous point clouds: collected from survey-grade vehicle-mounted Mobile Laser Scanning (MLS) systems and low-cost Portable helmet-mounted Laser Scanning (PLS) systems, each equipped with distinct mechanical and solid-state LiDAR sensors. 2) Complex localization scenes: encompassing real-time and long-term changes in both urban and campus road scenes. 3) Large-scale spatial coverage: featuring 82.3 km of trajectory over a 60-month period and an unrepeated route of approximately 30 km. Based on WHU-PCPR, we conduct extensive evaluation and in-depth analysis of several representative PCPR methods, and provide a concise discussion of key challenges and future research directions. The dataset and benchmark code are available at https://github.com/zouxianghong/WHU-PCPR.

arXiv

MATANet: A Multi-context Attention and Taxonomy-Aware Network for Fine-Grained Underwater Recognition of Marine Species

Donghwan Lee, Byeongjin Kim, Geunhee Kim, Hyukjin Kwon, Nahyeon Maeng, Wooju Kim

Accurate fine-grained recognition of marine organisms is important for scalable biodiversity monitoring and ecological assessment using underwater imagery. However, existing methods mainly focus on target appearance and make limited use of surrounding environmental cues and biological taxonomy. We propose the Multi-Context Attention and Taxonomy-Aware Network (MATANet) for region-of-interest (ROI)-guided marine organism recognition. MATANet contains two complementary components. The Multi-Context Environmental Attention Module uses the ROI representation as a query to aggregate spatial patch features from ROI-centered contextual views at multiple scales, enabling target-conditioned modeling of the surrounding environment. Level-wise auxiliary classifiers further incorporate higher taxonomic ranks during training, encouraging hierarchically consistent representations without changing the finest-label prediction space or inference procedure. On the official FathomNet 2025 Private test split, MATANet achieves a hierarchical distance of 1.570 with the base backbone and 1.423 with the large backbone, substantially outperforming the strongest benchmark value of 2.603. On FishCLEF2015, MATANet achieves an accuracy of 0.793 and a hierarchical distance of 1.120, outperforming the strongest benchmark values of 0.766 and 1.327, respectively. Ablation studies show that surrounding scene information provides complementary evidence beyond repeated multi-scale observations of the target and that target-conditioned aggregation outperforms direct multi-view concatenation. Additional experiments on FAIR1M v2.0 examine the applicability of the proposed design beyond underwater imagery. In post-detection evaluation, MATANet improves fine-grained classification accuracy on matched detector-generated ROIs from 0.828 to 0.959 supporting its engineering applicability to automated marine monitoring.

arXiv

GH-ESD: Grounded Hypothesis-Driven Error Slice Discovery for Instance-Level Vision Tasks

Wei Zhang, Chaoqun Wang, Zixuan Guan, Ping Sheng Kao, Pengfei Zhao, Peng Wu, Sifeng He

Systematic failures of vision models on semantically coherent subsets, known as error slices, reveal limitations in robustness and evaluation. Existing slice discovery approaches largely model slices as clusters in representation space or combinations of predefined attributes. While effective for image-level classification, such formulations are insufficient for instance-level tasks such as object detection and segmentation, where failures often arise from contextual relational and spatially grounded visual patterns. We propose GH-ESD (Grounded Hypothesis-Driven Error Slice Discovery), a generate and verify framework that reformulates slice discovery as grounded hypothesis generation and statistical verification. GH-ESD constructs relational failure hypotheses using LLM priors and grounded visual evidence, discovers hypothesis slices at the instance level via Vision Language Models, and verifies them through statistical trend analysis over instance-level errors. We also introduce GESD (Grounded Error Slice Dataset), a new benchmark for instance-level error slice discovery, providing expert-defined and spatially grounded slices derived from detection and segmentation failures. Extensive experiments demonstrate that GH-ESD consistently outperforms baselines, improving Precision@10 by 0.10 (0.73 vs. 0.63) on the GESD benchmark for detection tasks, while also supporting segmentation scenarios. GH-ESD identifies interpretable slices that facilitate actionable model improvements.

arXiv

Read or Ignore? A Unified Benchmark for Typographic-Attack Robustness and Text Recognition in Vision-Language Models

Futa Waseda, Shojiro Yamabe, Daiki Shiono, Kento Sasaki, Tsubasa Takahashi

Large vision-language models (LVLMs) are vulnerable to typographic attacks, where misleading text inserted into an image can override visual understanding. However, existing evaluation protocols and defenses are largely focused on object recognition and do not consider text-reading capability. This is a critical oversight: real-world scenarios often require both recognizing objects and reading scene text (e.g., recognizing pedestrians while reading traffic signs), where simply ignoring all text for robustness is unacceptable in practice. To address this gap, we introduce a novel task, Read-or-Ignore VQA (RIO-VQA), which jointly evaluates both requirements: models must decide, from context, when to read scene text and when to ignore inserted distractor text. To evaluate this capability, we present RIO-Bench, a same-scene counterfactual benchmark that holds the scene fixed while varying only question intent (object vs. text) and text condition (clean vs. attack), enabling direct comparisons of model behaviors with reduced confounding factors. Using RIO-Bench, we highlight a trade-off: representative defenses developed in object-centric settings can achieve robustness by suppressing text sensitivity, at the cost of text-reading performance (i.e., "ignoring" text). Motivated by this trade-off, we provide a data-driven defense baseline that improves both requirements on RIO-Bench, complementing prior text-ignoring baselines. Overall, this work highlights a fundamental misalignment between the current object-centric robustness scope and real-world multimodal requirements, providing a principled path toward reliable LVLMs.

arXiv

RubricRL: Simple Generalizable Rewards for Text-to-Image Generation

Xuelu Feng, Yunsheng Li, Ziyu Wan, Zixuan Gao, Junsong Yuan, Dongdong Chen, Chunming Qiao

Reinforcement learning (RL) has recently emerged as a promising approach for aligning text-to-image generative models with human preferences. A key challenge, however, lies in designing effective and interpretable rewards. Existing methods often rely on either composite metrics (e.g., CLIP, OCR, and realism scores) with fixed weights or a single scalar reward distilled from human preference models, which can limit interpretability and flexibility. We propose RubricRL, a simple and general framework for rubric-based reward design that offers greater interpretability, composability, and user control. Instead of using a black-box scalar signal, RubricRL dynamically constructs a structured rubric for each prompt--a decomposable checklist of fine-grained visual criteria such as object correctness, attribute accuracy, OCR fidelity, and realism--tailored to the input text. Each criterion is independently evaluated by a multimodal judge (e.g., o4-mini), and a prompt-adaptive weighting mechanism emphasizes the most relevant dimensions. This design not only produces interpretable and modular supervision signals for policy optimization (e.g., GRPO or PPO), but also enables users to directly adjust which aspects to reward or penalize. Experiments with an autoregressive text-to-image model demonstrate that RubricRL improves prompt faithfulness, visual detail, and generalizability, while offering a flexible and extensible foundation for interpretable RL alignment across text-to-image architectures.

arXiv

T2T-VICL: Cross-Task Visual In-Context Learning via Implicit Text-Driven VLMs

Shao-Jun Xia, Huixin Zhang, Zhengzhong Tu

Visual in-context learning (VICL) solves visual tasks by conditioning on a few input-output demonstrations without any model training. Recent advances in large vision-language models (VLMs) have shown promising VICL capability when the demonstration pair and the query belong to the same vision task, but real use cases often provide mismatched examples, making it unclear whether a VLM should imitate the demonstrated transformation or infer a new one from the query. This raises a fundamental question: Can VLMs perform cross-task VICL where demonstration and query differ? In the paper, we study this cross-task VICL setting and propose T2T-VICL, a collaborative prompt-transfer framework, which converts mismatched visual demonstrations into implicit textual guidance without explicitly naming the tasks. To do so, a large teacher VLM first generates structured descriptions of visual changes and task differences between task pairs, from which we construct a dataset of diverse implicit cross-task relations. We then distill this capability into a lightweight student VLM that produces content-dependent prompts from a task-A demonstration pair and a task-B query. The generated prompt is used to guide a frozen image-editing VLM, and a score-based inference strategy is introduced to rank multiple candidates. Experiments on 12 low-level vision tasks and over 20 evaluated cross-task pairs show that T2T-VICL consistently improves task-aware alignment over fixed prompting and often also improves image fidelity, revealing both the potential and limits of cross-task VICL. Our code is available on GitHub.

arXiv

Timeripple: Accelerating vDiTs by Understanding the Spatio-Temporal Correlations in Latent Space

Wenxuan Miao, Yulin Sun, Aiyue Chen, Jing Lin, Yiwu Yao, Yiming Gan, Jieru Zhao, Jingwen Leng, Minyi Guo, Yu Feng

The recent surge in video generation has shown the growing demand for high-quality video synthesis using large vision models. Existing video generation models are predominantly based on the video diffusion transformer (vDiT), however, they suffer from substantial inference delay due to self-attention. While prior studies have focused on reducing redundant computations in self-attention, they often overlook the inherent spatio-temporal correlations in video streams and directly leverage sparsity patterns from large language models to reduce attention computations. In this work, we take a principled approach to accelerate self-attention in vDiTs by leveraging the spatio-temporal correlations in the latent space. We show that the attention patterns within vDiT are primarily due to the dominant spatial and temporal correlations at the token channel level. Based on this insight, we propose a lightweight and adaptive reuse strategy that approximates attention computations by reusing partial attention scores of spatially or temporally correlated tokens along individual channels. We demonstrate that our method achieves significantly higher computational savings (85\%) compared to state-of-the-art techniques over 4 vDiTs, while preserving almost identical video quality ($<$0.06\% loss on VBench).

arXiv

SUPER Module for Detail-Sensitive and Cost-Efficient U-Net Variant Decoders

Siheon Joo, Hongjo Kim

Skip-connected U-Net variants are widely used for dense inverse problems, yet their decoders commonly recover resolution through spatial upscaling, which can blur or distort fine structures. Wavelet transforms provide an explicit perfect-reconstruction path, but prior wavelet networks often couple this property to wavelet-specific architectures or fixed output representations. We propose Selectively Suppressed Perfect Reconstruction (SUPER), a wavelet-domain decoder formulation that replaces unstructured spatial refinement with bounded frequency suppression. In its idealized equal-channel form, SUPER reduces to exact reconstruction when the suppression term is zero; in practical U-Net decoders, it provides a projected-subspace reconstruction fallback with learnable task-specific suppression. The resulting SUPER Module is a structurally plug-in decoder block: it replaces the upsampling/refinement stage of an existing U-Net-like decoder, while leaving the encoder and task head unchanged. We evaluate SUPER on monocular depth estimation, thin-crack segmentation, and smartphone image denoising. On iDisc depth estimation, SUPER improves the host model's edge AUC from 26.59% to 34.45% while reducing decoder MACs from 12.69G to 4.36G. On FACS-Net crack segmentation, SUPER improves average IoU and the extreme 0-2px crack regime. On SIDD denoising, where high-frequency enhancement is less directly rewarded, SUPER preserves PSNR/SSIM while reducing decoder MACs by 66.6%. These results support SUPER as a practical wavelet-domain suppression module for improving the observed detail-cost trade-off of U-Net-like decoders.

arXiv

Kontinuous Kontext: Continuous Strength Control for Instruction-based Image Editing

Rishubh Parihar, Or Patashnik, Daniil Ostashev, R. Venkatesh Babu, Daniel Cohen-Or, Kuan-Chieh Wang

Instruction-based image editing offers a powerful and intuitive way to manipulate images through natural language. Yet, relying solely on text instructions limits fine-grained control over the extent of edits. We introduce Kontinuous Kontext, an instruction-driven editing model that provides a new dimension of control over edit strength, enabling users to adjust edits gradually from no change to a fully realized result in a smooth and continuous manner. Kontinuous Kontext extends a state-of-the-art image editing model to accept an additional input, a scalar edit strength which is then paired with the edit instruction, enabling explicit control over the extent of the edit. To inject this scalar information, we train a lightweight projector network that maps the input scalar and the edit instruction to coefficients in the model's modulation space. For training our model, we synthesize a diverse dataset of image-edit-instruction-strength quadruplets using existing generative models, followed by a filtering stage to ensure quality and consistency. Kontinuous Kontext provides a unified approach for fine-grained control over edit strength for instruction driven editing from subtle to strong across diverse operations such as stylization, attribute, material, background, and shape changes, without requiring attribute-specific training.

arXiv

AIM-CoT: Active Information-driven Multimodal Chain-of-Thought for Vision-Language Reasoning

Xiping Li, Jianghong Ma

Interleaved-Modal Chain-of-Thought (I-MCoT) advances vision-language reasoning, such as Visual Question Answering (VQA). This paradigm integrates specially selected visual evidence from the input image into the context of Vision-Language Models (VLMs), enabling them to ground their reasoning logic in these details. Accordingly, the efficacy of an I-MCoT framework relies on identifying what to see (evidence selection) and when to see it (triggering of insertions). However, existing methods fall short in both aspects. First, for selection, they rely on attention signals, which are unreliable -- particularly under severe granularity imbalance between the brief textual query and the informative image. Second, for triggering, they adopt static triggers, which fail to capture the VLMs' dynamic needs for visual evidence. To this end, we propose a novel I-MCoT framework, Active Information-driven Multi-modal Chain-of-Thought (AIM-CoT), which aims to improve both evidence selection and insertion triggering via: (1) Context-enhanced Attention-map Generation (CAG) to mitigate granularity imbalance via textual context enhancement; (2) Active Visual Probing (AVP) to proactively select the most informative evidence via an information foraging process; and (3) Dynamic Attention-shift Trigger (DAT) to precisely activate insertions when VLM's attention shifts from text to visual context. Experiments across three benchmarks and four backbones demonstrate AIM-CoT's consistent superiority. Our code is available at https://anonymous.4open.science/r/AIMCoT.

arXiv

Pain in 3D: Generating Controllable Synthetic Faces for Automated Pain Assessment

Xin Lei Lin, Soroush Mehraban, Abhishek Moturu, Babak Taati

Automated pain assessment from facial expressions is crucial for non-communicative patient. Progress has been limited by two challenges: (i) existing datasets exhibit severe demographic and label imbalance due to ethical constraints, and (ii) current generative models cannot precisely control facial action units (AUs), facial structure, or clinically validated pain levels. We introduce 3DPain, a large-scale synthetic dataset designed to overcome data scarcity in automated pain assessment. Comprising 82,500 frames across 2,500 unique identities, 3DPain offers extensive heterogeneity in facial pain responses across demographic groups balanced by age, gender, and ethnicity. Our three-stage framework samples diverse 3D meshes, textures them with diffusion models, and applies AU-driven face rigging to synthesize multi-view faces with paired neutral/pain images, facial action units, PSPI scores, and pain-region heatmaps. We further introduce ViTPain, a Vision Transformer based framework leveraging cross-attention with a neutral reference face to achieve identityaware pain estimation. Together, 3DPain and ViTPain establish a controllable, diverse, and clinically grounded foundation for generalizable automated pain assessment.

arXiv

FedS2R: One-Shot Federated Domain Generalization for Synthetic-to-Real Semantic Segmentation in Autonomous Driving

Tao Lian, Jose L. Gómez, Antonio M. López

Federated domain generalization has shown promising progress in image classification by enabling collaborative training across multiple clients without sharing raw data. However, its potential in the semantic segmentation of autonomous driving remains underexplored. In this paper, we propose FedS2R, the first one-shot federated domain generalization framework for synthetic-to-real semantic segmentation in autonomous driving. FedS2R comprises two components: an inconsistency-driven data augmentation strategy that generates images for unstable classes, and a multi-client knowledge distillation scheme with feature fusion that distills a global model from multiple client models. Experiments on five real-world datasets, Cityscapes, BDD100K, Mapillary, IDD, and ACDC, show that the global model significantly outperforms individual client models and is only 2 mIoU points behind the model trained with simultaneous access to all client data. These results demonstrate the effectiveness of FedS2R in synthetic-to-real semantic segmentation for autonomous driving under federated learning

arXiv

Great X: A Unified Multi-Modal Simulator Bridging the Sim2Real Gap for 6G

Yuan Gao, Kongwu Huang, Jun Jiang, Shiyi Mu, Shugong Xu

Large-scale, precisely synchronized multi-modal datasets are critical for data-driven sixth-generation (6G) wireless research, yet real-world collection remains costly and difficult. Existing multi-platform simulators often suffer from timing misalignment, limited modality support, and simplified scene and material modeling, which reduce fidelity and enlarge the simulation-to-reality (Sim2Real) gap. We propose Great-X, a unified single-engine simulator implemented in Unreal Engine. Great-X integrates native ray tracing with co-located visual and electromagnetic material properties on shared 3D meshes, enabling pixel-level consistency between radio and visual outputs. A deterministic fixed-step clock and synchronous buffering provide frame-accurate alignment across CSI, RGB, depth, LiDAR, radar, and event data. Based on this framework, we construct Great-MCD, containing over three million synchronized samples across urban and rural environments, day and night conditions, multiple UAV types, and diverse trajectories. Experiments show that Great-X generates channel impulse responses consistent with real measurements and provides transferable data for wireless positioning and CSI feedback, outperforming existing simulators and measured-data baselines in both zero-shot and fine-tuned settings.

arXiv

FE-MCFormer: a novel time-frequency interpretable architecture for machinery fault diagnosis under strong noise environments

Yuhan Yuan, Xiaomo Jiang, Haibin Yang, Haixin Zhao, Shengbo Wang, Xueyu Cheng, Jigang Meng

Interpretable fault diagnosis (FD) plays a critical role in industrial manufacturing, as it improves human-machine understanding and operational efficiency. However, harsh operating environments often introduce strong background interference or noise, which weakens the discriminative capability and interpretability of existing FD methods. To address this issue, this paper proposes FE-MCFormer, a time-frequency fusion framework for robust and time-frequency interpretable fault diagnosis under strong noise conditions. A frequency adaptive learning layer (FALL) is developed to perform learnable spectral reconstruction, which explicitly suppresses noise-dominated frequency responses while preserving fault-sensitive harmonic structures. Furthermore, a multiscale time-frequency fusion (MSTFF) architecture is designed to jointly capture localized impulsive characteristics and structured global spectral interactions. Extensive experiments on a rolling bearing dataset and a real-world centrifugal compressor dataset demonstrate that the proposed method achieves stable and interpretable diagnostic performance under severe noise environments down to -10 dB SNR. The results indicate that FE-MCFormer provides an effective framework for turbomachinery fault diagnosis in complex noisy environments.

arXiv

Generalized Least Squares Kernelized Tensor Factorization

Mengying Lei, Lijun Sun

Recovering incomplete multidimensional tensor-structured data is a fundamental task in many real-world applications. Smoothness-constrained low-rank tensor factorization effectively captures global and long-range correlations, but often struggles to characterize short-scale, high-frequency, or locally varying structures. We propose GLSKF, a complementary Generalized Least Squares Kernelized Tensor Factorization framework, for multidimensional spatiotemporal data completion. GLSKF additively integrates a covariance-regularized low-rank global component with an explicitly modeled locally correlated residual component under a GLS objective, enabling effective modeling of both global dependencies and localized variations. A covariance norm regularizer encodes spatiotemporal dependencies in both components: structured covariances are imposed on the latent factor columns to enforce smoothness in the global factorization, whereas compactly supported sparse kernels are used to model local correlations in the residual. We develop an alternating least squares algorithm with blockwise linear-system updates that exploit the Kronecker structure of the covariance matrices under missing data and facilitate fast conjugate gradient solves. Additional computational gains are obtained by exploiting the sparsity and Toeplitz structure of the local residual covariance matrices for efficient matrix-vector multiplications. We evaluate GLSKF on four real-world multidimensional data-completion tasks: traffic speed imputation, color image completion, digital video recovery, and MRI data reconstruction. Experimental results demonstrate that GLSKF achieves superior reconstruction performance and favorable scalability across a range of tensor completion tasks, supporting its broad applicability to multidimensional data completion.

arXiv

Soft-TransFormers for Continual Learning

Haeyong Kang, Chang D. Yoo

Inspired by the Well-initialized Lottery Ticket Hypothesis (WLTH), we introduce Soft-TransFormers (Soft-TF), a continual learning framework that adapts a frozen pre-trained Transformer through task-specific soft subnetworks: real-valued multiplicative masks over the query, key, value, and output projections of selected self-attention layers. The masks are initialized at one, so optimization starts exactly at the pre-trained solution, and mask-space gradient descent is intrinsically biased toward modulating the backbone's dominant pathways; we prove that, under standard convex-Lipschitz assumptions, both the convergence rate and the parameter drift of mask-only fine-tuning are controlled by the distance from the pre-trained weights to a task-optimal configuration. This bounded drift yields two properties. Since the backbone and per-task masks are never overwritten, forgetting is structurally eliminated. And since every task subnetwork stays near the shared pre-trained solution, a wrong mask still evaluates a near-generalist function, so task-inference errors are largely harmless and class-incremental accuracy is decoupled from task-inference reliability. As a plug-in, Soft-TF couples with L2P, DualPrompt, HiDe-Prompt, and NoRGa, selecting masks by task-key matching, an entropy-gradient criterion, or a learned task-identity classifier. Across class-incremental learning benchmarks -- Split-CIFAR100, Split-ImageNet-R, CUB-200, and 5-Datasets -- Soft-TF consistently outperforms prompt-based, adapter-based, and LoRA-style baselines at comparable trainable-parameter budgets, while keeping inference cost identical to the unmodified backbone.

arXiv

Survival of the Cheapest: Cost-Aware Hardware Adaptation for Adversarial Robustness

Charles Meyers, Mohammad Reza Saleh Sedghpour, Tommy Löfstedt, Erik Elmroth

Deploying adversarially robust machine learning systems requires continuous trade-offs between robustness, cost, and latency. We present an autonomic decision-support framework providing a quantitative foundation for adaptive hardware selection and hyper-parameter tuning in cloud-native deep learning. The framework applies accelerated failure time (AFT) models to quantify the effect of hardware choice, batch size, epochs, and validation accuracy on model survival time. This framework can be naturally integrated into an autonomic control loop (monitor--analyse--plan--execute, MAPE-K), where system metrics such as cost, robustness, and latency are continuously evaluated and used to adapt model configurations and hardware selection. Experiments across three GPU architectures confirm the framework is both sound and cost-effective: the Nvidia L4 yields a 20% increase in adversarial survival time while costing 75% less than the V100, demonstrating that expensive hardware does not necessarily improve robustness. The analysis further reveals that model inference latency is a stronger predictor of adversarial robustness than training time or hardware configuration.

arXiv

Denoising Monte Carlo Renders with Diffusion Models

Vaibhav Vavilala, Rahul Vasanth, David Forsyth

Physically-based renderings contain Monte Carlo noise, with variance that increases as the number of rays per pixel decreases. This noise, while zero-mean for good modern renderers, can have heavy tails (most notably, for scenes containing specular or refractive objects). Learned methods for restoring low fidelity renders are highly developed, because suppressing render noise means one can save compute and use fast renders with few rays per pixel. We demonstrate that a diffusion model can denoise low fidelity renders successfully. Furthermore, our method can be conditioned on a variety of natural render information, and this conditioning helps performance. Quantitative experiments show that our method is competitive with SOTA across a range of sampling rates. Qualitative examination of the reconstructions suggests that the image prior applied by a diffusion method strongly favors reconstructions that are like real images -- so have straight shadow boundaries, curved specularities and no fireflies.

arXiv

EAR-Net: Pursuing End-to-End Absolute Rotations from Multi-View Images

Yuzhen Liu, Qiulei Dong

Absolute rotation estimation is an important topic in 3D computer vision. Existing works in literature generally employ a multi-stage (at least two-stage) estimation strategy where multiple independent operations (feature matching, two-view rotation estimation, and rotation averaging) are implemented sequentially. However, such a multi-stage strategy inevitably leads to the accumulation of the errors caused by each involved operation, and degrades its final estimation on global rotations accordingly. To address this problem, we propose an End-to-end method for estimating Absolution Rotations from multi-view images based on deep neural Networks, called EAR-Net. The proposed EAR-Net consists of an epipolar confidence graph construction module and a confidence-aware rotation averaging module. The epipolar confidence graph construction module is explored to simultaneously predict pairwise relative rotations among the input images and their corresponding confidences, resulting in a weighted graph (called epipolar confidence graph). Based on this graph, the confidence-aware rotation averaging module, which is differentiable, is explored to predict the absolute rotations. Thanks to the introduced confidences of the relative rotations, the proposed EAR-Net could effectively handle outlier cases. Experimental results on three public datasets demonstrate that EAR-Net outperforms the state-of-the-art methods by a large margin in terms of accuracy and speed.

Keep browsing

More papers from top AI conferences

30+ top conferences, by conference, year, and topic.