AI conference papers

AI conference paper library

Search papers from 30+ top conferences, including NeurIPS, ICLR, ICML, CVPR, ACL and AAAI. Filter by keyword, then open any paper for its abstract, authors and links.

arXiv

Potential failures of physics-informed machine learning in traffic flow modeling: theoretical and experimental analysis

Yuan-Zheng Lei, Yaobang Gong, Dianwei Chen, Yao Cheng, Xianfeng Terry Yang

This study investigates why physics-informed machine learning (PIML) can fail in macroscopic traffic flow modeling. We define failure as cases where a PIML model underperforms both purely data-driven and purely physics-based baselines by a given threshold. Unlike in other fields, physics residuals themselves do not hinder optimization in this setting. Instead, effective updates require both data and physics gradients to form acute angles with the true gradient, a condition difficult to satisfy with low-resolution loop data. In such cases, neural networks cannot accurately approximate density and speed, and the constructed physics residuals, already degraded by discrete sampling and temporal averaging, lose their ability to capture PDE dynamics, which directly leads to PIML failure. Theoretically, although LWR and ARZ solutions are weak solutions, for piecewise $C^k$ initial data they remain $C^k$ off the shock set under mild conditions, which has Lebesgue measure zero. Thus, almost all detector or collocation points lie in smooth regions where residuals are valid, and the MLP's inability to exactly represent discontinuities is immaterial. Finally, we establish MSE lower bounds of physics residuals: higher-order models such as ARZ have strictly larger consistency error bounds than LWR under mild conditions. This explains why LWR-based PIML can outperform ARZ-based PIML even with high-resolution data, with the gap shrinking as resolution increases, consistent with prior empirical findings.

arXiv

Topology-Driven Clustering: Enhancing Performance with Betti Number Filtration

Arghya Pratihar, Kushal Bose, Swagatam Das

Clustering aims at partitioning data points into groups of similar objects without knowing about the class labels. However, clustering datasets with complex geometric structures, such as nonconvex shapes, multiple scales, or intertwined manifolds, remains challenging for traditional algorithms that primarily rely on Euclidean or kernel-based similarity measures. Topological Data Analysis (TDA), particularly persistent homology, provides a powerful framework for capturing intrinsic structural properties of data, including connected components, loops, and higher-dimensional features across multiple scales. In this work, we propose a novel topological clustering algorithm called \textbf{Betti Number Filtration-based Topological Clustering (BFTC)}. The proposed method constructs local Vietoris-Rips filtrations around each data point and computes Betti numbers up to a prescribed dimension. These Betti numbers across filtration scales form \emph{Betti sequences}, which serve as multiscale topological signatures of local neighborhoods. By comparing Betti sequences among neighboring points, BFTC identifies topologically similar neighbors and refines the neighborhood graph to construct a topology-aware similarity structure and spectral clustering is applied to obtain the final clusters. Experimental results on several synthetic and real-world datasets demonstrate that BFTC effectively clusters complex and intertwined structures and consistently outperforms several state-of-the-art topology-based clustering methods.

arXiv

Learning, Reasoning, Refinement: A Framework for Kahneman's Dual-System Intelligence in GUI Agents

Jinjie Wei, Jiyao Liu, Lihao Liu, Ming Hu, Junzhi Ning, Mingcheng Li, Weijie Yin, Junjun He, Xiao Liang, Chao Feng, Dingkang Yang

Graphical User Interface (GUI) agents have made significant progress in automating digital tasks through the utilization of computer vision and language models. Nevertheless, existing agent systems encounter notable limitations. Firstly, they predominantly depend on trial and error decision making rather than progressive reasoning, thereby lacking the capability to learn and adapt from interactive encounters. Secondly, these systems are assessed using overly simplistic single step accuracy metrics, which do not adequately reflect the intricate nature of real world GUI interactions. In this paper, we present CogniGUI, a cognitive framework developed to overcome these limitations by enabling adaptive learning for GUI automation resembling human-like behavior. Inspired by Kahneman's Dual Process Theory, our approach combines two main components: (1) an omni parser engine that conducts immediate hierarchical parsing of GUI elements through quick visual semantic analysis to identify actionable components, and (2) a Group based Relative Policy Optimization (GRPO) grounding agent that assesses multiple interaction paths using a unique relative reward system, promoting minimal and efficient operational routes. This dual-system design facilitates iterative ''exploration learning mastery'' cycles, enabling the agent to enhance its strategies over time based on accumulated experience. Moreover, to assess the generalization and adaptability of agent systems, we introduce ScreenSeek, a comprehensive benchmark that includes multi application navigation, dynamic state transitions, and cross interface coherence, which are often overlooked challenges in current benchmarks. Experimental results demonstrate that CogniGUI surpasses state-of-the-art methods in both the current GUI grounding benchmarks and our newly proposed benchmark.

arXiv

FormGym: Doing Paperwork with Agents

Matthew Toles, Rattandeep Singh, Isaac Song, Zhou Yu

Completing paperwork is a challenging and time-consuming problem. Form filling is especially challenging in the pure-image domain without access to OCR, typeset PDF text, or a DOM. For computer agents, it requires multiple abilities, including multi-modal understanding, information retrieval, and tool-use. We present a novel form-filling benchmark consisting of 432 fields spread across 55 documents and 3 tasks, requiring knowledge of 236 features per user. We find that baseline VLAs achieve less than 1% accuracy in most cases, primarily due to poor localization ability. GUI agents also struggle, scoring between 10.6-68.0% despite high cost and latency. Therefore, we also contribute FieldFinder, a tool to assist LLMs in identifying where to place text on a form. With FieldFinder, all models achieve equal or better performance in all six study conditions, with a maximum increase from 2% to 56%.

arXiv

Chi-Square Wavelet Graph Neural Networks for Heterogeneous Graph Anomaly Detection

Xiping Li, Xiangyu Dong, Xingyi Zhang, Kun Xie, Yuanhao Feng, Bo Wang, Guilin Li, Wuxiong Zeng, Xiujun Shu, Sibo Wang

Graph Anomaly Detection (GAD) in heterogeneous networks presents unique challenges due to node and edge heterogeneity. Existing Graph Neural Network (GNN) methods primarily focus on homogeneous GAD and thus fail to address three key issues: (C1) Capturing abnormal signal and rich semantics across diverse meta-paths; (C2) Retaining high-frequency content in HIN dimension alignment; and (C3) Learning effectively from difficult anomaly samples with class imbalance. To overcome these, we propose ChiGAD, a spectral GNN framework based on a novel Chi-Square filter, inspired by the wavelet effectiveness in diverse domains. Specifically, ChiGAD consists of: (1) Multi-Graph Chi-Square Filter, which captures anomalous information via applying dedicated Chi-Square filters to each meta-path graph; (2) Interactive Meta-Graph Convolution, which aligns features while preserving high-frequency information and incorporates heterogeneous messages by a unified Chi-Square Filter; and (3) Contribution-Informed Cross-Entropy Loss, which prioritizes difficult anomalies to address class imbalance. Extensive experiments on public and industrial datasets show that ChiGAD outperforms state-of-the-art models on multiple metrics. Additionally, its homogeneous variant, ChiGNN, excels on seven GAD datasets, validating the effectiveness of Chi-Square filters. Our code is available at https://github.com/HsipingLi/ChiGAD.

arXiv

GSPRec: On Improving Item Representations in Graph Signal Processing for Collaborative Filtering

Ahmad Bin Rabiah, Julian McAuley

Graph-based collaborative filtering methods act as low-pass filters in the spectral domain and discard the intermediate-frequency components where community-level user preferences reside. Existing GSP-based methods address the loss through sophisticated filter designs, yet derive item representations from the user-item interaction matrix alone. The interaction matrix captures which items each user interacted with, but not which items appear close together in users' interaction sequences. We propose GSPRec, a graph spectral collaborative filtering framework that produces richer item spectral representations by incorporating item-item proximity derived from user interaction ordering before spectral filtering. GSPRec derives item-item edges from user interaction ordering and strengthens the edges through multi-hop diffusion with exponential decay. The unified graph topology incorporates the diffused edges alongside user-item interactions. The resulting Laplacian exposes intermediate-frequency structure that a Gaussian bandpass filter selectively amplifies. A low-pass filter retains broad popularity trends. Experiments on four real-world datasets show that GSPRec outperforms all graph CF baselines, with average improvements of 5.12% in NDCG@10. Ablation studies establish that graph construction and filter design are coupled. GSPRec without the bandpass filter falls below every GSP baseline, whereas GSPRec without item-item proximity still surpasses baselines.

arXiv

A Self-Supervised Framework for Space Object Behaviour Characterisation

Ian Groves, Andrew Campbell, James Fernandes, Diego Ramírez Rodríguez, Paul Murray, Massimiliano Vasile, Victoria Nockles

Foundation Models, which leverage large neural networks pre-trained on unlabelled data before fine-tuning for specific tasks, are increasingly being applied to specialised domains. Recent examples include ClimaX for climate and Clay for satellite Earth observation, but a Foundation Model for Space Object Behavioural Analysis has not yet been developed. As orbital populations grow, automated methods for characterising space object behaviour are crucial for space safety. Here, we present a self-supervised framework for space object behavioural analysis, representing a first step towards a Foundation Model for SOBA. The backbone is a Perceiver-Variational Autoencoder (VAE) architecture, pre-trained with self-supervised reconstruction and masked reconstruction on ~227,000 light curves from the MMT-9 observatory. The VAE enables anomaly detection, motion prediction, and synthetic light curve generation. We fine-tuned the model using two independent light curve simulators (CASSANDRA and GRIAL), with CAD models of boxwing, Sentinel-3, SMOS, and Starlink platforms. Our pre-trained model achieved a reconstruction mean squared error of 0.0012, identifying potentially anomalous light curves through reconstruction difficulty. After fine-tuning, the model scored 85% and 82% accuracy, with 0.92 and 0.95 ROC AUC scores in anomaly detection and motion mode prediction (e.g., sun-pointing, spin, tumbling). Analysis of high-confidence predictions on real data revealed distinct patterns including characteristic object profiles and satellite glinting. Our work demonstrates how self-supervised learning can simultaneously enable anomaly detection, motion prediction, and synthetic data generation from rich pre-trained representations, supporting space safety and sustainability through automated monitoring and simulation.

arXiv

Ab-initio simulation of excited-state potential energy surfaces with transferable deep quantum Monte Carlo

Zeno Schätzle, P. Bernát Szabó, Alice Cuzzocrea, Matěj Mezera, Frank Noé

The accurate quantum chemical calculation of excited states is a challenging task, often requiring computationally demanding methods. When entire ground and excited potential energy surfaces (PESs) are desired, for instance to predict the interaction of light excitation and structural changes, one is often forced to use cheaper computational methods at the cost of reduced accuracy. Here we introduce a method for the geometrically transferable optimization of neural network wave functions that leverages weight sharing and dynamical ordering of electronic states. Our method enables the efficient prediction of ground and excited-state PESs and their intersections at the highest accuracy, demonstrating up to two orders of magnitude cost reduction compared to single-point calculations.We validate our approach on four challenging excited-state PESs, namely ethylene, the carbon dimer, the methylenimmonium cation, and a rubredoxin active site model containing 96 electrons, illustrating the potential of transferable deep-learning QMC as a practical framework for studying electronic excitations in molecules.

arXiv

Linear convergence of proximal descent schemes on the Wasserstein space

Razvan-Andrei Lascu, Mateusz B. Majka, David Šiška, Łukasz Szpruch

We investigate proximal descent methods, inspired by the minimizing movement scheme introduced by Jordan, Kinderlehrer and Otto, for optimizing entropy-regularized functionals on the Wasserstein space. We establish linear convergence under flat convexity assumptions, thereby relaxing the common reliance on geodesic convexity. Our analysis circumvents the need for discrete-time adaptations of the Evolution Variational Inequality (EVI). Instead, we leverage a uniform logarithmic Sobolev inequality (LSI) and the entropy ``sandwich" lemma, extending the analysis from arXiv:2201.10469 and arXiv:2202.01009. The major challenge in the proof via LSI is to show that the relative Fisher information is well-defined at every step of the scheme. Since the relative entropy is not Wasserstein differentiable, we prove that along the scheme the iterates belong to a certain class of Sobolev regularity, and hence the relative entropy has a unique Wasserstein sub-gradient, and that the relative Fisher information is indeed finite.

arXiv

Cooperative Evolutionary Pressure and Diminishing Returns Might Explain the Fermi Paradox: On What Super-AIs Are Like

Daniel Vallstrom

With an evolutionary approach, the basis of morality can be explained as adaptations to problems of cooperation. With 'evolution' taken in a broad sense, AIs that satisfy the conditions for evolution to apply will be subject to the same cooperative evolutionary pressure as biological entities. Here the adaptiveness of increased cooperation as material safety and wealth increase is discussed -- for humans, for other societies, and for AIs. Diminishing beneficial returns from increased access to material resources also suggests the possibility that, on the whole, there will be no incentive to for instance colonize entire galaxies, thus providing a possible explanation of the Fermi paradox, wondering where everybody is. It is further argued that old societies could engender and eventually give way to super-AIs, since it is likely that super-AIs are feasible, and fitter. Closing is an aside on effective ways for morals and goals to affect life and society, emphasizing environments, cultures, and laws, and exemplified by how to eat. 'Diminishing returns' is defined, as less than roots, the inverse of infeasibility. It is also noted that there can be no exponential colonization or reproduction, for mathematical reasons, as each entity takes up a certain amount of space. Appended are an algorithm for colonizing for example a galaxy quickly, models of the evolution of cooperation and fairness under diminishing returns, and software for simulating signaling development.

arXiv

Bayesian inference of composition-dependent phase diagrams

Timofei Miryashkin, Olga Klimanova, Vladimir Ladygin, Alexander Shapeev

Phase diagrams serve as a highly informative tool for materials design, encapsulating information about the phases that a material can manifest under specific conditions. In this work, we develop a method in which Bayesian inference is employed to combine thermodynamic data from molecular dynamics (MD), melting point simulations, and phonon calculations, process these data, and yield a temperature-concentration phase diagram. The employed Bayesian framework yields not only the free energies of different phases as functions of temperature and concentration but also the uncertainties of these free energies originating from statistical errors inherent to finite-length MD trajectories. Furthermore, it extrapolates the results of the finite-atom calculations to the infinite-atom limit and facilitates the choice of temperature, chemical potentials, and the number of atoms conducting the next simulation with which will be the most efficient in reducing the uncertainty of the phase diagram. The developed algorithm was successfully tested on two binary systems, Ge-Si and K-Na, in the full range of concentrations and temperatures.

arXiv

Democratizing Advanced High-Throughput Imaging via Cross-Instrument Deep Learning-Enabled Modality Transfer

Dominik Panek, Carina Rząca, Maksymilian Szczypior, Joanna Sorysz, Krzysztof Misztal, Zbigniew Baster, Zenon Rajfur

High-throughput imaging is often constrained by a trade-off between acquisition speed and image quality. Fast imaging modalities, such as wide-field fluorescence microscopy, enable large-scale data acquisition but suffer from reduced contrast and resolution, whereas high-resolution techniques, like confocal or super-resolution techniques, provide superior image quality at the cost of reduced throughput and increased instrument time. Here, we present a deep learning-based approach for modality transfer across independent microscopes, enabling the transformation of low-quality images acquired on fast systems into high-quality representations comparable to those obtained using advanced imaging platforms. To achieve this, we employed a generative adversarial network (GAN)-based model trained on paired datasets acquired on physically separate wide-field and confocal microscopes, demonstrating that image quality can be reliably transferred between independent instruments. Importantly, this approach enables a workflow in which high-throughput imaging can be performed on fast, accessible microscopy systems while preserving the ability to computationally recover high-quality structural information. High-resolution microscopy can then be reserved for algorithm training and targeted validation, reducing acquisition time and improving overall experimental efficiency. This workflow supports a model in which shared imaging facilities provide access to advanced instrumentation without requiring individual research groups to procure dedicated high-end systems. Together, our results establish deep learning-enabled modality transfer as a practical strategy for bridging independent microscopy systems and supporting scalable, high-content imaging workflows.

arXiv

When Are Scoring Rules Proper? Bridging Theory and Practice in Survival Model Evaluation

John Zobolas, Raphael Sonabend, Riccardo De Bin, Johannes Piller, Philipp Kopper, Lukas Burk, Andreas Bender

Proper scoring rules encourage probabilistic predictions that match the true underlying distribution and are central to model evaluation, with increasing relevance in automated workflows such as AutoML. In survival analysis, however, their behavior under censoring is not fully understood. We study commonly used squared and logarithmic scoring rules for right-censored survival data under independent censoring, introducing a notion of marginal properness based on observable outcomes. Within this framework, we show that the SBS, evaluated at a fixed time point, along with its integrated version (ISBS) and the RCLL are strictly proper when all individuals eventually experience the event, but can become improper under finite follow-up or in the presence of cure fractions. For the SBS, we derive a closed-form expression that reveals the true mechanism: residual mass, corresponding to individuals who remain event-free at study end, systematically biases the score toward underestimating survival, with the effect increasing at later evaluation times and under heavier censoring. Through simulation experiments, we examine how these issues manifest in finite samples and under misspecification. The SBS exhibits pronounced improperness at late evaluation times and poor discrimination between models. The ISBS is more robust due to temporal integration but remains sensitive to tail regularity violations, exhibiting detectable improperness and reduced discriminatory power. The RCLL behaves consistently with strict properness and effectively separates misspecified models. Overall, our results demonstrate how theoretical improperness can translate into misleading model comparisons, underscoring the need for further methodological development in survival model evaluation under censoring and realistic data conditions.

arXiv

Pathologist Attention-Aligned Report Generation for Prostate Histopathology

Ruoyu Xue, Suryakant Singh, Souradeep Chakraborty, Pierre Marza, Oksana Yaskiv, Constantin Friedman, Natallia Sheuka, Paul Friedman, Bharat Ramlal, Beatrice Knudsen, Rajarsi Gupta, Joel Saltz, Prateek Prasanna, Gregory Zelinsky, Dimitris Samaras

The allocation of visual attention by pathologists during cancer diagnosis is a highly selective process that critically shapes the information extracted from whole-slide images (WSIs). Human attention helps medical imaging tasks such as classification and segmentation, and becomes a strong semantic cue for identifying diagnostically informative regions for report generation. In this paper, we introduce human attention into the training of pathologist report generation models. To this end, we collected a multimodal human-attention dataset of 121 prostate WSIs annotated with pathologists' multi-scale viewport trajectories synchronized with the pathologists' verbal descriptions and cursor movements for five clinically relevant components (e.g., Gleason patterns). Using this dataset, we finetune two report generation models with an attention-alignment loss that regularizes the model attention over image patches to match the distribution of pathologist attention. We evaluate our approach on prostate cancer report generation and visual question answering using two models with different internal attention mechanisms (i.e., how image tokens are integrated into the language decoder). Experiments show average gains of 10.9% on NLP-based metrics and 19.3% in accuracy across five clinically relevant report components. Further, model attention maps extracted at inference time, with minimal computational overhead, align more closely with pathologist attention, providing stronger visual support for the generated reports by highlighting the regions that most influence the output.

arXiv

EGRNet: A Lightweight Semantic Segmentation Network with Edge-Gated Refinement and Adversarial Sensing

Bareera Qaseem, Mohsin Kamal, Muhammad Naveed Aman

As autonomous systems and smart cities continue to evolve, the demand for efficient and robust scene understanding becomes increasingly critical. Semantic segmentation plays a key role in enabling autonomous vehicles to comprehend complex urban environments. However, achieving high accuracy with minimal computational cost remains a significant challenge. In this paper, we present Edge-Gated Refinement Network (EGRNet), a lightweight and efficient deep learning model designed for real-time semantic segmentation in urban scenarios. The model incorporates depthwise separable convolutions to reduce computational complexity and dilated residual blocks for capturing rich multi-scale contextual information. Additionally, we introduce a novel Edge-Gated Refinement (EGR) module, which adaptively fuses original and refined features through a learnable gating mechanism, enhancing boundary preservation and edge-sensitive regions. To further improve feature representation, Squeeze-and-Excitation (SE) attention is applied across the network. With only 0.46M parameters, EGRNet achieves state-of-the-art performance while maintaining low computational overhead. When evaluated on the Cityscapes dataset, the model attains a mean Intersection over Union (mIoU) of 65.28%, demonstrating strong accuracy with minimal resource consumption. Moreover, we introduce a lightweight adversarial attack detection strategy, ensuring robustness against adversarial inputs without compromising real-time performance. By combining efficiency, accuracy, and resilience, EGRNet is well-suited for deployment on edge devices in safety-critical real-time applications.

arXiv

Deep Shape Regression for Planar Curves with Multimodal Covariates

Manuel Pfeuffer, Roshan Prakash Rane, Hadya Yassin, Kerstin Ritter, Sonja Greven

The shape of a planar curve is the geometric information that remains once translation, rotation, scale and reparametrisation are removed and is of interest in many health applications, e.g. in neuroimaging. We propose a deep shape regression model for open planar curves that admits multimodal and high-dimensional covariates. Representing curves as complex-valued functions, we show that the conditional full Procrustes mean is the leading eigenfunction of the conditional covariance. To estimate this covariance surface, we propose a novel deep conditional covariance smoother with modality-specific encoders - e.g. splines for scalar covariates and convolutional networks for images, which classical spline smoothers cannot accommodate. Our model is by construction invariant to the translation, rotation and scaling of the input curves and handles sparsely and irregularly sampled curves. We further provide an algorithm for elastic mean estimation that also removes parametrisation by iterating covariance smoothing, rotational alignment and parametrisation alignment. We illustrate the method on simulated outlines with known conditional mean and multimodal covariates, and give a first application to hippocampal outlines from the ADNI cohort, recovering covariate effects consistent with the literature. Code is available at https://github.com/mpff/dnn-shapes.

arXiv

VQ-Transplant: Efficient VQ-Module Integration for Pre-trained Visual Tokenizers

Xianghong Fang, Yuan Yuan, Dehan Kong, Tim G. J. Rudner

Vector Quantization (VQ) underpins modern discrete visual tokenization. However, training quantization modules for state-of-the-art VQ-based models requires significant computational resources which, in practice, all but prevents the development of novel, cutting-edge VQ techniques under resource constraints. To address this limitation, we propose {\bf VQ-Transplant}, a simple framework that enables plug-and-play integration of new VQ modules into frozen, pre-trained tokenizers by replacing their native VQ modules. Crucially, the proposed transplantation process preserves all encoder-decoder parameters, obviating the need for costly end-to-end retraining when modifying the quantization method. To mitigate decoder-quantization mismatch, we introduce a lightweight decoder adaptation strategy (trained for only 5 epochs on ImageNet-1k) to align feature priors with the new quantization space. In our empirical evaluation, we find that VQ-Transplant allows obtaining near state-of-the-art reconstruction fidelity for industry-level models like VAR while reducing the training cost by 95\%. VQ-Transplant democratizes quantization research by enabling resource-efficient integration of novel VQ techniques while matching industry-level reconstruction performance.

arXiv

ChronoStitch: Training-Free Composition of Visual KV Memories for Long-Horizon Temporal Reasoning

Santiram Tiwari, Nishant Sinha, Kunal Kislay

Long-video question answering requires a model to preserve visual evidence over time without repeatedly reprocessing the same video. A practical approach is to store the vision-language model's internal key-value (KV) cache for each video chunk and retrieve that state at query time. However, independently cached video chunks do not compose correctly: every chunk is prefilled from local rotary position zero, so naive concatenation collides temporal phases and removes the global order required for questions about what happened first, how often events occurred, or what changed across the video. This paper presents ChronoStitch, a training-free method for composing independently stored visual KV memories. The method first re-bases stored post-rotary keys onto a global three-axis multimodal RoPE coordinate system that preserves time, height, and width structure. We show why a one-dimensional scalar re-indexing is geometrically inconsistent for visual tokens because it turns spatial order within a frame into false temporal displacement. We then address the residual content gap left by positional repair: later chunks were originally encoded without attending to earlier chunks. ChronoStitch therefore selectively recomputes a small fraction of high-deviation later-chunk visual tokens while allowing them to attend over the composed cache. On Qwen2.5-VL-3B and the temporal split of TempCompass, ChronoStitch outperforms naive composition and position-only variants, improving event-ordering accuracy while running 3.3x faster than full joint re-prefilling.

arXiv

D3VL: Understanding Driving Scenes from 3D Time Series Data and Video with Language Models

Heesang Han, A. Lynn Abbott, Abhijit Sarkar

Recent advances in Multimodal Large Language Models (MLLMs) have triggered the development of end-to-end MLLMs for autonomous driving. However, the main emphasis to date has been for MLLMs using 2D images and videos. In contrast, this paper considers MLLM effectiveness using 3D sensors, particularly LiDAR and stereo cameras. LiDAR presents unique challenges to integration within an MLLM, largely because of data sparsity and lack of a grid structure for the data. For similar reasons, fusion of camera and LiDAR data within an MLLM pipeline is also uncommon. However, most autonomous systems rely on LiDAR-based sensing, and incorporating 3D data has been proven to improve performance in traditional 3D scene perception tasks. This paper presents D3VL, a novel MLLM framework that integrates 2D and 3D time-series data in a single but simple architecture. The model aims to answer questions involving traffic scene understanding and safety. D3VL shows an 11% improvement in the KITTI Question-Answering (QA) dataset compared to baseline methods in processing 2D and 3D time-series data. This paper further introduces the Waymo QA dataset extension, which assesses models' capabilities in processing 3D and time-series data under diverse driving conditions. D3VL implementation code and WaymoQA extension can be found on our supplemental website: https://automotivesafety-lvlm.github.io

arXiv

Geospatial Diffusion-based Evolution Synthesis (GeoDES) for Storm-Centered Weather Augmentation

Sonia Cromp, Satya Sai Srinath Namburi GNVV, Youran Wang, Grace Kisslinger, Frederic Sala, James Booth, Allegra LeGrande

While machine learning-based weather models hold significant promise, they struggle to predict the detailed structure of large-scale weather systems such as cyclonic storms. Regional models are constrained by limited historical records within fixed geographic boundaries, while global models are computationally expensive and often operate at resolutions too coarse to capture fine-grained storm dynamics. To bridge this gap, we introduce the Geospatial Diffusion-based Evolution Synthesis (GeoDES) model, a custom image-to-video diffusion model. By focusing generation strictly on the evolving storm structure, GeoDES synthesizes physically consistent, high-fidelity weather events suitable for stress-testing forecast models and expanding meteorological datasets. Evaluations demonstrate that GeoDES outperforms prior methods on key metrics, achieving $52\%$ lower Peak Vorticity Error and $8\%$ higher Anomaly Correlation Coefficient than the next strongest methods on the North Atlantic test set.

Keep browsing

More papers from top AI conferences

30+ top conferences, by conference, year, and topic.