AI conference papers

AI conference paper library

Search papers from 30+ top conferences, including NeurIPS, ICLR, ICML, CVPR, ACL and AAAI. Filter by keyword, then open any paper for its abstract, authors and links.

arXiv

Ground4D: Consistency-Aware 4D Reconstruction from Monocular Video

Qing Zhao, Weijian Deng, Pengxu Wei, Liang Lin

Learning a 4D scene representation from a single monocular video that supports dynamic novel-view synthesis while maintaining faithful geometry over time remains challenging. Dynamic Gaussian Splatting achieves strong rendering performance through photometric optimization, yet does not explicitly enforce multi-view geometric consistency. In contrast, 3D foundation models recover coherent scene geometry and camera motion, but their point-based outputs are not designed for photorealistic rendering. We propose Ground4D, a geometry-grounded framework built on two stages. First, we perform geometry initialization via 3D foundation models, leveraging VGGT in a training-free manner to reconstruct multi-view-consistent 3D geometry and camera poses from monocular video. The recovered geometry provides a structured and reliable initialization for dynamic Gaussian representations. Second, we conduct geometry-consistency-aware refinement via dynamic Gaussian Splatting, optimizing the representation through differentiable rendering while maintaining multi-view geometric consistency across both observed and synthesized viewpoints. Furthermore, Ground4D inherently models the continuous 4D dynamics of the scene, naturally supporting rendering at arbitrary timestamps. By integrating foundation-level geometric priors into dynamic Gaussian optimization, Ground4D achieves stronger reconstruction fidelity and rendering performance, underscoring the role of geometry-grounded constraints in robust 4D scene modeling.

arXiv

Enhancing MedSAM with a Lightweight Box Predictor for Medical Image Segmentation

Amirhossein Movahedisefat, Amirreza Fateh, Mohammad Reza Mohammadi

Semantic segmentation in medical imaging is a critical yet challenging task due to data scarcity and high variability across modalities. While foundation models like the Segment Anything Model (SAM) show promise, they often struggle with medical images without specific adaptation. Moreover, point prompts, despite being the most natural form of user interaction, provide insufficient spatial context for reliable segmentation, particularly when target structures are irregular or poorly contrasted. In this paper, we propose an enhanced segmentation framework that integrates a lightweight Box Predictor module into the MedSAM architecture. The Box Predictor estimates an approximate bounding box from a single user click using localized image embedding features, providing spatial guidance that reduces the ambiguity of point prompts, while introducing only 1.6M additional parameters and negligible inference overhead. We introduce a two-stage training pipeline where the Box Predictor is trained independently before being integrated into MedSAM. To validate the generalization capability of our method, we conduct extensive evaluations on four diverse datasets (FLARE22, BRISC, BUSI, LungSegDB) spanning distinct imaging modalities, including CT, MRI, and Ultrasound. Our method improves segmentation accuracy and robustness across varied anatomical structures and imaging domains, achieving Dice scores of 0.89 (BUSI), 0.93 (FLARE22), 0.88 (BRISC), and 0.98 (LungSegDB). Code is available at https://github.com/Amirhosseinmovahedi/MedSAM-BoxPredictor

arXiv

LAGS: Low-Altitude Gaussian Splatting with Groupwise Heterogeneous Graph Learning

Yikun Wang, Yujie Wan, Wei Zuo, Shuai Wang, Yik-Chung Wu, Chengzhong Xu, Huseyin Arslan

Low-altitude Gaussian splatting (LAGS) facilitates 3D scene reconstruction by aggregating aerial images from distributed drones. However, as LAGS prioritizes maximizing reconstruction quality over communication throughput, existing low-altitude resource allocation schemes become inefficient. This inefficiency stems from their failure to account for image diversity introduced by varying viewpoints. To fill this gap, we propose a groupwise heterogeneous graph neural network (GW-HGNN) for LAGS resource allocation. GW-HGNN explicitly models the non-uniform contribution of different image groups to the reconstruction process, thus automatically balancing data fidelity and transmission cost. The key insight of GW-HGNN is to transform LAGS losses and communication constraints into graph learning costs for dual-level message passing. Experiments on real-world LAGS datasets demonstrate that GW-HGNN significantly outperforms state-of-the-art benchmarks across key rendering metrics, including PSNR, SSIM, and LPIPS. Furthermore, GW-HGNN reduces computational latency by approximately 100x compared to the widely-used MOSEK solver, achieving millisecond-level inference suitable for real-time deployment.

arXiv

Which Workloads Belong in Orbit? A Workload-First Framework for Orbital Data Centers Using Semantic Abstraction

Durgendra Narayan Singh

Space-based compute is becoming plausible as launch costs fall and data-intensive AI workloads grow. This paper proposes a workload-centric framework for deciding which tasks belong in orbit versus terrestrial cloud, along with a phased adoption model tied to orbital data center maturity. We ground the framework with in-orbit semantic-reduction prototypes. An Earth-observation pipeline on Sentinel-2 imagery from Seattle and Bengaluru (formerly Bangalore) achieves 99.7-99.99% payload reduction by converting raw imagery to compact semantic artifacts. A multi-pass stereo reconstruction prototype reduces ~306 MB to ~1.57 MB of derived 3D representations (99.49% reduction). These results support a workload-first view in which semantic abstraction, not raw compute scale, drives early workload suitability.

arXiv

Act, Think or Abstain: Complexity-Aware Adaptive Inference for Vision-Language-Action Models

Riccardo Andrea Izzo, Gianluca Bardaro, Matteo Matteucci

Current research on Vision-Language-Action (VLA) models predominantly focuses on enhancing generalization through reasoning techniques. While effective, these improvements increase computational complexity and inference latency. Furthermore, these mechanisms are typically applied indiscriminately, wasting resources on trivial tasks while failing to provide the uncertainty estimation necessary to prevent catastrophic failure on out-of-distribution scenarios. Inspired by human cognition, we propose an adaptive framework that dynamically routes VLA execution based on the complexity of the perceived state. Our approach transforms the VLA's vision-language backbone into an active detection tool by projecting latent embeddings into a set of parametric and non-parametric estimators. This allows the system to execute known tasks immediately (Act), reason about ambiguous scenarios (Think), and preemptively halt execution when encountering physical or semantic anomalies (Abstain). We find that a Gaussian Mixture Model fitted to fused vision-language embeddings provides the most reliable task-complexity signal, combining visual novelty with instruction context and cross-modal compatibility. Evaluated on the LIBERO and LIBERO-PRO benchmarks as well as on a real robot, our fused configuration achieves up to 87.5% F1-score across two VLA backbones (SmolVLA and $π_0$), retains 83% with as little as 5% of training data, and surpasses state-of-the-art failure detectors.

arXiv

Parallel Swin Transformer-Enhanced 3D MRI-to-CT Synthesis for MRI-Only Radiotherapy Planning

Zolnamar Dorjsembe, Hung-Yi Chen, Furen Xiao, Hsing-Kuo Pao

MRI provides superior soft tissue contrast without ionizing radiation; however, the absence of electron density information limits its direct use for dose calculation. As a result, current radiotherapy workflows rely on combined MRI and CT acquisitions, increasing registration uncertainty and procedural complexity. Synthetic CT generation enables MRI only planning but remains challenging due to nonlinear MRI-CT relationships and anatomical variability. We propose Parallel Swin Transformer-Enhanced Med2Transformer, a 3D architecture that integrates convolutional encoding with dual Swin Transformer branches to model both local anatomical detail and long-range contextual dependencies. Multi-scale shifted window attention with hierarchical feature aggregation improves anatomical fidelity. Experiments on public and clinical datasets demonstrate higher image similarity and improved geometric accuracy compared with baseline methods. Dosimetric evaluation shows clinically acceptable performance, with a mean target dose error of 1.69%. Code is available at: https://github.com/mobaidoctor/med2transformer.

arXiv

Infinite-Precision Autoregressive Modeling for Vector Graphics and Layouts

Yeonsang Shin, Insoo Kim, Bongkeun Kim, Keonwoo Bae, Bohyung Han

While Transformer-based autoregressive models excel in data generation, their token discretization strategy inherently limits their precision in continuous domains. We analyze the scalability limitations of existing discretization-based approaches for generating hybrid discrete-continuous sequences, particularly in high-precision domains such as logos, layouts, and semiconductor circuit designs, where precision loss potentially leads to visual artifacts, aesthetic degradation, and even functional failure. To address the challenge, we propose a novel unified framework that jointly models discrete and continuous values for variable-length sequences. Our approach employs a hybrid approach that combines categorical prediction for discrete values with diffusion-based modeling for continuous values, incorporating two key technical components: an end-of-sequence (EOS) logit adjustment mechanism that uses an MLP to dynamically adjust EOS token logits based on sequence context, and a length regularization term integrated into the loss function. Additionally, we present ContLayNet, a large-scale benchmark comprising 334K high-precision semiconductor layout samples with specialized evaluation metrics that capture functional correctness, where precision errors significantly impact performance. Experiments on multiple domains show that our approach achieves higher-fidelity hybrid vector representations than discretization-based and fixed-schema baselines, while effectively scaling to high-precision generation.

arXiv

From Camera-Based Sensing to Reasoning: A Comprehensive Review Toward Proactive Vulnerable Road User Safety

Shucheng Zhang, Yan Shi, Bingzhang Wang, Yuang Zhang, Muhammad Monjurul Karim, Kehua Chen, Chenxi Liu, Mehrdad Nasri, Yinhai Wang

Ensuring the safety of vulnerable road users (VRUs), such as pedestrians and cyclists, remains a critical challenge, as conventional infrastructure-based measures are often insufficient in dynamic urban environments. Recent advances in learning-based visual sensing systems have created new opportunities for more adaptive and context-aware VRU protection. This paper presents a comprehensive review of camera-based methods for proactive VRU safety, with a focus on developments over the past five years. Unlike prior surveys that primarily emphasize detection, we organize the literature into three interconnected components: visual perception (detection and classification), motion modeling (tracking and trajectory prediction), and behavior understanding (intent recognition and reasoning). These components form a unified hierarchical pipeline that enables early risk anticipation and timely intervention. Furthermore, this survey systematically incorporates emerging AI paradigms, including Vision Transformers (ViTs), Large Language Models (LLMs), and diffusion models, highlighting their roles in representation learning, uncertainty modeling, and semantic reasoning. Finally, we identify four key challenges specific to VRU safety, including data scarcity, behavioral uncertainty, edge deployment efficiency, and real-world sensing constraints, and discuss corresponding research directions. This work provides a unified foundation for the development of reliable, scalable, and deployable VRU safety systems.

arXiv

Improving Adversarial Robustness of Zero-Shot CLIP with Confidence-Aware Weighting

Nikoo Naghavian, Mostafa Tavassolipour

Vision-language models such as CLIP demonstrate impressive zero-shot generalization but remain highly vulnerable to adversarial attacks. Prior adversarial methods treat all samples equally in the loss function, despite the fact that not all inputs contribute equally to adversarial vulnerability. Some samples retain high confidence even under large perturbations, while others are unstable and change predictions with minor noise. To address this, we propose Confidence-Aware Weighting (CAW) to enhance the zero-shot robustness of vision-language models. CAW introduces two key components: (1) a Confidence-Aware Loss that prioritizes uncertain adversarial examples by scaling the KL divergence between clean and adversarial predictions, and (2) a Feature Alignment Regularization that preserves semantic consistency by minimizing the distance between frozen and fine-tuned image encoder features on adversarial inputs. Together, these components improve both clean and robust accuracy without compromising generalization. Extensive experiments on TinyImageNet and 14 additional datasets show that CAW outperforms recent state-of-the-art methods such as PMG-AFT and TGA-ZSR under strong adversarial attacks like AutoAttack, while also being more memory-efficient.

arXiv

MaskAttn-SDXL: Controllable Region-Level Text-To-Image Generation

Yu Chang, Jiahao Chen, Anzhe Cheng, Paul Bogdan

Diffusion models have achieved strong results in text-to-image generation, but important limitations remain as prompts become more structured and multi-object. On the architecture side, U-Net backbones are efficient and stable, yet their locality makes global coordination harder, while Transformer-based diffusion models improve global interactions but at substantially higher compute and memory cost. In parallel, compositional reliability remains weak: models often mix attributes across objects, violate spatial relations, or omit requested entities, and these errors are not reliably reflected by global metrics such as FID or CLIP-based scores. To address these issues without changing the SDXL pipeline, we propose MaskAttn-SDXL, a plug-in module that injects token-conditioned spatial gating into cross-attention logits before softmax. The gating sparsifies token-to-location interactions to suppress irrelevant bindings while preserving the pretrained backbone and standard sampling process, requiring no external supervision or inference-time editing.

arXiv

GazeLT: Visual attention-guided long-tailed disease classification in chest radiographs

Moinak Bhattacharya, Gagandeep Singh, Shubham Jain, Prateek Prasanna

In this work, we present GazeLT, a human visual attention integration-disintegration approach for long-tailed disease classification. A radiologist's eye gaze has distinct patterns that capture both fine-grained and coarser level disease related information. While interpreting an image, a radiologist's attention varies throughout the duration; it is critical to incorporate this into a deep learning framework to improve automated image interpretation. Another important aspect of visual attention is that apart from looking at major/obvious disease patterns, experts also look at minor/incidental findings (few of these constituting long-tailed classes) during the course of image interpretation. GazeLT harnesses the temporal aspect of the visual search process, via an integration and disintegration mechanism, to improve long-tailed disease classification. We show the efficacy of GazeLT on two publicly available datasets for long-tailed disease classification, namely the NIH-CXR-LT (n=89237) and the MIMIC-CXR-LT (n=111898) datasets. GazeLT outperforms the best long-tailed loss by 4.1% and the visual attention-based baseline by 21.7% in average accuracy metrics for these datasets. Our code is available at https://github.com/lordmoinak1/gazelt.

arXiv

Spectral Principal Paths: A Spectral Perspective on Linear Representation Formation in LLMs

Bowei Tian, Xuntao Lyu, Meng Liu, Hongyi Wang, Ang Li

High-level representations have become a central focus in enhancing AI transparency and control, shifting attention from individual neurons or circuits to structured semantic directions that align with human-interpretable concepts. While the Linear Representation Hypothesis (LRH) suggests that such directions emerge in representations, it remains unclear how these representations originate and why they become increasingly stable across layers. To solve this issue, we introduce the Input-Space Linearity Hypothesis, positing that concept-aligned directions originate in the input space and are steadily maintained with increasing depth. We then propose the Spectral Principal Path (SPP) framework, which formalizes how deep networks progressively distill linear representations along the spectral principal directions. We provide rigorous stability guarantees for the SPP based on the Wedin $\sinΘ$ perturbation theorem, identifying testable conditions, including spectral gap and context incoherence, that jointly ensure layer-wise directional preservation. By bridging theoretical analysis with empirical evidence, this work identifies a spectral view of how linear representations arise in LLMs, and suggests potential implications for concept-level controllable, robust, and coherent approaches to fairness and transparency in modern AI systems.

arXiv

Shift-Aware Calibration for Fine-Tuned CLIP: Leveraging Image-Text Alignment

Song-Lin Lv, Yu-Yang Chen, Zhi Zhou, Lan-Zhe Guo

Vision-language models (VLMs), such as CLIP, adapt effectively to downstream tasks through prompt tuning, but fine-tuning can misalign predictive confidence and accuracy, particularly on unseen classes. Existing VLM-specific calibration methods mainly rely on textual features of train classes, limiting their applicability across class and distribution shifts. We propose \textbf{Shift-Aware Calibration (SAC)}, a training-free, sample-wise method that uses the discrepancy between the output logits of the original and fine-tuned CLIP as a calibration signal. We explicitly call this quantity \emph{logit shift}, since it is measured in prediction space rather than between hidden representations. SAC maps this logit shift to a positive scaling factor that adjusts confidence while preserving the predicted class. Across 11 datasets and five fine-tuning methods, SAC improves calibration on train and unseen classes and under cross-dataset and domain-generalization evaluation. Mutual Information (MI) analysis shows a statistical dependency between the logit-shift signal and ECE across methods and datasets, providing empirical evidence that the signal used by SAC is informative for calibration and supporting the effectiveness of its shift-aware design.

arXiv

Mamba-CL: Optimizing Selective State Space Model in Null Space for Continual Learning

De Cheng, Yue Lu, Lingfeng He, Shizhou Zhang, Xi Yang, Nannan Wang, Xinbo Gao

Continual Learning (CL) aims to equip AI models with the ability to learn a sequence of tasks over time, without forgetting previously learned knowledge. Recently, State Space Models (SSMs), particularly the Mamba model, have achieved notable success in computer vision. Building on the strengths of SSMs, this study explores leveraging the Mamba model for CL. Therefore, we introduce Mamba-CL, a framework that continuously fine-tunes the core SSMs of the large-scale Mamba foundation model by updating parameters orthogonal to the feature subspace of previous tasks. This approach theoretically guarantees the consistency objective aiming to preserves consistent output for each SSM module across both previous and current tasks, so as to overcome catastrophic forgetting issue. Specifically, we achieve this goal by deducing the overall consistency constraints on four key time-invariant parameters in the Mamba model, streamlining its recurrent state-space structure and non-linear discretization process in SSM. In practice, we apply the null-space projection to efficiently implement the orthogonality within Mamba model. Extensive experiments on four class-incremental benchmarks demonstrate the effectiveness of Mamba-CL for anti-forgetting, achieving superior performances to state-of-the-art methods. Code is available in the supplementary materials.

arXiv

Phenology-based learning framework for yield estimation and harvest forecasting of raspberry fruits

Parham Jafary, Lesley G. Campbell, Anna Bazangeya, Michelle Pham, Sajad Saeedi, Kourosh Zareinia, Habiba Bougherara

The future of agriculture is intertwined with automation. Accurate fruit detection, yield estimation, and harvest time prediction are crucial for efficient supply chain management by optimizing resources and logistic utilization. Computer vision can automate these tasks to reduce labour costs and improve efficiency by training deep learning models on appropriate data to perform knowledge-based tasks. Although fruit detection has been the focus of literature, yield prediction and harvest time estimation remain practical challenges for farmers. This is particularly important for high-value, highly perishable crops, such as raspberries, where contractual obligations require precise harvest timing. This paper addresses this gap by providing a learning-based framework for raspberry maturity detection and harvest time estimation. For this end, a phenology study of raspberry plants from three different cultivars was carried out, and seven development stages were identified based on the BBCH-scale; for the first time, a phenology-based dataset of developing raspberry flowers and fruit was curated and made available publicly, which contains 1,853 high-resolution images and 6,907 manually labelled annotations. A comprehensive benchmark was developed from state-of-the-art object detection models, and finally, a tailored deep learning model capable of real-time inference was established that achieved 92.2% detection accuracy in the vertical farm field test. This paper provides a model that can estimate raspberry yield 28 to 33 days in advance of harvesting, a dataset that can be used by other researchers for object detection and segmentation model development, and a framework that can be extended for precise yield and harvest time estimation.

arXiv

Evaluation of Blood Vessel Segmentation Methods on Hard-to-Detect Vascular Structures

João Pedro Parella, Matheus Viana da Silva, Cesar Henrique Comin

Due to the intricate structure of vascular trees, minor segmentation errors can significantly alter connectivity patterns and increase variability in extracted morphological properties. Global metrics such as the Dice coefficient, precision, and recall often overlook inaccuracies in specific regions of a sample. To address this, we define a Local Vessel Salience (LVS) index to quantify the difficulty of identifying specific vessel segments. This index is used to evaluate the performance of 16 segmentation methods across six widely used 2D datasets. The LVS index is calculated for each vessel pixel by comparing local vessel intensity against the surrounding background. We introduce a metric termed mean Low-Salience Recall (mLSR) to quantify how effectively algorithms recover hard-to-detect vessels. Furthermore, we propose a proof-of-concept data augmentation procedure guided by the LVS index aimed at improving neural network segmentation performance. Our findings demonstrate that segmentation performance strongly correlates with LVS, revealing systematic errors in vessels with low salience. The mLSR across all evaluated methods was significantly lower than standard recall values, with a typical performance decrease of 20 percentage points. In benchmark datasets such as DRIVE and OCTA-500, mLSR values for the top-performing methods were approximately 58%. The developed methodology provides a quantitative basis for the design of segmentation algorithms with improved sensitivity to hard-to-detect vessels and enhanced capabilities for preserving vascular connectivity.

arXiv

When Activation Oracles Learn Not to Read: Concept-Specific Blind Spots in Fine-Tuned Oracles

Tobias Bersia, Tatiana Gaintseva

Activation Oracles (AOs) are language models trained to answer natural-language questions about another model's internal activations. They offer a flexible interface for reading hidden information from model states, especially when relevant information is internally represented but absent or incomplete in visible behavior. However, AOs are themselves learned systems: their answers are shaped by training data, objectives, and learned reporting behavior, rather than being neutral readouts of represented information. We study this in a controlled Taboo Word Guessing setting, where subject models are fine-tuned to internally use a hidden concept while avoiding direct disclosure. Contrary to the expectation that an AO trained on such a subject becomes a specialist reader, we find that fine-tuned AOs can become concept-specific anti-readers: they selectively fail to recover the concept persistently present during their own training. This failure is not simply explained by absence of the concept from the subject or oracle representations: the target remains decodable inside the oracle, while LogitLens and layer-ablation analyses indicate that the failure arises in the AO readout pathway. Our results show that behavioral leakage, representation-level decodability, and AO-verbalizability can come apart, raising a reliability concern for learned interpretability interfaces.

arXiv

Explaining BiomedCLIP with Weighted Banzhaf Interactions Supported by Tree-Gram Parsing

Jakub Rymarski, Adam Rempała, Bartłomiej Sobieski, Przemysław Biecek

Vision-Language Models (VLMs) are demonstrating significant capabilities in medical tasks like radiology analysis, yet providing faithful and interpretable explanations remains a key consideration for their responsible deployment in clinical settings. However, existing explanation methods, such as the widely used FIxLIP framework, often struggle with the fine-grained nature of modern tokenizers. The tokenization problem fragments clinical concepts---splitting terms like "saddle embolus" into scattered, meaningless subwords---which leads to noisy, semantically incoherent cross-modal attributions. Such fragmentation also results in a combinatorial explosion of interaction possibilities, obscuring the model's true reasoning. To address this, we introduce ParseFIxLIP, an extension that incorporates the Tree-Gram Parsing into the Banzhaf interaction game used by FIxLIP. This semantically informed strategy utilizes dependency parsing trees to define explanation players by grouping related text tokens into semantically coherent units. Our smart_depth grouping strategy, merging tokens according to spaCy token dependency tree, successfully mitigates concept fragmentation, yielding substantially more interpretable cross-modal interactions by unifying complex medical concepts. Quantitatively, while baselines struggled with the high dimensionality of long captions, our parsing approach maintained statistical robustness and semantic parsimony. Qualitative analysis on BiomedCLIP, validated on medical imagery (ROCOv2) and general examples, confirms that the approach accurately captures the synergistic influence of grouped words on model predictions. In conclusion, our work offers intuitive and clinically relevant insights into VLM decision-making, fulfilling the critical need for coherent explanations in the medical domain.

arXiv

Joint Optimization for Greedy Longest-match Tokenization

Adhiraj Singh, Deepanshu Mody, Ghina Al Shdaifat, Hamza Alshamy, Adam Wiemerslage, Varshini Reddy, Craig W. Schmidt

Recent work has shown that subword vocabularies can be trained to optimize compression for a specific inference rule rather than relying on greedy heuristics such as Byte Pair Encoding (BPE). We extend this approach to greedy left-to-right longest-match decoding, the fast and widely used inference rule underlying WordPiece. We introduce Joint Optimization for Greedy Longest-Match Tokenization (JOLT), which formulates vocabulary learning as an integer program over vocabulary-selection and segmentation-choice variables. Greedy-consistency constraints ensure that each optimized segmentation exactly matches the segmentation produced by longest-match decoding under the selected vocabulary, aligning the training objective with deployment-time tokenization. To scale the optimization, we solve a linear programming relaxation and selectively introduce higher-order segmentations only for unresolved pretokens. The resulting relaxation is nearly integral: rounded solutions fall within 0.008 - 0.176 % of the LP lower bound on the training scope. The bound also shows that BPE is already within 1 - 2 % of the best achievable compression under greedy longest-match decoding, while JOLT closes 89.6 - 99.4 % of the remaining gap. On held-out validation data across four training scopes and vocabulary sizes of 32,000 and 64,000, JOLT produces up to 0.78 % fewer tokens than BPE, with improvements generally increasing as the training scope grows. These results demonstrate that inference-aligned vocabulary optimization can recover most of the limited compression headroom left by BPE while providing a certificate of near-optimality.

arXiv

Hallucination Rates in Language Generation

Debmalya Panigrahi, Fan Wei, Ian Zhang

Language generation in the limit is an elegant model introduced by Kleinberg and Mullainathan [KM24] to formally study language generation by an algorithm that learns solely based on example strings. In this model, an algorithm is said to correctly generate from a language if it never makes an error after some finite time. In contrast, even sophisticated language models are known to regularly hallucinate in practice. In this paper, we initiate the study of language generation in the limit with (infinite) hallucination, i.e., the algorithm may generate incorrect strings infinitely often, but the errors occur at a limited rate (possibly even with 0-measure). We first show that hallucination, even at rate 0, makes generation in the limit strictly more powerful: there are language collections that cannot be generated with finite error but can be generated with infinite error, even when errors occur on a 0-measure set of time-steps. Furthermore, while all countable collections are generatable with finite error, we show a strict hierarchy of (uncountable) language collections characterized by the hallucination rate. This hierarchy extends to breadth, the fraction of the target language generated. While all countable collections can attain the optimal breadth of 1/2 [KW26b], we show strict separation at every breadth and hallucination rate. Finally, we study generation in the limit without repetition, where the algorithm may not repeat strings. This lets us compare the sets of correct and incorrect strings generated, rather than the fractions of correct and incorrect time-steps. Once again, we demonstrate a strict hierarchy at every hallucination rate and breadth. Taken together, these results reveal rich structure in language collections generatable in the limit with hallucination and establish hallucination rate as an important parameter in the theoretical study of language generation.

Keep browsing

More papers from top AI conferences

30+ top conferences, by conference, year, and topic.