NeurIPS 2026 论文列表

Visual Text Comprehension (VTC) renders text into images for a vision-language model (VLM) to read, sidestepping LLM context-window limits and powering applications from long-page OCR to multi-page memory QA. Yet existing VTC pipelines treat rendering and layout as a fixed, content-agnostic preprocessing step, and offer little mechanistic understanding of how VLMs internally process visualized text. Through a focused empirical study on VTC QA tasks, we find that VLMs exhibit a localization-without-utilization regime: evidence-localizing attention emerges sharply in the middle-to-late layers and is largely decoupled from answer correctness, yet simply enlarging the localized spans on the rendered page recovers a large fraction of the failures. Building on these observations, we propose AGAR (Attention-Guided Adaptive Rendering), a training-free, model-agnostic method that leverages a VLM's own middle-to-late layer attention to identify the top-K important visual patches, maps them back to word spans, and re-renders the page with those spans enlarged before re-inferring the answer. Extensive experiments across nine VTC benchmarks (short-form, long-context, and multi-page memory QA) and four VLM backbones show that AGAR (i) consistently improves off-the-shelf VLMs as a plug-and-play enhancement, (ii) composes with VLM post-training to yield further gains, and (iii) remains robust under both visual- and text-side input degradation.

Concept Bottleneck Models (CBMs) provide interpretability but typically incur a penalty in accuracy. Recent CBMs rely on LLM-supplied concept vocabularies that are misaligned with the underlying image evidence and obscure dataset-specific discriminative signals. Augmenting these representations with task-specific linear probes recovers little of the lost accuracy. Methods that instead learn concepts directly from data also fall short of the supervised CLIP linear-probe ceiling. We introduce MOSAIC (Mixtures Of Sparse Anchored Interpretable Concepts), a CBM whose concepts are discovered directly from image representations via text-anchored optimal transport, a balanced-marginal pass that softly assigns each image to a small set of learned concepts while keeping them tethered to class-text embeddings, and then named post-hoc by a VLM. Across 5 datasets, MOSAIC matches the supervised CLIP linear-probe ceiling on most dataset-backbone settings and beats raw zero-shot CLIP by +5.2 to +24.4 top-1 and class-level retrieval R@1 by +5.7 to +27.0. Beyond classification, the discrete concept basis enables a sparse per-image decomposition that exposes which features drive each prediction and aggressive bottleneck compression to as little as 2.5% of dense CLIP.

The ability to reliably distinguish human-written text from that generated by large language models is of profound societal importance. The dominant approach to this problem exploits the likelihood hypothesis: that machine-generated text should appear more probable to a detector language model than human-written text. However, we demonstrate that the token-level signal distinguishing human and machine text is non-uniform across the hidden space of the detector model, and naively averaging likelihood-based token scores across regions with fundamentally different statistical structure, as most detectors do, causes a form of Simpson's paradox: a strong local signal is destroyed by inappropriate aggregation. To correct for this, we introduce a learned local calibration step grounded in Bayesian decision theory. Rather than aggregating raw token scores, we first learn lightweight predictors of the score distributions conditioned on position in hidden space, and aggregate calibrated log-likelihood ratios instead. This single intervention dramatically and consistently improves detection performance across all baseline detectors and all datasets we consider. For example, our calibrated variant of Fast-DetectGPT improves AUROC from $0.63$ to $0.85$ on GPT-5.4 text, and a locally-calibrated DMAP detector we introduce achieves state-of-the-art performance across the board. That said, our central contribution is not a new detector, but a precise diagnosis of a significant cause of under-performance of existing detectors and a principled, modular remedy compatible with any token-averaging pipeline. This will serve as a foundation for the community to build upon, with natural avenues including richer distributional models, improved calibration strategies, and principled ensembling with hidden-space geometry signals via the full Bayes-optimal decision rule.

We introduce LibriBrain100, a large-scale MEG dataset for speech decoding designed from the ground up for reproducible, standardised evaluation. LibriBrain100 more than doubles the size of the original LibriBrain release, resulting in over 100 hours of high-quality MEG acquired while subjects listened to naturalistic continuous speech. With ~80 hours from a single subject, LibriBrain100 sets a new record for ‘deep’, within-subject neural data (8x more than the next comparable dataset and roughly 80x more than other datasets). To demonstrate the payoff of this depth-first design, we evaluate on a word-classification benchmark—an increasingly well-established stepping stone towards the open challenge of non-invasive brain-to-text decoding. Using an existing decoding model, we achieve state-of-the-art performance—validating both the quality of the recordings and the value of within-subject data at scale. Because collecting 80 hours of data per user is impractical for real-world applications, we also collected ~40 minutes of additional data from each of 32 subjects. Using the same word-classification benchmark, we demonstrate the value of ’broad’ multi-subject data: supervised fine-tuning of a pre-trained model can substantially compensate for limited per-subject data. We provide standard train, validation, and test splits, all reproducible through an open-sourced Python library that supports easy downloading, optional preprocessing, and data loading for common deep learning frameworks. In addition, the dataset and evaluation infrastructure are being released alongside an open machine-learning competition with a public leaderboard for standardised benchmarking. Ultimately, our hope is that LibriBrain100 will accelerate progress towards practical non-invasive brain-computer interfaces, capable of restoring communication to people living with severe paralysis.

LLMs are increasingly used not only to generate text from scratch, but also to polish and extend human writing. This makes AI detection a fine-grained problem: downstream users often need to understand the level and form of AI involvement in a text, rather than merely separate human from machine. Existing detectors are poorly suited to this setting because they compress the evidence into scalar scores, selected features, or surface linguistic cues, which miss the subtle patterns left by different forms of AI involvement. We propose ***LensCT***, which makes a frozen LLM's internal predictive behavior visible as a tomogram: an image-like map of how the model responds across tokens and layers. The tomogram captures both temporal patterns along the input sequence and hierarchical patterns across transformer layers, providing a richer view of AI involvement than compressed detector scores. A vision backbone then reads this tomogram to classify the level of AI involvement. On a **237,500**-sample English benchmark spanning five domains and six frontier LLM generators, ***LensCT*** achieves **0.990** in-distribution macro-AUROC, reducing error by **28.6\%** over the strongest baseline. Across **697,500** total evaluation samples, ***LensCT*** consistently outperforms prior detectors under cross-generator, cross-domain, cross-language, and adversarial shifts, showing that temporal-hierarchical tomograms provide a robust and transferable signal for fine-grained AI-involvement detection.

The choice of language model pretraining objective was once contested among causal language modeling (CLM), masked LM, T5 span corruption, and UL2-style mixtures of denoisers. It was settled in favor of CLM during an era when high-quality text was abundant relative to compute. That era has ended. Compute now routinely outpaces the supply of unique data, and modern pipelines rely heavily on data repetition both during pretraining on curated corpora and during midtraining on small domain corpora. We revisit the objective question in this data-constrained regime. Across a large sweep over models from 15M to 1B parameters, unique data budgets up to 6B tokens, and across repetition budgets, we compare CLM against three denoising objectives: fill-in-the-middle (FIM), T5-style span corruption, and a UL2-style mixture-of-denoisers. All four scale similarly when every training token is unique, but they differ significantly in their tolerance for repeated data. Denoising objectives, which naturally augment language data on every pass, accumulate substantially less overfitting cost than CLM. We find that span corruption and mixture-of-denoisers are the most robust. We then validate the practical payoff in a midtraining setting. We continue web-text pretrained checkpoints on a limited mixture of web-text and math data. We observe denoising midtraining objectives outperform CLM on GSM8k, and the gap widens with increased repetition. Choosing a denoising objective is a simple, complementary lever to model and data scaling for practitioners working under tight data budgets.

Recent work has demonstrated surprisingly good performance of pre-trained LLMs on regression tasks (for example, time-series prediction), with the ability to incorporate expert prior knowledge and the information contained in textual metadata. However we observe major error cascades even in short sequences $\lesssim 100$ points; these models are also computationally intensive and difficult to parallelise. Marginal LLM predictions do not suffer this issue and are trivially parallelised, but can predict over-broad densities. To address this, we propose combining these densities with a lightweight (diffusion-based) neural process. We show that this combination leads to better-calibrated predictions overall, outputs locally consistent trajectories, and leads to text-conditioned function space selection in the meta-learner. As part of this work we propose a gradient-free (and non-Monte Carlo) method for sampling from a product-of-experts of a score model and an `expert' (here the LLM predictive densities). We believe this general method is of independent interest as it is applicable whenever an expert can be convolved with a Gaussian in closed form.

Metric-induced discrete flow matching (MI-DFM) exploits token-latent geometry for discrete generation, but its practical use is limited by two issues: heuristic schedulers requiring hyperparameter search, and finite-step path-tracking error from its first-order continuous-time Markov chain (CTMC) solver. We address both issues. First, we derive a kinetic-optimal scheduler for prescribed scalar-parameterized probability paths, and instantiate it for MI-DFM as a training-free numerical schedule that traverses the path at constant Fisher–Rao speed. Second, we introduce a finite-step moment correction that adjusts the jump probability while preserving the CTMC jump destination distribution. We validate the resulting method, GibbsTTS, on codec-based zero-shot text-to-speech (TTS). Under controlled comparisons with a unified architecture and large-scale dataset, GibbsTTS achieves the best objective naturalness and is preferred in subjective evaluations over masked discrete generative baselines. Additionally, in comparison with the evaluated state-of-the-art TTS systems, GibbsTTS shows strong speaker similarity, achieving the highest similarity on three of four test sets and ranking second on the fourth. Demo and code are available in the supplementary materials.

Large language models (LLMs) provide a powerful reasoning backbone for speech understanding, but integrating continuous acoustic signals into a frozen LLM remains challenging. Existing speech-to-LLM interfaces typically operate at two extremes: either enforcing near-discrete token alignment, which benefits transcription but loses paralinguistic information, or learning unconstrained continuous representations, which can drift away from the LLM’s input space and degrade autoregressive decoding. In this work, we propose Convex Gate (C-Gate), a speech–LLM bridge that constrains all speech representations to lie within the LLM’s input embedding manifold with an architectural convex-hull constraint. Concretely, each frame is represented as a convex combination of token embeddings, ensuring compatibility with the pretrained LLM while preserving continuous expressivity. Across automatic speech recognition (ASR) and emotion recognition, C-Gate achieves strong joint performance, improving LibriSpeech WER by up to 48.7\% relative while matching or exceeding single-task emotion accuracy. Beyond performance, our analysis reveals a key insight: information is not carried by discrete token identities, but by time-resolved trajectories in the embedding space. Causal interventions confirm that both the trajectory structure and alignment to the pretrained embedding manifold are critical for performance. These results suggest that geometry, rather than token discreteness, is the fundamental design factor in speech–LLM interfaces, and provide a controlled regime for studying multimodal integration in frozen LLMs. We release the checkpoint, per-sample outputs, mechanism dumps, and intervention suite for replication.

In self-supervised pretraining for Handwritten Text Recognition (HTR), pixel reconstruction methods tend to outperform contrastive ones, a pattern that sits awkwardly with recent evidence that pixel reconstruction yields uninformative features for natural-image classification. We argue this discrepancy is not an accident, but a consequence of how HTR signal is distributed in pixel space. By probing the input distribution directly, we show that HTR's discriminative content is concentrated in the high-variance pixel subspace and is essentially absent from the low-variance one, the inverse of the structure observed for image classification. Under this view, the right SSL family becomes predictable: objectives that preserve high-variance pixel content should transfer best. We test this prediction across six SSL methods spanning three families (pixel-grounded MIM, JEPA-style, contrastive image--image and image--text), under matched encoder, data, and evaluation protocols on six handwriting benchmarks across five languages. Pixel-grounded SSL produces the lowest CER on every benchmark and every probe, exposes per-position character information that other families recover only via the readout, and is the only family that benefits from real-data pretraining. A geometric property of the encoder, its alignment with the high-variance pixel subspace, predicts CER within every SSL method we test. With a pretrained LLM decoder, a frozen pixel-grounded encoder is already competitive with fully fine-tuned supervised baselines, and full fine-tuning beats them on the mean and ranks first or second on every benchmark. These results challenge the prevailing view that pixel reconstruction wastes capacity on irrelevant detail: whether reconstruction is wasted depends on where the discriminative signal lives in the input.

Diffusion-based image restoration models produce visually compelling results but systematically hallucinate text---generating characters that appear sharp and legible yet are factually incorrect. As text fidelity emerges as a key requirement for real-world image restoration, this failure mode poses a fundamental barrier to wide deployments of diffusion-based models: a model that replaces degraded-but-faithful text with sharp-but-wrong text misleads users and erodes trust. Recent text-aware models mitigate this by feeding text extracted from intermediate outputs or low-resolution (LR) images into the prompt, but they provide no mechanism to detect or correct the hallucinations that still commonly arise. In this work, we present a systematic study on how to improve text restoration during denoising, demonstrating that standard global conditioning alone is inadequate and that optical character recognition (OCR) confidence scores reliably predict word-level correctness. Grounded in these findings, we propose HalluciText, a training-free framework consisting of two complementary interventions: (i) Confidence-Weighted Classifier-Free Guidance (CW-CFG), which concentrates guidance energy on text regions weighted by OCR confidence to improve text recall; and (ii) VLM-Guided Test-Time Guidance (V-TTG), which leverages a vision-language model (VLM) to identify hallucinated regions and corrects them towards a faithful reference, improving text precision. Both operate solely on the noise prediction or latent representation, making them agnostic to the diffusion backbone and text spotting model. We demonstrate the effectiveness of HalluciText on state-of-the-art (SOTA) text-aware models, achieving up to 5.75 points of improvement in end-to-end text recognition F1 on several datasets without degrading image quality.

End-to-end speech LLMs often answer the same question less accurately from speech than from text, even when the spoken content is semantically matched to the text prompt. We study this modality gap as a layer-wise inference problem rather than a static embedding mismatch. Across four open-weight speech LLMs on SpeechMMLU and VoiceBench BBH, cross-layer Centered Kernel Alignment (CKA) with speech–text token alignment shows that speech can become geometrically text-like in middle layers while still failing to form stable late-layer answer margins. The central finding is that mid-layer alignment does not imply functional equivalence. ASR-to-LLM controls show that recognition explains part of the gap in some settings but not all of it. Matched-instance margin analysis localizes the failure to weak late-layer answer separation. Attention diagnostics further show that speech evidence remains more diffuse at the decision token. Speech-faithful temporal compaction improves BBH accuracy from 59.9% to 62.1% with tempo 1.2× and to 61.9% with silence trimming, while recovering 34.3% and 38.2% of text-correct/speech-wrong failures. These conditional repairs reveal a 2.35–2.62× asymmetry between temporal compaction and quantity-only KV merging on the same gap subset, motivating post-projection temporal abstraction rather than only input-level feature matching.

We present DuplexMotion, a streaming, full-duplex speech-and-motion model designed for dyadic interaction. To capture the continuous and reciprocal nature of human communication, this full-duplex capability empowers the agent to simultaneously perceive and generate both speech and physical motion in a streaming fashion. At its core, our method leverages the strong priors of a foundational full-duplex speech model and integrates a novel motion pathway, thereby achieving fully synchronized multi-modal interaction. Specifically, we design a dual-tower Transformer architecture that preserves the zero-shot conversational reasoning of a frozen base speech model while constructing a deeply coupled, streaming motion pathway. By introducing a unified dyadic token interleaving mechanism and guiding cross-attention via a time-aligned speech-motion RoPE, our model effectively aligns autoregressive motions with rich latent speech features. Trained on the 4,000-hour Seamless Interaction dataset, our model effectively captures cross-speaker dependencies and establishes new state-of-the-art performance across both monadic and dyadic human interaction benchmarks.

Brain-to-speech decoding aims to restore communication by translating neural activity into speech-related representations. From a modeling perspective, this task can be formulated as a weakly supervised neural sequence decoding problem, where phoneme sequences must be recovered without frame-level annotations. Existing speech decoders commonly rely on recurrent sequence models to integrate temporal context, but their dense sequential computation can limit their suitability for resource-constrained portable or implantable neuroprosthetic systems. In this study, we propose a residual spiking neural network (SNN) for phoneme-level decoding of intracortical speech signals under constrained model capacity and weak sequence-level supervision. Our approach incorporates biologically inspired fixed connection-specific delays to integrate temporal context, providing each postsynaptic unit with access to recent presynaptic spike activity without recurrent state or dense temporal convolutions. Because each directed connection is assigned a single fixed delay, the temporal field is expanded without adding trainable delay parameters. Across 500k, 2M, and 5M core-parameter budgets, fixed delays consistently improve phoneme error rate (PER), with gains saturating once the delay range covers the task-relevant temporal horizon. Boundary-aligned CTC analysis further suggests that delayed temporal context reduces local uncertainty around phoneme transitions, indicating that fixed delays support weakly supervised phoneme alignment rather than simply increasing model capacity. Compared with recurrent, convolutional, and feedforward baselines under matched core-parameter budgets, the proposed SNN achieves the best PER at 500k and 2M parameters and the second-best PER at 5M parameters, while reducing estimated operation-level energy by approximately 2-4$\times$ relative to the strongest non-spiking baselines. These results suggest that fixed-delay SNNs provide an efficient temporal modeling mechanism for brain-to-speech decoding, potentially enabling lightweight decoding for on-device speech neuroprostheses.

Recent speech brain-computer interfaces (BCIs) increasingly rely on large models and complex multi-stage pipelines for neural decoding. However, in speech BCI, where training data are scarce, noisy, and expensive to collect, the benefits of model scaling remain unclear. This work revisits a fundamental question: $\textbf{are large models truly necessary for accurate neural decoding?}$ We propose $\textbf{\textit{BEST}}$ ($\textbf{\textit{B}}$rain-$\textbf{\textit{E}}$ncoding-$\textbf{\textit{S}}$peech-$\textbf{\textit{T}}$ext), a two-stage framework for efficient neural-to-text decoding. Instead of relying on large generative models or phoneme-centric pipelines, $\textbf{\textit{BEST}}$ introduces hierarchical intermediate representations and a compact speech-oriented decoder that directly maps neural activity to text. This design preserves essential structure while significantly reducing computational cost, enabling robust generalization in low-resource settings and practical deployment under constrained resources. On the Brain-to-Text ’24 and ’25 benchmarks, $\textbf{\textit{BEST}}$ achieves word error rates (WERs) of 6.19\% and 1.62\%, espectively, using only 3.7\% of the parameters required by comparable methods. Beyond these results, we provide three key insights: (1) decoder scaling yields non-monotonic gains; (2) cross-modal alignment is more critical than model size; and (3) overfitting degards sequence-level performance. Representation analysis further reveals a consistent neural→audio→text alignment pathway.

Autoregressive language models are widely used for text evaluation, however, their left-to-right factorization introduces positional bias, i.e., early tokens are scored with only leftward context, conflating architectural asymmetry with true text quality. We propose masked reconstruction as an alternative paradigm, where every token is scored using full bidirectional context. We introduce DiffScore, an evaluation framework built on Masked Large Diffusion Language Models. By measuring text recoverability across continuous masking rates, DiffScore eliminates positional bias and naturally establishes an evaluation hierarchy from local fluency to global coherence. We further provide diagnostic tools unavailable to autoregressive frameworks: multi-timestep quality profiles that decompose scores across masking rates, and bidirectional Pointwise Mutual Information (PMI) decomposition that disentangles fluency from faithfulness. Experiments across ten benchmarks show that DiffScore consistently outperforms autoregressive baselines in both zero-shot and fine-tuned settings.

Diffusion and flow-matching based text-to-speech (TTS) models excel in naturalness but often lack explicit emotion control, as emotional signals remain entangled with speaker identity. We discover that emotion embedding emerges as a linearly decodable direction of frozen hidden states, nearly orthogonal to the direction embedding speaker identity. This inspires a plug-and-play framework DUET for emotion control over pretrained diffusion and flow-matching based TTS models. During generation, DUET unifies dual-space control to achieve fine-grained emotion intervention in a single per-step update: hidden space steering shifts generation along the target emotion direction, while mel-space guidance refines spectral details through gradients backpropagated from a differentiable vocoder. We validate DUET on five architecturally diverse pretrained TTS backbones across three datasets, where it outperforms 10 supervised state-of-the-art emotional TTS baselines across paradigms and achieves the highest human-rated emotion appropriateness. To further showcase its qualitative behavior, we deploy DUET on an Ameca humanoid robot, where it produces richly expressive emotional speech on the humanoid, demonstrating the strong potential for plug-and-play affective interaction for embodied agents. The demonstration is available at: https://anonymous.4open.science/w/duet-demo-fresh-D8E0

We present an encoding method that maps each input token to a randomized sequence of binary values (0s and 1s). By operating at the finest representational granularity and applying compression, this approach reduces the effective vocabulary to just two symbols, yielding a compact, task-agnostic representation. Across eleven main tasks spanning six modalities (human text, genetic/clinical data, programming code, system logs, biological signals, and tabular medical data), it achieves up to two orders of magnitude stronger privacy guarantees under six gradient-based attack settings compared to seven encoder baselines and four defense baselines, including up to 549× improvement on FILM and 62× on TAG. At the same time, it reduces embedding model size by up to 98%, while maintaining competitive accuracy and computational efficiency. It also improves multilingual generalization, achieving up to a 4× increase in accuracy in low-resource settings.

Simultaneous Speech Translation (SimulST) requires balancing translation quality and latency. While Large Language Models (LLMs) excel in offline translation, current LLM-based SimulST systems predominantly rely on prefix-incremental generation. This paradigm suffers from redundant Key-Value cache recomputation and necessitates external Voice Activity Detection to process unbounded streams. To overcome these bottlenecks, we propose an end-to-end interleaved read-write framework built upon an omni-modal LLM. By interleaving speech chunks and text tokens, our architecture achieves full KV cache reuse, natively supporting segmentation-free, infinite-length inference. Furthermore, to learn optimal streaming policies without rigid offline alignments, we introduce a novel Actor-Critic self-enhancement pipeline. An RNN-T Actor dynamically samples read-write trajectories, while an LLM-based Critic evaluates these paths via intrinsic semantic and latency rewards, distilling the optimal policy back to the Actor. Extensive experiments on sentence-level (FLEURS, CoVoST2) and document-level (MuST-C, ACL 60/60) benchmarks demonstrate that our model establishes new state-of-the-art quality-latency Pareto frontiers, substantially outperforming existing cascaded and static-alignment baselines.