NeurIPS 2026 论文列表

Post-hoc scoring functions for out-of-distribution (OOD) detection in transformer models are typically validated on a single architecture and a single modality, leaving cross-architecture behavior largely uncharted. We report a cross-modality empirical study of attention-head-masking-based OOD detection across more than 1,200 AUROC measurements, spanning LayoutLMv3 on document classification, ViT-B/16 on natural images, and RoBERTa-base on text. Each setting is evaluated against seventeen baselines under a uniform protocol with five fixed seeds, including the Attention Head Sensitivity Scoring (AHSS) and AHM-augmented distance methods of Constantinou et al. (2024) and the recent X-Mahalanobis from NeurIPS 2025. Paired statistical tests are run on every head-to-head claim. Within the AHM family, mahAvgAvg_AHM significantly beats X-Mahalanobis on Tobacco-Cross (Wilcoxon $p=0.031$) and on RoBERTa SST-2$\to$20-Newsgroups ($p=0.031$); on standard CIFAR vision benchmarks, X-Mahalanobis remains state-of-the-art. Three architecture-dependent phenomena surface that current evaluation practice tends to miss: Monte Carlo Dropout collapses to chance-level AUROC across all four ViT settings tested; the AHSS displacement score inverts polarity on RoBERTa, yielding a mean AUROC of 0.296 even though it functions on vision and document transformers; and on near-OOD splits where ID and OOD share visual or semantic structure, every method including the strongest baselines degrades by 25 to 42 percentage points relative to standard OOD. We release per-seed measurements, AHM masks, and code to support replication.

LLM-based peer review is now used at production scale, but rigorous evaluation remains limited. We introduce a benchmark for evaluating AI peer reviewers’ detection of injected errors and quantifying false-positive rates across error type, severity, prompting strategy, and reviewer architecture. We also present skepthical, a multi-agent AI reviewer that verifies citations by retrieving and reading cited papers, checks mathematics with computer algebra, audits numerical claims with code, and uses a cross-model ensemble for general scientific issues. On 15 astrophysics preprints containing 177 injected errors, we benchmark 12 reviewer systems: skepthical; six single-LLM reviewers from three providers, each run with free-form and structured prompts; two Claude Code reviewers using the same Claude model and tools, one with a free-form prompt and one with a structured prompt; and three external reviewers. Detection performance varies sharply by model, prompt, tool use, and error type. Under the same free-form prompt, three frontier LLMs differ by 48.6 pp in total detection rate; a structured prompt halves this spread by improving the weaker models. Within Claude Code, changing only the prompt from free-form to structured raises detection by 32.2 pp. skepthical achieves the highest overall detection rate, 70.6±6.2%, with the best performance on general issues, 82.2%, and citation errors, 46.7%. On numerical errors, it matches both GPT-5.5 configurations at 80.0%, behind structured Claude Code at 91.1%; on mathematics, its 73.8% rate overlaps with both GPT-5.5 configurations within confidence intervals. False-positive rates also vary substantially: free-form Opus 4.7 has the lowest overall rate at 1.0%, while skepthical is the only reviewer with no false-positive failure mode jointly across mathematics, numerics, and citations, yielding a 2.8% overall rate. skepthical is deployed publicly at skepthical-ai.org, and we release the evaluation set as a benchmark.

Recent audio embedding models learn general-purpose representations that transfer across clip- and frame-level tasks, yet they operate almost exclusively on single-channel signals and are blind to spatial structure. Spatial audio models capture that structure but rely on clip-level supervision or contrastive objectives, yielding embeddings unsuitable for frame-level tasks like sound event detection and localization (SELD). We argue this dichotomy is not fundamental: patch-level spatial self-supervision can jointly learn acoustic and spatial features at both temporal granularities. We introduce AudioSphere, a multi-channel masked autoencoder trained on Ambisonics recordings that reconstructs both spectral content and directional cues via a joint spatiotemporal masking strategy that masks both time-frequency patches and intensity vectors. AudioSphere supports frame-level predictions while producing strong clip-level representations via aggregation. Alongside the model, we release RealSELD, a standardized benchmark for evaluating frozen backbones on real-world multi-channel SELD. AudioSphere enables, for the first time, SELD with a frozen self-supervised backbone, while matching or exceeding prior general-purpose audio embedding models on the HEAR benchmark.

Unifying speech, sound, and music generation in one model is hindered by tradeoffs between fidelity, end-to-end training, in-context conditioning, and variable-length synthesis that no current paradigm fully resolves. To address this challenge, we present **AudioCALM**, a universal audio generation framework that extends autoregressive (AR) next-token prediction from discrete tokens to continuous audio latents: a thin flow-matching head replaces the softmax to predict rectified-flow velocities at each position, and a block-causal AR-Flow attention pattern produces arbitrary-length output. Joint training of multiple audio generation tasks faces an asymmetric text--audio mismatch: speech transcripts align to specific time spans and demand \textit{tight, time-aligned} attention, whereas sound and music captions describe only overall semantics and rely on diffuse, holistic attention; mixing the two disproportionately degrades sound and music generation. We address this asymmetry at two levels: a \textbf{data reformulation} strategy that unifies all three tasks under a single description-style conditioning interface, and a novel architecture Asymmetric Mixture-of-Modality-Experts (A-MoME), which adds a dedicated residual expert for speech while sound and music share the backbone, incurring no inference overhead on non-speech inputs. Experimental results demonstrate that AudioCALM matches modality-specific state-of-the-art and outperforms prior unified baselines on speech, sound, and music generation benchmarks.

Text- and image-conditioned world generators can now produce visually rich 3D environments, but these worlds are usually silent or paired with a soundtrack generated only from text or rendered video. Such audio can describe what should be heard, but it does not explicitly represent where sounds live in the world or how they should change as a listener moves. We propose Audible World Models, a training-free framework that treats sound as part of a generated world state. Given a text prompt, our system builds a panoramic 3D proxy, decomposes it into semantic layers, identifies audible foreground objects and ambient background regions, synthesizes dry audio for each sound label, attaches sources to reconstructed geometry, and renders listener-dependent spatial audio through geometric acoustic propagation. This explicit coupling of semantics, geometry, and propagation produces audio that remains tied to persistent source locations and responds to viewpoint and motion. Across 80 generated scenes, our method substantially improves spatial consistency over text-, video-, and panorama-conditioned baselines while maintaining competitive semantic alignment. VLM-based and human evaluations further show that the resulting soundtracks are preferred for audio--visual consistency, spatial plausibility, and motion-dependent behavior.

Early-exit neural networks improve inference efficiency by enabling predictions at intermediate layers, but their effectiveness critically depends on the exit policy that decides whether to stop or continue computation. Existing policies, often based on confidence thresholds, enable early termination but offer limited insight into the reliability of such decisions. To address this, we introduce an explainability-aware framework that defines an attribution-grounded reliability signal for early-exit decisions. We propose Progressive Feature Attribution Maps (PFAM), which capture how feature importance evolves across exits, and the Interpretability-Based Early-Exit Score (IEES), a unified decision metric that integrates prediction confidence with attribution strength and cross-exit stability. To enable efficient deployment, we further introduce a lightweight proxy predictor that estimates IEES using inexpensive, attribution-free features, allowing attribution-informed decisions without incurring inference-time overhead. Experiments on convolutional architectures (ResNet, MobileNet, MSDNet) show that our approach improves accuracy–efficiency trade-offs, achieving up to 2.5\% accuracy gains while reducing inference time by up to 24\% compared to full-depth inference. Similar trends are observed on a vision transformer and a BERT-based language model, suggesting applicability beyond convolutional architectures.

Attribute-efficient PAC learning of sparse halfspaces has been a fundamental problem in machine learning theory. In recent years, machine learning algorithms are faced with prevalent data corruptions or even malicious attacks. It is of central interest to design computationally-efficient algorithms that are robust to malicious corruptions. In this paper, we consider that there exists a constant amount of malicious noise in the data and show that it is possible to learn an underlying $s$-sparse halfspace $w^* \in \mathbb{R}^d$ with $O(s^2\log^5 d)$ samples. Specifically, we follow a recent line of works and assume that the underlying distribution satisfies a certain concentration condition and a margin condition at the same time. As a complementary result, we provide an information-theoretic sample lower bound under such conditions. Strong evidence shows that our sample complexity is nearly optimal. To show the robustness of our algorithm, we provide a new gradient analysis that carefully handles the sparsity admitted constraints in hinge loss minimization program, which could be of independent interest.

Understanding how the brain integrates information across sensory modalities under naturalistic settings remains a central challenge in cognitive computational neuroscience. Despite recent progress in multimodal brain encoding driven by the advancement of deep learning, many existing approaches combine representations from pretrained modality-specific models using static aggregation or linear readouts, and therefore do not explicitly capture how information is selectively integrated across cortical regions. Here, we introduce Multimodal Transformer Brain Encoder (M-TBEn), a neural encoding framework that models multimodal integration as a routing problem. M-TBEn extends prior vision-based brain encoding approaches by introducing a cross-attention mechanism for multimodality, in which learnable parcel-level queries dynamically select and aggregate modality-specific representations from visual, auditory, and linguistic inputs. This design is not only parameter-efficient but also yields parcel-specific multimodal representations that support a flexible linear readout for predicting brain responses at various spatial resolutions. We evaluate M-TBEn on naturalistic video-viewing data, including both in-distribution and out-of-distribution stimulus conditions. The model is able to accurately predict parcel-level and voxel-level fMRI responses and generalizes across stimulus distributions. Furthermore, the learned attention patterns provide a structured descriptive basis for analyzing modality integration across cortical parcels, enabling a connection between parcel-specific routing profiles and established functional brain networks. Together, these results suggest that attention–based routing offers a principled computational framework for modeling multimodal integration in the human brain.

Masked diffusion language models can reduce inference steps by revealing multiple tokens per denoising iteration, but this parallelism is fragile: positions that are individually confident may be unsafe to commit together when their predictions are coupled. Existing training-free samplers such as Top-\(k\), Fast-dLLM, and EB-Sampler mainly control how many tokens to reveal, while often ranking candidates by token-wise scores that ignore interactions within the selected set. We propose ADAS, a training-free reranking rule for parallel masked diffusion decoding. ADAS leaves the base sampler's stopping rule unchanged and modifies only subset construction: it greedily discounts a candidate when it attends strongly to already selected positions whose predictions remain uncertain. Unlike graph-constrained methods that turn attention into hard compatibility constraints, ADAS keeps attention continuous and uses it as a soft marginal penalty. Across LLaDA-8B-Base and Dream-7B-Base on GSM8K, MATH500, HumanEval, and MBPP, plugging ADAS into Top-\(k\), Fast-dLLM, and EB-Sampler improves low-NFE performance at matched denoiser evaluations by \(9.11\) and \(10.46\) percentage points on average, respectively, with \(3.1\%\) per-forward runtime overhead. These results show that soft attention-discounted reranking is a simple and modular way to improve quality in highly parallel decoding for masked diffusion language models.

Neuro-symbolic systems typically couple a neural perception module with a discrete symbolic solver, where exact constraint inference is intractable at training time and a discrete solver must be invoked at inference time, precluding end-to-end optimization of the model. We introduce AS2 (Attention-Based Soft Answer Sets), a fully differentiable neuro-symbolic architecture that replaces the discrete solver with a continuous approximation of the Answer Set Programming (ASP)immediate-consequence operator $T_P$. AS2 maintains per-position probability distributions over a finite symbol domain throughout the forward pass and trains end-to-end by minimizing the fixed-point residual of a probabilistic lift of $T_P$, thereby differentiating through the constraint check without invoking an external solver during either training or inference. On spatial constraint-satisfaction tasks, AS2 replaces conventional positional embeddings entirely, and problem structure is encoded through constraint-group membership embeddings derived directly from the declarative ASP specification, making the model agnostic to arbitrary position indexing. On Visual Sudoku, AS2 achieves 99.89\% cell accuracy and 100\%constraint satisfaction, without invoking any external solver. On CLEVR-Hans, AS2 achieves 99.94% on CLEVR-Hans3 and 87.77% on CLEVR-Hans7. These results demonstrate that a soft differentiable fixpoint operator, combined with constraint-aware attention and declarative constraint specification, can match or exceed pipeline and solver-based neuro-symbolic systems while maintaining full end-to-end differentiability.

Auto-regressive models (ARMs) have established a dominant paradigm in language modeling. However, their strictly sequential sampling paradigm imposes fundamental constraints on both inference efficiency and modeling flexibility. To address these limitations, diffusion-based large language models (dLLMs) have been proposed, offering the potential for parallel sampling and flexible language modeling. Despite these advantages, current dLLMs sampling strategies rely primarily on token level information, which fails to account for global sequence structure and often yields suboptimal results. In this paper, we study the sampling order selection problem from the perspective of log-likelihood maximization. We show that this problem is NP-hard and propose an optimal sampling-rank-based approximation that makes the objective computationally tractable. We further prove that the tractable objective is optimized by sampling tokens in descending order of their attention-matrix column sums. This finding provides a principled justification for attention-guided sampling and offers a theoretically grounded alternative to greedy search. We instantiate this theoretical insight in a new training-free sampling algorithm, termed Attn-Sampler, and further propose dynamic attention thresholding for practical acceleration. Extensive experiments across multiple benchmarks validate the effectiveness of our proposed method, demonstrating that it achieves superior generation quality while enhancing the sampling parallelism.

Directed Acyclic Graphs (DAGs) are foundational to many areas of AI research, providing a formal language for causal reasoning and model interpretability. However, learning DAGs remains challenging, due to super-exponential computational cost and diminished accuracy in small sample regimes. To address these challenges, we introduce Attention-DAG (ADAG), a novel linear transformer architecture for unsupervised amortized DAG learning. Unlike traditional unsupervised DAG learning methods which recover the graph structure from data of each individual domain, ADAG leverages the knowledge from multiple domains. It provides a nonlinear mapping from observational data of each domain to both the corresponding graph structure and the underlying parameters of Structural Equation Models (SEMs). This enables efficient zero-shot inference of causal structures in new domains with unseen SEM parameters. Remarkably, we demonstrate that for DAG learning problem attention acts as a fixed-point iterative scheme, so the training on multiple domains effectively discovers an efficient solver for the constraint optimization problem. Extensive evaluations on synthetic and realistic benchmarks demonstrate that ADAG significantly outperforms existing baselines in accuracy and efficiency, particularly when data is scarce.

We study minimal attention-only transformers under all-token corruption and show they admit a two-stage empirical Bayes interpretation. A single attention step computes a kernel-weighted posterior mean with respect to the empirical distribution defined by the context. Depth refines this distribution through particle dynamics (Stage 1), while a long-range skip-connection carries the noisy input as a query for posterior inference (Stage 2), revealing distinct statistical roles for depth and attention residuals. The framework isolates a minimal setting in which the context itself induces a depth-dependent energy landscape governing in-context inference. We show that effective denoising can emerge without an explicit noise schedule: a fixed kernel bandwidth and finite integration horizon suffice, yielding a principled depth–noise relationship. We further establish a posterior-mean recovery guarantee for a class of well-behaved priors, where the empirical estimator converges to the Bayes-optimal predictor under asymptotic conditions. Connecting these dynamics to reverse-diffusion limits, our results provide a statistical interpretation of attention as in-context inference via sample-based posterior estimation, without explicit density modeling.

A recent work shows that Attention Transfer, which transfers only the attention patterns from a pre-trained teacher Vision Transformer (ViT) to a randomly initialized standard student ViT, is sufficient to recover the full benefit of the teacher's pre-trained weights. We revisit this finding on a comprehensive benchmark of 20 teachers from 11 well-known ViT families and reveal that Attention Transfer is not universally effective. While 7 families transfer successfully, 4 consistently fail, falling up to 5.1\% below the from-scratch no-transfer baseline. Further results demonstrate that this failure is family-consistent across model sizes, and persists under extended training durations, different transfer datasets, and out-of-distribution evaluations. Controlled analyses then consistently localize the problem to the attention-routing channel, indicating that the key issue is not whether the student can match the teacher's attention patterns, but whether the matched patterns remain functional for the student. Crucially, we identify architectural mismatch between the pre-trained teacher and the standard student as the primary mechanism. By adding only the teacher's native architectural components to the student in a randomly initialized state, we completely reverse the failure for all 4 families. Notably, these components alone do not improve from-scratch training, confirming that they specifically unlock the usability of the teacher's attention. We further systematically show that this failure is not explained by the inadequate choice of transfer loss or by differences in pre-training recipes. Our findings refine the prevailing understanding of attention in ViT representations: attention is sufficient only when the student architecture matches the teacher.

We propose OASIS, an outlier- and sink-aware technique built on inter-layer null signaling. As AttnResidual architectures introduce an additional depth-wise normalization channel, they improve inter-layer routing flexibility but also exacerbate attention sinks, activation outliers, and the resulting degradation in inference stability and quantization robustness. OASIS addresses this issue by introducing a $\mathop{\rm{Softmax}}_1$-based null space and coupling token-level null evidence to depth routing through an inter-layer null signal, thereby reducing sink-dominated routing and improving structural robustness. Theoretically, we show that the dual-normalization design of AttnResidual intensifies sink formation and quantization brittleness. Experimentally, we compare OASIS against five baselines on three real-world datasets and observe consistent improvements in both attention sink and post-quantization performance. Notably, OASIS achieves an average reduction of 9.26\% in maximum infinity norm and 2.60\% in average kurtosis across the evaluated settings, while lowering perplexity by 75.85\% under W8A8 and improving GSM8K Pass@1 by 12.42\% under W4A4.

Visual Geometry Grounded Transformer (VGGT) advances 3D reconstruction via scalable Transformer architecture, but the quadratic complexity of global attention prevents long context application. StreamVGGT enables streaming with causal attention, yet its KV cache grows linearly with frames, causing memory overflow and quality degradation. We present RetrieveVGGT, a training-free framework, which formulates context construction for VGGT as a retrieval problem. By retrieving a fixed number of relevant frames at each step, VGGT maintains a controllable memory budget, which is close to its training context length. Interestingly, we find that the similarity between current frame queries and cached history frame keys at the first global attention layer of VGGT is already a strong indicator of relevance, eliminating the need for additional learned scoring. To enhance information diversity similar to a recommender system, we propose Segment Sampling so that the retrieval spans distinct relevant segments rather than a single high-similarity region. We design a pose-aware spatial memory mechanism that organizes history frames according to their already estimated camera poses, enabling location-aware retrieval. Extensive experiments demonstrate that RetrieveVGGT achieves state-of-the-art performance, outperforming StreamVGGT, TTT3R, and InfiniteVGGT while maintaining constant memory usage regardless of sequence length.

Recent direct-sum hardness results show that computing multi-head attention by evaluating each head independently is essentially optimal under standard complexity assumptions, but leave open whether the key-value sharing in modern MQA and GQA architectures can circumvent this barrier. We prove that it cannot: even one-layer MQA with a single shared key-value head requires one full quadratic routing computation per query head in the worst case. To quantify how far practice departs from this worst case, we develop a calibration-stacked spectral framework that measures the effective number of independent routing-message directions in a layer, carefully separating signed, stochastic, and softmax-realizable compression; we also prove tight error-compounding bounds and NP-hardness of the natural route-merging compiler problem. Empirically, across fourteen open-weight GQA checkpoints from four model families (0.6B-70B parameters), eight shared routes capture over 90% of the routing-message energy in every model, and larger models use a progressively smaller fraction of their theoretical routing capacity, revealing a substantial and widening gap between worst-case hardness and the redundancy present in deployed transformers.

Recent studies have explored attention dynamics in Large Vision-Language Models (LVLMs). However, most existing approaches aggregate attention across heads, potentially obscuring head-specific visual information and weakening visual grounding, leading to hallucinations. In this work, we revisit hallucination from the perspective of how visual information is distributed and utilized across attention heads. We observe that a small subset of visual tokens accounts for most of the attention within each attention head, with these tokens—defined as head-wise visual cues—being complementary across heads, suggesting that effective grounding requires preserving head-specific support. We further find that the proportion of visual cues within visual attention declines during generation, leading to key visual information loss and hallucination. In addition, hallucinated tokens show weaker utilization of long-range textual context compared to correctly generated tokens. Building on these findings, we propose **V**isual Cu**e**s **R**einforcement via Head-Adaptive **D**e-redundancy and Distance-Aware **A**ttenuation (**VerDA**), a training-free method to mitigate hallucinations. Specifically, we remove redundant visual tokens via head-adaptive de-redundancy, suppress local textual bias through distance-aware attenuation, and reinforce visual cues by redistributing attention. Experiments across multiple LVLMs and benchmarks demonstrate that VerDA consistently outperforms existing methods in mitigating hallucinations, with negligible inference overhead. ***Code is available in the supplementary materials.***

Speculative decoding accelerates LLM inference by drafting future tokens with a small model, but drafter models degrade sharply under template perturbation and long-context inputs. We identify a previously-unreported phenomenon we call \textbf{attention drift}: as the drafter generates successive tokens within a speculation chain, attention progressively moves from the prompt onto its own recently-generated tokens. We observe this across both \emph{EAGLE3} drafters and \emph{MTP heads}, suggesting drift is a property of drafter designs. We trace this to the un-normalized residual path between chain steps: the drafter's hidden state magnitude grows monotonically with chain depth, which exhibits dynamics consistent with additional pre-norm transformer layers stacked on the target rather than as a standalone autoregressive predictor. In order to limit the growth, we propose two architectural changes: Post-norm on the drafter hidden states and per-hidden-state RMSNorm after capturing target hidden states. Our interventions improve acceptance length over the current leading model, pre-norm EAGLE3, by up to $2\times$ under template perturbation, $1.18\times$ on long-context tasks, and $1.10\times$ on seven standard benchmarks spanning multi-turn chat, math, and coding. Our changes also allow shorter train-time-test depths to generalize over longer drafting sequences.