NeurIPS 2026 Papers

We investigate whether neuron populations that shape the internal organization of neural networks evolve predictably with scale, extending scaling laws beyond macroscopic observables such as loss. To probe this question, we study *Rosetta Neurons*, a previously characterized class of neurons whose activation patterns are similar across independently trained models (Dravid et al., 2023). In separate analyses of language models up to 30B parameters and vision models up to 5B parameters, we observe that the population of Rosetta Neurons follows a sublinear power law in model size, growing in absolute number but occupying a shrinking fraction of the total neuron count. We further observe a *Neuron Polarization Effect*: Rosetta Neurons become more selective and increasingly monosemantic with scale, separating from a growing non-Rosetta population that remains less selective. An analytical model balancing feature utility against limited neuron capacity explains the sublinear power-law scaling and this polarization effect. Finally, we find that Rosetta Neurons become more domain-specialized with scale and illustrate their selectivity through a targeted data-filtering case study for continued pretraining. Our results point to a scaling law for shared neuron-level structure, linking model size to systematic changes in neuron universality, selectivity, and specialization.

Epilepsy affects approximately $1$% of the global population, where seizure detection and reliable forecasting are critical for patient safety and quality of life. While emerging foundation models increasingly recast EEG analysis as a language modeling task, most remain confined to single-modality inputs and narrow domain generalization. In this work, we present NeuroNTP, a modular multimodal foundation model built on an xLSTM backbone and pretrained on one of the largest retrospective epilepsy datasets ever collected, comprising $150,000$ hours of neural (EEG) and physiological (ECG, SpO$_2$) signals paired with rich clinical text. NeuroNTP employs a flexible, montage-geometry grounded tokenizer and optimizes a composite objective, synergizing causal next-token prediction with seizure-specific auxiliary tasks to enable unified sequence modeling across heterogeneous modalities. This approach allows the model to capture long-range temporal dependencies and cross-modal interactions that unimodal baselines ignore. In extensive experiments, we show that NeuroNTP outperforms supervised baselines by a substantial margin on held-out clinical cohorts. Furthermore, the model generalizes to public EEG benchmarks such as CHB-MIT and Siena Scalp EEG, achieving state-of-the-art performance under standard evaluation settings. We release code and model weights to facilitate further research.

Foundation models (FMs) promise to extract unified representations that generalize across downstream tasks. They have emerged across fields, including electroencephalography (EEG), but it is less clear how effective they are in this particular field. Published evaluations differ in datasets, in the EEG-specific preprocessing that might influence reported results, and in the reported metrics, frequently obscuring the clinical relevance in EEG. We introduce NeuroAtlas, the largest EEG benchmark to date: 42 datasets and $\sim$260k hours covering clinical EEG (epilepsy, sleep medicine, brain age estimation) and brain-computer interfaces, and include multiple datasets per task along with bespoke clinical evaluation metrics. Besides evaluating EEG-FMs with respect to supervised baselines, we present results from generic time-series FMs. We report three findings. First, EEG-specific FMs do not consistently outperform time-series FMs, which have neither EEG-focused architectures nor been pretrained on EEG. Second, standard machine learning metrics are insufficient to assess clinical utility: thus, we thoroughly evaluate more appropriate measures such as the quality of event-level decision-making, hypnogram-derived features, and the brain-age gap in the domains of epilepsy, sleep, and brain age, respectively. Third, model rankings and performance can vary substantially within domains. We conclude that pretrained models perform largely on par, with only narrow advantages for a few, and that current models do not yet deliver on the promise of an out-of-the-box unified EEG model. NeuroAtlas exposes this gap and provides the datasets and metrics for the next generation of unified EEG FMs.

Foundation Models for Electroencephalography (EEG) are trained with masked reconstruction, contrastive alignment, and language-model interfaces, yet their learned representations remain difficult to inspect: raw EEG lacks canonical token semantics, and models may preserve waveform detail while discarding physiologically meaningful structure. We introduce Neuro-KE (Neuro-Knowledge Engine), a knowledge interface that computes EEG descriptors without additional human annotation and renders them as paradigm-specific supervision for semantic grounding. Rather than using handcrafted features as standalone classifiers, Neuro-KE converts the same domain knowledge into numeric prediction targets, feature-state text anchors, and feature-grounded instruction answers. Across three interfaces, Neuro-KE improves aggregate grounding and transfer metrics, with task-dependent exceptions: it improves balanced accuracy (BAcc) by 2.60 points over reconstruction-only masked pretraining across 12 datasets and three backbones, improves six-dataset EEG--text validation BAcc by 2.23 points over label-only anchors, and improves EEG-MLLM feature-grounded generation to ROUGE-L F1 0.607 and BERTScore F1 0.929. These results suggest that established EEG descriptors can serve as reusable grounding signals for inspecting and supervising EEG foundation-model interfaces.

Modern Transformer design and compression both reduce to allocating capacity under a budget. The standard scalars for these decisions, #Params and #FLOPs, capture size and compute but not architectural structure—two architectures with identical parameter budgets but different depth-width, head, or FFN allocations receive identical scores yet behave differently. We propose Neural Spectral Capacity (NSC), a closed-form scalar grounded in the singular-value spectrum of each weight matrix. Under standard random initialization, the Marchenko–Pastur law renders NSC computable from the architectural specification alone, with no model instantiation, data, or gradients. Its layer-wise additive structure admits NSC-DP, an exact dynamic-programming solver returning the architecture globally maximizing NSC under resource constraints in seconds on a CPU—a guarantee that black-box search over existing training-free proxies cannot provide. Empirically, NSC outperforms #Params, #FLOPs, and representative training-free proxies in ranking across seven Transformer and CNN families (on FlexiBERT, τ=0.505 on pairs differing in #Params by <10%, where #Params collapses to 0.082); NSC-DP discovers a Transformer-XL architecture on WikiText-103 that beats the human-designed baseline in 2 seconds; and prunes LLaMA-7B to the best 5.7B model across eight commonsense reasoning tasks without any calibration data, ∼5900× faster than the strongest training-free proxy baseline.

Deep neural networks demonstrate remarkable generalization despite being highly over-parameterized, challenging traditional statistical learning theories. While several theoretical paradigms such as margin-based bounds, PAC-Bayesian analyses, and neural tangent kernels provide global explanations, they often lack mechanistic insight into how generalization emerges from training dynamics. We introduce Neural Expansion, a unified geometric framework that characterizes generalization through the expansion of directionally-selective ray coverage in the input space. Using Generalization Intervals (GIs), we quantify direction-specific regions around training samples where the model's predictions remain stable. Our extensive empirical studies across diverse architectures and datasets reveal that training leads to progressive expansion of ray coverage, particularly along directions with low Jacobian singular values. This mechanism helps explain a large fraction of correctly classified test inputs, adversarial examples, and even mislabeled or out-of-distribution samples through the geometry of ray coverage rather than instance-level memorization. By connecting input-space curvature to predictive behavior, our framework provides a mechanistic foundation that complements existing global theories of deep learning. The link to our code and data will be made available in the published version.

We present Neural Dual Bounds, a calibration-free framework that produces deterministic upper bounds for MAP inference in probabilistic graphical models using a single neural-network forward pass. The framework trains a network to output the edge messages of a mini-bucket style upper bound on a given join-graph decomposition in a self-supervised manner by directly minimizing the a smooth dual objective. The output of this network can be used at test time to warm-start a standard JGLP optimizer. Across diverse benchmarks, this warm start yields dual gaps that are orders of magnitude tighter than those from zero initialization, with the largest gains in the early-iteration regime. We also apply our framework for the constrained MAP (CMPE) task, and we show that neural warm starts with test-time adaptation consistently achieve the tightest dual gaps across all iteration budgets against an adaptive baseline.

Sequential estimation under partial observability requires tracking belief distributions that may be high-dimensional, multimodal, and non-Gaussian. Classical Bayesian filters generalize zero-shot to any system whose dynamics can be evaluated, but their representations scale poorly: parametric filters struggle to capture multimodality, and particle filters require exponentially many particles in the state dimension. Generative Distribution Embeddings (GDEs) learn compact representations of complex distributions but provide no mechanism for sequential updates. We present Neural Bayesian Filtering (NBF), a method that combines classical Bayesian filtering with learned distribution embeddings. NBF represents beliefs as embeddings and updates them by sampling particles from a learned conditional generator, propagating them through the system dynamics, and finally re-embedding the resulting particles. For in-family distributions that the embeddings are trained to represent, NBF inherits the zero-shot adaptability of classical filters, but with the benefit that regenerating particles from the embedding at each step also mitigates particle impoverishment. We validate NBF on a pursuit--evasion domain, demonstrating accurate tracking of complex multimodal posteriors, graceful scaling with state dimension, and zero-shot generalization to dynamics held out from training.

Vision-Language-Action (VLA) models promise generalist robotic agents, yet consolidating skills learned across heterogeneous tasks, environments, and embodiments remains difficult: training data is fragmented across platforms, joint multi-task training suffers from interference, and model merging tends to introduce parameter conflicts or require architectural changes. We instead advocate a paradigm in which skills are first learned independently and then consolidated post hoc. To this end, we propose **NestedVLA**, which replaces parameter aggregation with **parameter generation**: a nested hypernetwork, conditioned on the target task, synthesizes task-adaptive parameters that compose with a frozen base VLA. This formulation enables flexible task-conditioned skill reuse while mitigating interference. Across three RL benchmarks (MetaWorld, ManiSkill, CALVIN), NestedVLA outperforms the strongest model-merging baseline by **41 pp on average**, matches per-task fine-tuning **within 1 pp on ManiSkill**, and substantially closes the gap on CALVIN, pointing toward a scalable route to skill consolidation.

We give computable necessary conditions for replacing a fixed attention head by a value-linear recurrent surrogate. The semiseparable spectrum of the causal attention matrix lower-bounds the operator-norm error of any finite-state value-linear recurrence, and retrieval-style blocks yield sharp K and Kd_v state-scaling laws. Empirically, these lower bounds are tight on retrieval-shaped targets but can be loose on smooth or clustered targets, where rank feasibility does not imply trainability or downstream quality. Large pretrained-head surveys suggest that this rank obstruction is often small, including in modern 1B, 7B, and 8B checkpoints. The resulting diagnostic is best used as a rejection test for under-sized recurrent state, not as a certificate that a recurrent replacement will train or improve downstream performance.

Orthogonal trace-sum maximization (OTSM) problems arise in a wide range of data processing applications, including canonical correlation analysis and cryogenic electron microscopy. Despite their practical importance, the existing theoretical understanding of these problems remains incomplete. In this paper, we show that the generalized power method (GPM) converges linearly to the global optimal solution with high probability under an additive Gaussian noise model, provided that the noise level satisfies the nearly optimal bound $(O(\sqrt{n/\log n}))$. In addition, we prove that the semidefinite programming (SDP) relaxation of OTSM is tight and admits a unique optimal solution under the same noise regime, improving upon the best known theoretical guarantee of $(O(n^{1/4}))$. Extensive numerical experiments further demonstrate that our theoretical predictions closely match empirical observations.

We consider the problem of estimating the Attention mechanism in small space, and prove the existence of coresets for it of nearly optimal size. Specifically, we show that for any set of unit-norm keys and values $(K,V)$ in $R^d$, there exists a subset $(K',V')$ of size at most $O({\sqrt{d} e^{\zeta+o(\zeta)}/\epsilon})$ such that $$ \left\| Attn(q,K,V)- Attn(q,K',V') \right\| \le \epsilon $$ simultaneously for all queries whose norm is bounded by $\zeta$. This outperforms the best known results for this problem. We also offer an improved lower bound showing that $\epsilon$-coresets must have size $\Omega({\sqrt{d} e^{\zeta}/\epsilon})$.

Textual cues provide important navigation-relevant semantic information like room names, store signs, and floor labels. However, conventional optical character recognition (OCR) systems and datasets are designed to detect all visible text, including advertisements and price tags, regardless of their relevance to navigation tasks. This characteristic of OCR can produce noisy semantic cues, making it difficult to identify which text is useful for robot navigation. A straightforward solution is to curate existing OCR datasets by retaining only navigation-relevant text and removing irrelevant annotations, but this process requires substantial manual effort. To address this problem, we introduce NavOCR, a dataset generator for navigation-relevant text detection that does not require manual curation. By leveraging conventional OCR, open map resources, and filtering with a vision-language embedding model, NavOCR identifies which text instances serve as place identifiers for robot navigation and constructs datasets from images available on the web. NavOCR aims to complete the entire process within a single day, from data collection to model training, enabling rapid adaptation to diverse environments and languages. Compact models trained on datasets generated by NavOCR achieve higher detection accuracy and substantially better computational efficiency than vision language models and open vocabulary detectors. We further show that detecting navigation-relevant text improves the quality of semantic cues used for localization and mapping in robot navigation applications.

Vision-Language-Action (VLA) models are formulated to ground instructions in visual context and generate action sequences for robotic manipulation. Despite recent progress, VLA models still face challenges in learning related and reusable primitives, reducing reliance on large-scale data and complex architectures, and enabling exploration beyond demonstrations. To address these challenges, we propose a novel \textbf{N}euro-\textbf{S}ymbolic \textbf{V}ision-\textbf{L}anguage-\textbf{A}ction (NS-VLA) framework via online reinforcement learning (RL). It introduces a symbolic encoder to embedding vision and language features and extract structured primitives, utilizes a symbolic solver for data-efficient action sequencing, and leverages online RL to optimize generation via expansive exploration. Experiments on robotic manipulation benchmarks demonstrate that NS-VLA outperforms previous methods in both one-shot training and data-perturbed settings, while simultaneously exhibiting superior zero-shot generalizability, high data efficiency, and an expanded exploration space. Our code is available at https://anonymous.4open.science/r/NS-VLA/.

Current medical LLM benchmarks typically decompose clinical competence into isolated factual questions, single-image interpretation, or brief case scenarios. This fragmented paradigm primarily measures local correctness but provides limited insight into whether models can progressively integrate subspecialty knowledge, multimodal evidence, and evolving patient states into safe, guideline-constrained longitudinal reasoning. To address this gap, we introduce NPCBench, a multimodal clinical apprenticeship benchmark for nasopharyngeal carcinoma (NPC). NPCBench operationalizes specialist training as a staged evaluation of (M)LLMs across guideline recall, atomic rule application, multimodal evidence grounding, longitudinal care planning, state revision, and expert-style consultation. It comprises 4,077 staged evaluation problems and 25 multimodal full-care path episodes, covering pathology diagnosis, MRI staging, and 146 downstream clinical decisions. The benchmark explicitly links stepwise tasks to complete patient-level care trajectories, enabling systematic evaluation of all clinical pathways. Beyond isolated precision, NPCBench evaluates whether the models ground decisions in multimodal evidence, maintain temporally consistent patient states, and provide safe expert-level consultation across complete care trajectories. Across 30 evaluated models, we observe a persistent composition gap: strong performance in local case-level tasks does not translate into episode-level clinical competence required for end-to-end patient management. Overall, NPCBench advances medical AI evaluation from static question answering toward apprenticeship-style workflow evaluation, providing a rigorous testbed for evidence-grounded and temporally coherent reasoning in complex oncology care.

Low-Rank Adaptation (LoRA) fusion enables the composition of subject and style representations for controllable generation without retraining. However, existing approaches primarily operate through weight-level merging, without explicitly modeling how independently trained LoRAs interact in the shared parameter space. We adopt a geometric perspective on LoRA fusion, interpreting content and style LoRAs as occupying overlapping, non-orthogonal low-rank subspaces, where such overlap can lead to conflicting parameter updates that affect generation quality. This observation motivates us to reformulate LoRA fusion not merely as parameter combination, but as a problem of controlling how updates from overlapping subspaces are combined. Based on this insight, we propose Null Space Projection LoRA (NP-LoRA), a training-free framework that employs projection as a fusion operator to explicitly modulate cross-LoRA interactions. Specifically, NP-LoRA uses principal directions of the style LoRA to define a projection subspace and projects the content LoRA onto the complementary subspace (i.e., the null space of the style LoRA), suppressing interference along dominant style directions while preserving complementary information. To avoid the overly aggressive suppression of hard projection, we further formulate soft projection as a regularized optimization problem that balances content preservation against style-subspace suppression. This objective admits a closed-form solution, yielding a projection operator controlled by a single parameter that continuously interpolates between linear merging and hard projection. Extensive experiments across multiple pretrained LoRA pairs show that NP-LoRA achieves more balanced content-style composition compared to strong baselines, without requiring retraining.

Muon and its variants have shown strong empirical performance in a variety of deep learning tasks. Existing convergence analyses of Muon rely on smoothness assumptions, though arguably the most successful function class for developing deep learning methods (such as AdaGrad, Shampoo, Schedule-Free and more) has been the class of convex and Lipschitz functions. In this paper we question whether the classical convex Lipschitz model is a useful one for understanding Muon. Our answer is no. We show that Muon does not converge on the class of convex and Lipschitz functions, regardless of the choice of learning rate schedule. We also show that error feedback restores convergence of Muon and all the non-Euclidean subgradient methods with momentum. However, this theoretical fix using error feedback degrades the performance of Muon in two representative settings for image classification (CIFAR-10) and language modeling (nanoGPT on FineWeb-Edu 10B). Our conclusion is that convex Lipschitz theory, despite having a prominent role in the design of practical methods for deep learning, is not the most suited one for Muon. This suggests that Muon's success must come from structure absent from this model, most plausibly related to smoothness.

State-of-the-art pathology foundation models, trained on millions of histology tiles, can fail to preserve tissue similarity when comparisons cross slide or institution boundaries. We show that general-purpose multimodal LLMs, without being trained as pathology foundation models, consistently outperform these specialized models in cross-domain histological similarity judgments. Using a relative similarity framework that we release as the MOSAIC (Model Similarity Assessment across Institutions and Cohorts) benchmark, we evaluate 17 models across 6 datasets and find that pathology encoders often rank same-institution, different-disease tiles as more similar than same-disease, different-institution tiles, a clinically dangerous failure mode invisible to standard within-domain evaluations. LLMs appear less susceptible to this failure, likely because they perform semantic visual comparison of morphology and tissue architecture rather than relying on shortcut features tied to acquisition context. Scaling training data does not resolve the problem for pathology encoders, implicating the learning objective rather than data coverage. Our results expose a fundamental robustness gap in current pathology foundation models and establish multimodal LLMs as a viable alternative for cross-institutional retrieval, dataset harmonization, and multi-site quality control. Code and data will be released upon acceptance.

This position paper argues that multimodal foundation models should use brain recordings as privileged internal-state supervision during training. External behavior alone leaves human-like cognition, robustness, and reasoning weakly constrained. Neural recordings can serve as a training-time modality that shapes representations, latent-state dynamics, private computation policies such as reasoning tokens, and robustness before distillation into agents that operate from ordinary sensory, language, and action streams. We develop this position through Foundational Human Modeling (FHM), a program centered on scaling generative brain models, aligning heterogeneous neural modalities through shared latent geometry, inserting brain tokens into unified multimodal token streams, and testing whether the privileged channel improves distilled agents after removal. The argument is grounded in privileged-information theory and prior work showing that neural or human-derived targets can improve robustness, semantic transfer, data efficiency, and behavioral similarity after auxiliary signals are removed.