NeurIPS 2026 论文列表

Ubiquitous IoT devices, such as companion toys, wearables, and micro-robots, are increasingly embracing language models, yet current cloud-based deployments suffer from latency, privacy, and weak-network reliability issues. These resource-constrained devices typically cap deployable model size below 100M parameters, while existing sub-100M models fall short in basic language and commonsense capabilities. In this extreme regime, we argue that the memory bottleneck supersedes the traditional compute bottleneck as the binding constraint, calling for memory-first design that maximizes model capability under a fixed parameter budget. Notably, the stringent budget binds both what the model stores and what it can afford to learn, demanding architecture-data co-design. In this paper, we propose $\mu$LM, a memory-first sub-100M language model framework instantiating this co-design. For architecture, we present DMLA, which preserves latent attention's compact cache while relieving its feature-squeezing bottleneck under aggressive compression via explicit content-position latent decoupling. For data, we present $\mu$Mix, a capacity-aware multi-stage curriculum tailored to sub-100M models, prioritizing high-quality natural language and commonsense data aligned with embedded scenarios over brute-force token scaling. $\mu$LM-70M surpasses all sub-300M baselines and rivals several sub-1B models on standard benchmarks; even at 9M, it remains competitive. Fine-tuned into a tool-calling agent, $\mu$LM outperforms same-scale baselines and achieves a 97.2\% end-to-end executable rate, presenting a promising direction toward practical sub-100M language modeling.

Explanations are central to causal reasoning, and cognitive science has long established that the human drive to explain is itself a mechanism for learning about causality. Despite this, learning from those abductive signals is largely ignored in artificial intelligence. While explainable AI (XAI) increasingly draws on causal models to generate explanations, the converse direction about *what explanations can do for causality* remains largely unexplored. To fill this gap, we propose xWhyL, a formal framework connecting causality and XAI by learning causal models from explanations. We develop a mathematical theory that translates explanations into a learning signal complementary to observational data, and demonstrate how it enables overcoming the limits of observational causal discovery. As explanations can be derived from incorrect beliefs and clash with data, a tension we call the *Causal Tug-of-War*, we prove conditions under which our framework rejects misspecified explanations rather than absorbing them. Our practical instantiation, *Causal Interactive Learning* (CIL), shows how expert explanations can efficiently support causal discovery and distinguish correct from incorrect explanations.

Graphs play a crucial role in many applications, ranging from drug discovery to social network modeling and astronomy. Generative models for graphs have substantially advanced in recent years, but we identify relatively simple scenarios where state-of-the-art models struggle to scale and exhibit prohibitively-slow generation times. We propose a generative architecture that matches or exceeds the state-of-the-art, but can scale to regimes where their training fails, and is orders of magnitude faster at generation. Our approach is inspired by the Aldous-Hoover theorem, a classic representation theorem that characterizes any probability distribution of graphs that is invariant to permutations of node indices (i.e., exchangeable). This theorem establishes that three ingredients are needed: a graph-level variable, a set of node-specific variables, and a so-called graphon function that maps graph- and node-level variables to the probability of an edge between any node pair. Given a training set of graphs, our exchangeable Variational Graph AutoEncoder (xVGAE) learns an approximation to their underlying graphons, as well as graph- and node-level latent variables. We show that our xVGAE encodes interpretable representations of graph- and node-level properties, and can (quickly, in one shot) generate new graphs that closely match key training set statistics.

Mixture-of-Experts (MoE) models have become prevailing, yet their distributed deployment suffers from severe dynamic load imbalance due to the conflict between static expert placement and inherent routing dynamism. While host DRAM offers capacity to avoid static device storage for load balancing, we identify existing offloading works fundamentally fail in distributed inference due to two overlooked challenges: (1) coupled performance impacts of runtime Host-to-Device transfers due to load dynamism and shared PCIe topology; (2) system-level memory overhead incurred by DRAM usage at scale. In this paper, we propose vExpert that shifts DRAM offloading paradigm from partial residency to full redundancy by virtualized expert storage. We pioneer the first systematic performance model that jointly captures H2D latency, PCIe contention, and load imbalance, building a lightweight allocator to dynamically map physical experts to virtual slots. A novel disaggregated manager is designed to reduce memory overhead in multi-process environments. Evaluated on DeepSeek-V3.1 within an 8-node H100 cluster, vExpert improves load balance ratio by up to 55\% and achieves 1.2$\times$ end-to-end speedup. Code is available at https://anonymous.4open.science/r/vExpert-B743/.

High-dimensional point cloud data arise across many scientific domains, notably single-cell biology. The "shapes" or topologies of these datasets are informative of types of information that can be extracted from the datasets. For example clustered data admits the extraction of cell types or cell states in a static analysis of the datasets. Continuous trajectory structures admit continuous transition or trajectory analysis, while other shapes such as archetypal shapes admit continuum extraction with a range of cells spanning behaviors. While analysis pipelines exist, they often presuppose shape in data. For example, the standard Seurat pipeline combines UMAP visualization with Louvain clustering. This assumes clustered data. Tools like Monocle and Spade assume a tree-like shape, and flow-models like MIOFlow and Conditional Flow Matching are suitable for trajectories. Deciding which pipeline to apply to which part of the data is often the realm of bioinformaticians who visualize and qualitatively analyze the data before selecting one. However, with the advent of agentic AI scientists, it becomes important to automate data shape detection, particularly into categories that are relevant for downstream analysis pipelines. Towards this end we introduce \textsc{scShapeBench} a benchmark dataset comprising both synthetic and single-cell expert-annotated datasets that are meant for the task of shape detection. Synthetic datasets are sampled from a ground truth “skeleton graph” with variance. Real single-cell datasets are curated from a variety of sources and are annotated by experts, classifying four categories; clusters, single trajectory, multi-branches and archetypes. In addition, we provide a baseline method, scReebTower, to bridge the gap between data visualization and pipeline selection. scReebTower relies on the diffusion geometry to extract Reeb graphs. We provide new topology-aware metrics with which we evaluate scReebTower and existing methods PAGA and Mapper on synthetic data. On single-cell data we curate expert annotations of shapes, and showcase evaluations of methods. Our comparisons indicate scReebTower outperforming other baselines. Overall, our contributions span benchmarks, evaluation metrics, and a novel baseline method for automated shape detection in high-dimensional single-cell data.

Spatial transcriptomics has created a compelling opportunity to test whether tissue morphology can predict molecular state, but existing benchmarks are constrained by limited scale, spot-level supervision, and incomplete biological context. We present sMMC-22M, a cell-aligned multimodal resource comprising over 20 mil- lion cells across 25 organ categories, 66 studies, and multiple spatial-transcriptomic assays. We organize the benchmark around three data-centric axes that determine whether histology-to-molecular modeling can move beyond local interpolation. First, sMMC-22M provides scale: broad organ and study coverage enables con- trolled encoder benchmarking and reveals a scaling trend in which larger pathology foundation models improve morphology–gene correspondence under matched evaluation. Second, sMMC-22M provides resolution: by decomposing assays into aligned cell-level records, our framework converts spot-level histology–omics pipelines into single-cell predictors and evaluates them under strict in-domain and cross-patient splits. Third, sMMC-22M provides rich context: each cell is paired with spatial, molecular, and sample-level metadata, allowing analyses such as age-band shift in ovarian samples, where age-mismatched transfer sharply reduces prediction quality despite misleading global-distance summaries. Together, sMMC- 22M and STBoost establish a practical framework for single-cell histology-to-gene prediction while showing that robust generalization still depends on scale, cellular resolution, and explicit biological context.

Recent progress in neural theorem proving has been driven largely by mathematics-oriented and, more recently, by verification-condition and repository-scale benchmarks. These settings are valuable, but they do not directly test whether a model can synthesize machine-checkable proofs about concrete low-level implementations under an industrial proof-engineering stack. We address this gap with *s2n-bignum-bench*, a benchmark derived from AWS *s2n-bignum*, a formally verified library of hand-tuned big-integer assembly routines for ARM and x86. The current corpus packages $\mathbf{2{,}301}$ HOL Light proof obligations as standalone `setup.ml`/`query.txt` tasks spanning big-integer arithmetic, elliptic-curve routines, ML-KEM, SHA-3/Keccak, and shared ISA infrastructure. Each task reproduces the relevant proving environment, exposes the theorem statement to be proved, and expects a tactic expression that is accepted by HOL Light within a fixed timeout. The benchmark spans five categories so that one can separate generic HOL Light fluency from ISA-specific reasoning. We release the extraction pipeline, retrieval utilities, an offline evaluator with syntax/type pre-checking and integrity checks based on axiom-difference and forbidden-tactic detection, paired verbatim and obfuscated query variants, and a checkpointed assessment workflow to support iterative feedback-driven $\operatorname{pass}@K$ studies and future step-level proof construction workflows. Initial zero-shot baselines using GPT-5.3-based and other frontier models solve at most $\mathbf{6.35}$% of the problem set (when provided only with the goal term), highlighting the difficulty of HOL Light tactic synthesis for low-level verification obligations under trusted ISA semantics. The code to set up and use the benchmark is available at [s2n-bignum-bench](https://github.com/kings-crown/s2n-bignum-bench).

Training data attribution (TDA) methods estimate the influence of individual training samples on a model's predictions, offering a principled lens for interpreting neural networks. Despite rapid methodological progress, TDA evaluation remains scattered across papers and rarely reproducible, making fair comparison between methods challenging. We introduce **quanda**, a Python toolkit that standardizes TDA evaluation. **quanda** features a comprehensive collection of evaluation metrics and ready-to-use benchmarks spanning image classification, text classification, and language modeling, alongside a uniform interface for integrating diverse TDA implementations. We showcase **quanda** through a comparative study of existing TDA methods and find that no approach excels uniformly across metrics, with clear trade-offs between ground-truth faithfulness and downstream task performance. By consolidating the tooling around TDA, **quanda** equips the community to evaluate attribution methods rigorously and reproducibly. The toolkit is thoroughly tested, documented, and available as an open-source library on PyPI and at https://anonymous.4open.science/r/quanda.

Cognitive diagnosis (CD) infers students' mastery of knowledge components (KCs) from their responses, supporting personalized feedback and exercise recommendation. Cognitive diagnosis models (CDMs) are often ranked by response-prediction AUC under inconsistent experimental protocols, while their KC-level mastery profiles are not consistently audited under a shared protocol. We introduce \texttt{pyCD}, a unified benchmark of eleven CDMs on six public datasets that fixes preprocessing, splits, hyperparameter search, and stopping rules; the benchmark reports KC-level diagnostic metrics for learned mastery alongside standard prediction metrics. Two findings show that gains in AUC do not translate into gains in diagnostic quality. First, the AUC leaderboard is sharply compressed and sensitive to protocol choices: modern CDMs often cluster within $0.005$--$0.020$ AUC, with the per-dataset best non-IRT CDM improving over IRT by only $0.017$ AUC on average; removing validation-AUC early stopping can shift AUC by up to $0.13$ on individual cells. Second, the AUC gaps that remain do not reliably transfer into diagnosis: the top-1 AUC model is typically outside the top-3 by KC-level diagnostic agreement (DOA). The same mismatch extends to exercise difficulty: against Junyi's human pairwise difficulty ratings, learned CDM difficulty parameters rank worse than both an empirical accuracy-based baseline and zero-shot LLMs given only exercise titles. CDM evaluations should therefore report diagnostic outputs directly rather than infer them from predictive gains; \texttt{pyCD} provides a unified protocol for doing so. Code, splits, and configurations are available at \url{https://anonymous.4open.science/r/pyCD-NeurIPS2026-60B8}.

As machine learning systems are increasingly deployed in real-world applications, the question of how to represent and quantify uncertainty has moved from a methodological side issue to a central concern. In practice, however, making a model uncertainty-aware is still surprisingly difficult: relevant tools tend to be scattered across libraries, each tied to a particular framework and a particular approach to modeling uncertainty, and the choice of representation, quantification, and evaluation is typically left to the user without much guidance. In this paper, we present probly, a Python package that addresses theses issues in a single, modular framework. probly offers (i) lightweight transformations that turn existing models into uncertainty-aware ones, currently supporting PyTorch, scikit-learn, and Flax, (ii) several representations, including second-order distributions, credal sets and conformal prediction, (iii) the corresponding quantification measures, and (iv) a number of evaluation protocols, which can be combined more or less arbitrarily. Using probly, we conduct a benchmark study on three standard tasks --- selective prediction, out-of-distribution detection, and active learning. In addition, and unlike existing benchmarks, we evaluate suitable methods on first-order data, i.e., datasets for which the target itself is a distribution over outcomes. We further illustrate the flexibility of the package on a number of less standard case studies, including large language models, graph neural networks, and data streams. The package is publicly available at \url{https://anonymous.4open.science/r/probly}.

Reasoning is essential for autonomous driving (AD) in long-tail scenarios, where vehicles must apply commonsense knowledge, understand spatial relations, infer agent interactions, and make safe decisions. However, existing AD datasets and benchmarks mainly target perception, prediction, or planning, and provide limited supervision for reasoning over realistic long-tail driving scenes. We introduce nuReasoning, a large-scale real-world dataset and benchmark for reasoning-centric AD. Following the lineage of nuScenes and nuPlan, nuReasoning advances real-world AD datasets and benchmarks toward structured reasoning in long-tail driving scenarios. It contains 20K 20-second clips collected across multiple cities, with synchronized multi-camera images, LiDAR, HD maps, object annotations, and complementary forms of human-verified reasoning annotations: Spatial Reasoning, Decision Reasoning, and Counterfactual Reasoning. Unlike prior datasets that focus primarily on visual question answering, nuReasoning supports both reasoning evaluation and planning evaluation, enabling a direct study of how reasoning supervision affects driving performance. Experiments show that fine-tuning VLMs on nuReasoning substantially improves driving-specific question answering, while incorporating reasoning supervision into VLA training improves planning performance even when textual reasoning outputs are disabled at inference time. These results establish nuReasoning as a foundation for evaluating and improving robust, interpretable, reasoning-driven AD systems in realistic long-tail settings.

Distributed training is essential for scaling LLM training across thousands of GPUs. However, as distributed training requires complex implementations, they are prone to silent bugs, which do not produce explicit error signals but lead to in correct training outcomes. Common debugging practices based on monitoring training loss or gradient norm curves are slow or unable to detect bugs and also do not help localize bugs. We design and implement nnTrace, the first systematic differential testing system for detecting and localizing silent bugs in distributed training. nnTrace aligns intermediate tensors from distributed training with those from a trusted reference implementation. To properly compare the floating-point values in the corresponding tensors, we propose a novel mathematical analysis that provides a guideline for setting tolerances, enabling nnTrace to distinguish bug-induced errors from numerical errors. Experimental results demonstrate that nnTrace effectively detects 11 existing bugs and 3 new bugs in the widely used Megatron-LM framework. nnTrace is effective in various training recipes, including low-precision recipes involving BF16 and FP8. Notably, Megatron-LM has already adopted the method proposed by nnTrace in its development workflow. Our code is available at https://anonymous.4open.science/r/NECK-3C61/.

Millimeter-wave (mmWave) radar provides core sensing capabilities to traditional vision, particularly under occlusion, low-light, and privacy-constrained conditions. Recent efforts have explored integrating radar signals with large language models (LLMs) for high-level semantic reasoning over the received radar data. However, existing approaches typically project radar inputs into borrowed vision or LiDAR embedding spaces, which are not tailored to radar-specific physical cues such as Doppler and reflection intensity. This design introduces a representational bottleneck, hindering the extraction of informative radar semantics. To overcome this limitation, we propose \textbf{mmLIP}, a novel radar-language interactive pretraining framework that learns radar-specific alignment without relying on external embedding spaces. Our approach directly aligns radar representations with the text embedding space via a point-level contrastive objective, enabling fine-grained correspondence between radar points and textual tokens. In addition, we introduce a confidence-aware contrastive learning mechanism that adaptively reweights radar tokens based on their semantic relevance, promoting informative signals while suppressing clutter. Supported by our newly curated radar-text pairs, mmLIP captures structured radar semantics. Extensive experiments demonstrate that mmLIP successfully integrates with diverse LLMs or vision-language models, achieving state-of-the-art performance on zero-shot bidirectional retrieval as well as text generation tasks, including captioning and question answering.

Higher-order learning exploits compositional features whose predictive value arises from nonlinear interactions beyond pairwise relations among variables, scales, or modalities. To address the complexity of modeling higher-order interactions in modern large-scale deep learning models, a kernelized Volterra Neural Network (kVNN) is proposed in this paper. Specifically, the proposed learnable multi-kernel representation models different interaction orders using distinct polynomial-kernel components with compact learnable centers, thereby yielding an order-adaptive parameterization. The resulting kVNN layers are composed of parallel order-specific branches and can readily replace standard convolutional kernels in existing learning architectures. Theoretical results are supported by experiments on video action recognition, image denoising, and image classification. The results show marked performance-efficiency trade-offs: kVNN consistently reduces model (parameters) and computational (GFLOPs) complexity while achieving competitive and often improved performance, even when trained from scratch without large-scale pretraining. In summary, structured kernelized higher-order layers offer a practical path to balancing expressivity and computational cost in modern deep networks.

Tiny object detection (TOD) remains challenging due to the extremely limited spatial details of small objects. Existing methods heavily rely on multi-scale feature pyramids with discrete feature maps, which suffer from insufficient spatial resolution, quantization errors, and poor localization accuracy, while imposing heavy computational and memory overhead. We present the implicit DEtection TRansformer (iDETR), an efficient detector that operates exclusively on low-resolution single-scale features. It incorporates two novel components: (1) implicit Attention (iAttn), which leverages implicit neural representations to model continuous features and enables precise sub-pixel querying beyond discrete grid limitations; and (2) Centroid-Guided Query Initialization (CGQI) for robust query initialization under single-scale constraints. We further propose head-conditional sampling in iAttn, which reduces querying computational cost by 4× and memory footprint by 3× without sacrificing performance. Extensive experiments demonstrate the effectiveness of iDETR. Especially, on AI-TODv2, iDETR outperforms state-of-the-art methods by 0.2% AP overall, with particularly strong gains of 1.1% AP on very tiny objects, while reducing computational cost by 37%. The code is available at https://anonymous.4open.science/r/iDETR-FC25/.

Deep Equilibrium Models (DEQs) are an established framework for image restoration that learn a problem-adapted regularization by solving a fixed-point (i.e. equilibrium) problem. While flexible and expressive, DEQs are often hindered by high computational cost and training instability. We propose an inertial DEQ (i-DEQ) that learn an explicit nonconvex regularization within the DEQ formulation. By using momentum within the fixed-point iterations, i-DEQ has convergence guarantees and accelerated rates. Moreover, we observe that i-DEQ is significantly more stable during the training and robust to rough initialization than DEQs. Numerical experiments on various linear and nonlinear inverse problems demonstrate that i-DEQ achieves reconstruction quality comparable to state-of-the-art methods, while reducing DEQ's inference time by a factor of two.

Haphazard inputs arise in online learning when feature availability varies across instances and the complete feature space is unknown in advance. However, the landscape of haphazard inputs research has been fragmented due to inconsistent data preprocessing, non-uniform evaluation metrics, limited coverage of streaming regimes and normalization, and model specific experimental code, making direct comparison across existing methods unreliable. We present haphazard, an open source Python library for reproducible benchmarking of haphazard input methods. The library provides an order preserving data pipeline with streaming normalization, unified evaluation, four streaming regimes, and shared interfaces for datasets, models, normalizers, and metrics. The current release supports 22 datasets, 11 model implementations, 5 normalization strategies, and 7 evaluation metrics, enabling systematic benchmarking across datasets and stream settings. By consolidating the experimental pipeline and exposing extensible interfaces for datasets, models, normalizers, metrics, and streaming regimes, haphazard provides a common reference point that can grow with future research on haphazard inputs. The anonymized source code is available at https://anonymous.4open.science/r/haphazard, and the PyPI package and docker container will be released publicly upon acceptance.

In this paper, we present gfnx, a fast and scalable package for training and evaluating Generative Flow Networks (GFlowNets) written in JAX. gfnx provides an extensive set of environments and metrics for benchmarking, accompanied with single-file implementations of core objectives for training GFlowNets. We include synthetic hypergrids, multiple sequence generation environments with various editing regimes and particular reward designs for molecular generation, phylogenetic tree construction, Bayesian structure learning, and sampling from the Ising model energy. Across different tasks, gfnx achieves significant wall-clock speedups compared to Pytorch-based benchmarks (such as torchgfn library) and author implementations. For example, gfnx achieves up to 55 times speedup on CPU-based sequence generation environments, and up to 80 times speedup with the GPU-based Bayesian network structure learning setup. Our package provides a diverse set of benchmarks and aims to standardize empirical evaluation and accelerate research and applications of GFlowNets.

Spreadsheets are among the most widely used tools for computations, and formulas are their core building blocks. Besides agentic spreadsheet tools, proactive formula suggestions enable significant productivity gains and are now available in popular spreadsheet software (Microsoft Excel and Google Sheets). Such suggestions pose interesting challenges: the right context must be collected and represented for a machine learning model, which must then be able to write a correct formula, in a way that is fast and cheap enough to trigger on every $\texttt{=}$ (or subsequent keystrokes for auto-completion) without taking users out of their editing flow. Surprisingly, there is no public benchmark for formula suggestions on real spreadsheets. In this paper, we therefore (1) introduce fxbench as a manually curated benchmark of 503 formula suggestion tasks based on Sheetpedia, (2) break down the formula suggestion problem into four capabilities---gather context, understand context, understand intent, write formula---that a formula suggestion system requires, and (3) evaluate existing and new approaches to understand bottlenecks in these capabilities that pose interesting research directions. Specifically, we annotate each benchmark with the cells that describe intent to determine the theoretical performance limit of a given context selection strategy, we evaluate performance of small and large models on different representations of context---including a dense format with high theoretical coverage at low token counts---and we fine-tune smaller language models to show that we can learn to understand such dense formats. Additionally, we perform quantitative and qualitative analysis on failure modes to understand bottlenecks in the formula writing capability---the best configuration only achieved 58.8\% correct suggestions.