NeurIPS 2026 Papers

Multimodal large language models (MLLMs) have heterogeneous strengths across OCR, chart understanding, spatial reasoning, visual question answering, cost, and latency. As a result, effective MLLM routing requires more than estimating query difficulty: a router must match the multimodal requirements of the current image-question input with the capabilities of each candidate model. We propose \textsc{LatentRouter}, a router that formulates MLLM routing as counterfactual multimodal utility prediction. Given an image-question query, \textsc{LatentRouter} extracts learned multimodal routing capsules, represents each candidate MLLM with a model capability token, and performs latent communication between these states to estimate how each model would perform if selected. A distributional outcome head predicts model-specific counterfactual quality, and a bounded capsule correction refines close decisions while limiting the influence of noisy routing signals. The resulting utility-based policy supports both performance-oriented and performance-cost routing, and can handle changing candidate pools through shared per-model scoring with availability masking. Experiments on MMR-Bench and VL-RouterBench show that \textsc{LatentRouter} outperforms strong fixed-model, feature-level, and learned-router baselines. Additional analyses show that the gains are strongest on multimodal task groups where model choice depends on visual, layout-sensitive, or reasoning-oriented requirements, and that latent communication is the main contributor to the improvement. The code is available at: https://anonymous.4open.science/r/LatentRouter-8718.

Transformer-based large language models are increasingly used for long-horizon tasks; however, their attention mechanism scales poorly with context length. To handle this, we study a sleep-like consolidation mechanism in which a model periodically converts recent context into persistent fast model weights before clearing its key-value cache. During the sleep phase, the model performs $N$ offline recurrent passes over the accumulated context and updates the fast weights in its state-space model (SSM) blocks through a learned local rule. This shifts extra computation to the sleep phase while preserving the latency of wake-time prediction. We test our method on controlled synthetic tasks, including cellular automata and multi-hop graph retrieval, as well as a more realistic math reasoning task, on which a regular transformer as well as SSM-attention hybrid models fail. We then show that increasing sleep duration $N$ for our models improves performance, with the largest gains on examples that require deeper reasoning.

Large language models frequently ignore in-context evidence that contradicts their parametric priors, producing answers that reflect default beliefs rather than the supplied passage, a phenomenon termed *knowledge conflict*. This failure is commonly attributed to insufficient encoding of the contextual signal. We challenge this explanation. Through systematic, layer-by-layer analysis in a controlled multi-hop question-answering setting, we show that models faithfully encode the context-supported answer in their intermediate representations, yet suppress it in the final layers before generation. We term this *late-layer reversal*: the representational support for the evidence answer rises through the middle layers, then collapses in the final ~40\% of the network. To explain this phenomenon, we identify a *Dual-Track* computational structure within the transformer's hidden state. A *knowledge direction*, recovered by a learned linear probe, encodes the contextually supported answer; a nearly orthogonal *output direction*, aligned with the unembedding geometry, determines the model's prediction. These two directions are causally severed: interventions along the knowledge direction produce no change in output, whereas targeted suppression of prior-restoring computation along the output direction reliably causes the model to follow the contextual evidence. Our findings hold across nine model-dataset combinations and reveal that knowledge conflict failures arise not because models fail to represent contextual knowledge, but because their generation pathway declines to consult it.

We study label-free, topology-conditioned zero-shot cross-graph link prediction: at test time the observed target topology is the input graph, but the model receives no target labels, no positive/negative context links, no target fine-tuning, no text attributes, and no post-hoc alignment. This setting is zero-shot with respect to target supervision and adaptation, but it is not topology-free inference. We identify a critical geometric barrier for hyperbolic transfer: radial non-identifiability. Hyperbolic graph learning is motivated by the intuition that radius encodes hierarchy or popularity and angle encodes similarity, yet standard HGNN objectives do not make radii semantically calibrated across disjoint graphs. The same structural role can therefore occupy incompatible radial scales on different target graphs. We propose Invariant Hyperbolic Unfolding (IHU), which restores the intended radial hierarchy channel through selective invariance: radii are fixed by graph-internal structural percentiles while angular representations remain learnable. Its core module, Invariant Structural Anchoring (ISA), rank-canonicalizes structural scores such as coreness into shared hyperbolic radii, producing a comparable radial skeleton without target supervision. In a controlled comparison among matched frozen learned encoders, IHU improves average HR@50 by 2.7 percentage points over the strongest learned hyperbolic baseline and by 7.6 percentage points over the mean of the learned hyperbolic baseline family. The gain amplifies under missing-edge and low-degree regimes, while hierarchy-gap analysis reveals a boundary condition for fixed radial shells: 50% edge sparsification causes 52.5% less degradation, and low-degree nodes show a 4.2× larger advantage. We further position IHU against recent universal link prediction and graph foundation methods through an access-aware taxonomy, showing that the closest recent LP methods use labeled target context links, whereas IHU isolates the geometric effect of fixed versus learned radii in frozen structural hyperbolic encoders.

Modern image encoders achieve high generalization by decoupling semantic meaning from resolution, an ability yet to be fully realized in the 3D domain. We investigate the failure of 3D point cloud encoders to achieve similar generalization and find that existing models are highly sensitive to sampling resolution and scale changes, leading to significant performance degradation. This sensitivity is a major bottleneck for real-world deployment in robotics, as it suggests models overfit to specific quantization densities and object scales rather than learning invariant semantic features. To mitigate this dependency, we propose Invaria, a point cloud encoder that achieves scale and density invariance through next-resolution prediction and receptive field calibration. While our objective is not the explicit generation of high-resolution point clouds, we find that this training objective encourages the model to learn robust, structural invariants. The resulting encoder achieves significant performance gains during resolution shifts while maintaining high efficiency through a compact model size and reduced token requirements. Specifically, on ScanNet, Invaria achieves a 56.0\% higher mIoU at 3$\times$ lower resolution and a 20\% improvement when the objects scale is reduced by a factor of 3. These gains are achieved with a 45\% smaller model size and an average reduction of 40\% in input tokens. The code will be released once the paper is published.

Video-generative world models are increasingly used as neural simulators for embodied planning and policy learning, yet their ability to predict physical risk and severe consequences is rarely evaluated.We find that these models often downplay or omit key danger cues and severe outcomes for hazardous actions, which can induce unsafe preferences during planning and training on imagined rollouts. We propose ICAT, which grounds testing in real incident reports and safety manuals by building structured risk memories and retrieving/composing them to constrain the generation of risk cases with causal chains and severity labels. Experiments on an ICAT-based benchmark show that mainstream world models frequently miss mechanisms and triggering conditions and miscalibrate severity, falling short of the reliability required for safety-critical embodied deployment.

We study models for human-AI teaming through the lens of statistical calibration. We assume the team consists of an AI model and human---both of which are calibrated with respect to some partitioning of the feature space---and expose how the calibration assumptions propagate into the teaming framework. In particular, we consider frameworks that either (i) combine human and model predictions or (ii) delegate prediction responsibility to either a human or model. We show via theoretical and empirical results that existing methods for combination do not preserve the human's degree of calibration. Methods for delegation (by the very act of delegation) preserve calibration of the downstream predictors but can place high-capacity demands on the meta-classifier that determines which should predict. This latter result suggests human-AI complementarity and correct delegation place opposite demands on the delegation model.

Predicting how a person's first-person view will evolve (what action will follow, what plan completes a task, whether an in-progress shot will score) is fundamentally under-specified: the same context admits many plausible futures, and a model trained to minimize prediction error is forced to hedge or average across them, getting it wrong either way. Two findings shape our approach. First, the future camera trajectory, the path the head carves through space, lets the model commit to one of those futures: it carries the operator's intent in a form fine enough to determine how an action will unfold, substantially outperforming language as a conditioning signal. Second, this same intent makes the trajectory itself partially predictable from the context at hand, enough that trajectory need not be observed at test time to recover most of the gain. We instantiate these findings as TrajPilot, a model that predicts candidate future trajectories from egocentric context and uses them to pilot action prediction in an action-aligned embedding space where language shapes the structure but is never used as a conditioning input. TrajPilot beats VLM and structured-planner baselines on procedural planning across Ego-Exo4D atomic, Ego-Exo4D Keystep, Ego4D GoalStep, and EgoPER, with the trajectory advantage widening with horizon (exactly where prior planners collapse) and holding under RGB-only camera-pose estimation. With the goal masked at inference, the same model performs goal-free anticipation, beating VLM baselines on Ego-Exo4D atomic and extending to EPIC-Kitchens-100 and basketball shot-outcome prediction.

Post-training pruning reduces the memory and computation costs of large language models (LLMs), and N:M semi-structured sparsity provides a hardware-friendly pruning pattern for efficient inference. However, pruned models are often evaluated mainly by perplexity and average downstream accuracy, which are useful aggregate metrics but do not fully explain how pruning changes individual candidate-level decisions. We evaluate magnitude pruning, Wanda, SparseGPT, Wanda++, and our proposed MR-GOBS across multiple model families and sparsity settings. We further conduct a paired decision-level evaluation that measures true-label confidence, answer margin, prediction flips, candidate-level distribution shift, and near-boundary vulnerability. Our results show that pruning weakens decision reliability by lowering confidence in the correct answer, shrinking answer margins, and increasing correct-to-wrong flips, especially on near-boundary examples. To examine whether reconstruction-oriented improvements reliably improve decision reliability, we revisit SparseGPT through its local least-squares reconstruction formulation and refine its N:M mask-selection step. MR-GOBS uses GroupOBS, a set-level OBS reconstruction cost for jointly pruning a group of weights, to score each legal local N:M mask. It then selects masks using a minimax-regret criterion over multiple calibration views. Although MR-GOBS often improves average perplexity over SparseGPT, these gains do not always translate into better downstream accuracy or decision-level metrics. Our findings suggest that reliable evaluation of pruned LLMs requires both aggregate performance metrics and decision-level diagnostics.

Existing conformal prediction methods for time-to-event outcomes leverage only baseline covariates, producing prediction intervals that are insufficiently informative to facilitate decision making. We propose History-Aware Prediction Sets (HAPS), a conformal framework that constructs prediction sets for individual event times using covariate histories observed up to a decision time, targeting coverage among individuals who have survived to this time. HAPS handles right censoring adjusted for time-varying confounders via inverse probability of censoring weighting. When the censoring weights are consistently estimated, it achieves PAAC (probably asymptotically approximately correct) coverage among survivors. We further propose two doubly robust extensions of HAPS to weaken reliance on consistent estimation of the censoring distribution. In simulations, HAPS and its extensions reduce median prediction interval length by up to 75\% relative to baseline comparators while maintaining close to nominal coverage. On two public benchmark data sets, HAPS reduces the median interval length by up to 60\% for predictions at year 5, compared to the baseline comparators.

Glycosylation is among the most diverse and important post-translational modifications in biology, governing immunogenicity, self-recognition, and the clinical viability of biologic drugs. Glycans cannot be sequenced from a template, and de novo structure prediction from tandem mass spectrometry (MS/MS) remains a longstanding bottleneck. The current state of the art, GlycoBART, is a $207$M autoregressive transformer whose $O(\mathrm{beam} \cdot \mathrm{output\_length})$ decoder passes preclude real-time annotation. We introduce $\textbf{GlycoGen}$, a $50$M Discrete Flow Matching (DFM) model that predicts glycan structure from MS/MS spectra by direct, non-autoregressive generation in a few parallel forward passes. We pair it at inference time with $\textbf{Crystallizing Flows}$, a deterministic sampler that propagates the per-token distribution on the probability simplex and freezes each token to a one-hot (*crystallizes* it) the moment its argmax leaves the mask, exiting early once all tokens are unmasked. The sampler feeds the denoiser mixture embeddings, and runs out-of-the-box with no retraining. GlycoGen + Crystallizing Flows reach $\mathbf{0.9266}$ top-1 structural accuracy on the CandyCrunch test set, three orders of magnitude faster than GlycoBART, Pareto-dominating the prior state of the art on *both* accuracy *and* speed. We extend our experiments to a matched-architecture MDLM denoiser (Sahoo et al. 2024) and observe a comparable lift: the sampler generalizes from continuous-time flow matching to discrete-time masked diffusion. To our knowledge, this is a new state of the art for de novo open-vocabulary glycan structure prediction, and the first system fast enough for accurate real-time inline annotation on modern LC-MS/MS instruments.

Conformal prediction provides model-agnostic uncertainty quantification with guaranteed coverage, but conventional methods often yield overly conservative uncertainty sets, particularly in multimodal or heterogeneous settings. This inefficiency arises from two sources: (i) limited expressiveness of the predictive model and (ii) simplistic nonconformity scores design. Most existing approaches advance only one of these axes, leaving the other underexplored. We propose generative conformal prediction with **O**ptimized **R**anking and **C**overage **A**llocation (ORCA), a three-stage framework that advances both aspects jointly. ORCA leverages generative models to capture the full conditional distribution and introduces a rank-dependent optimization procedure that adaptively allocates coverage for efficiency while maintaining validity. We cast this coverage allocation as an optimization problem, derive an exact mixed-integer linear programming formulation, and show that the solution converges asymptotically to the oracle density-level set. Across synthetic, semi-synthetic, and real datasets, ORCA produces substantially more efficient uncertainty sets than state-of-the-art baselines, demonstrating robust gains in scenarios where conventional conformal prediction methods fail.

Model merging aims to combine multiple models into one without additional training. Naïve parameter-space averaging can be fragile under architectural symmetries, as their geometry does not take them into account. In this work we show that not only the geometry, but also the averaging procedure itself, must be symmetry-invariant to achieve symmetry-aware merges. Consequently, we propose a general solution: merging as Fréchet averaging, i.e., selecting parameters that minimize a sum of geodesic distances on an appropriate manifold. In this view, the key design choice is the overall geometry, i.e., the choice of metric, manifold, and distance approximation, that determines what it means for two models to be “close.” We show that Fréchet averaging, combined with simplifying assumptions, contains Fisher merging. Building on this, we examine the particular case of low-rank adapters (LoRA), whose symmetries induce a distinct geometry: that of a quotient manifold. We outline the limitations of current LoRA merging methods, propose a practical algorithm for this setting, and show how they compare with other commonly used approaches.

Explaining the generalization and training dynamics in neural networks remains a challenge, and various approaches have been developed to study different aspects of these phenomena. In this paper, we introduce the *magnitude potential* to this study -- a quantity based on the theory of metric magnitude -- that reflects how well an arbitrary point is represented by a given set. We find that this basic quantity can be applied to examine various features in neural generalization. The ratio between the magnitude potential with respect to a class and with respect to the entire data, computed at the logit layer, is informative of the representation of the point. In experiments, these ratios for individual training points are found to be correlated with the Feldman memorization scores. Magnitude potential ratios aggregated across points detect structural changes in the decision boundaries and provide a geometric indicator of grokking in modular arithmetic. Although the magnitude potential ratio and neural collapse are both closely associated with intra-class and inter-class geometric structure, the magnitude potential ratio remains informative even when neural collapse is explicitly suppressed.

Conformal prediction (CP) provides a distribution-free approach to uncertainty quantification (UQ) with finite samples. However, applying CP to graph neural networks (GNNs) remains challenging. The combinatorial nature of graphs makes their encoding non-trivial, leading to insufficiently uncertain prediction logits and indiscriminative embeddings. Existing methods mostly rely on the embedding space for conformal prediction, which can be unreliable for graphs and often yields inefficient prediction sets. We propose GRAPHLCP, a structure-aware weighted conformal prediction framework that explicitly incorporates the graph topology and inter-node dependencies into localization and weighting. Based on experimental results that show a strong correlation between node homophily and sample coverage, we propose GRAPHLCP that utilizes a feature-aware densification followed by Personalized PageRank–based kernel computation to model structural proximity between nodes. Accounting for both structural and feature similarity allows GRAPHLCP to efficiently guarantee marginal coverage while attaining favorable test conditional coverage across extensive experiments on multiple datasets for both regression and classification.

Predicting tandem mass spectra (MS/MS) from molecular structures represents a central task in analytical chemistry with direct relevance to clinical metabolomics, systems biology, and adjacent disciplines. In this work, we revisit the problem through the lens of object detection on molecular graphs. Molecular fragmentation, a central step in MS/MS prediction, can be approximated as detecting a set of subgraphs (i.e., fragments) and their associated spectral contributions. Existing fragment-based models follow a two-stage paradigm—first generating candidate fragments and then scoring them—analogous to two-stage R-CNNs in computer vision. Towards higher accuracy and faster inference, we introduce GLACIER, a single-stage transformer-based fragment detection neural network for molecular graphs. This unified formulation eliminates the need for candidate enumeration, enabling scalable and globally consistent modeling of molecular fragmentation. GLACIER is faster and more accurate than existing state-of-the-art methods by a significant margin, achieving 62.4% and 62.1% Top-1 retrieval accuracy with and without contrastive finetuning on the MassSpecGym dataset (from 55.3%), and 55.2% and 38.2%, respectively, on the NIST'20 dataset (from 33.5%). Furthermore, GLACIER provides nearly 3-fold inference speedup over existing two-stage models.

Link prediction is a standard pretraining objective for graph neural networks: hide edges, train an encoder--decoder to predict them, and reuse the node embeddings for downstream node tasks. Despite its empirical success, there is little quantitative theory explaining why an edge-level objective should produce embeddings that are linearly useful for node classification.We study this question in the finite-sample stochastic block model (SBM). For any bounded GNN encoder paired with a symmetric bilinear decoder, we prove a conditional reduction: small population held-out LP excess risk implies small community-classification error for a transductive linear classifier on the learned embeddings.The bound decomposes into an optimization residual, an architectural approximation residual, and explicit statistical terms of order $\widetilde{O}(\sqrt{d/n})$ in bounded dense regimes. We further give a constructive few-label classifier: nearest-centroid classification under the PSD quadratic form $G = W Z^\top Z W$, whose label error is coupled to LP quality.The framework is architecture-agnostic through the approximation residual, and we instantiate this residual as $O(\log n / n)$ for a spectral-positional-encoding GIN with a bilinear decoder. Synthetic SBM experiments validate the theorem chain from LP risk to score error to classification error, and show when the worst-case constants become conservative.

Visual state space models (SSMs) have emerged as efficient alternatives to attention-based backbones, yet their performance remains sensitive to how visual evidence is sampled and propagated. Deformable visual Mamba methods address fixed scanning by predicting adaptive sampling locations, but offset prediction is still driven primarily by local responses, producing locally plausible samples that are weakly coupled to the SSM's recurrent state evolution, particularly in off-center or multi-object scenes. We present **Foveal-Mamba**, a scan-path-aware deformable visual Mamba framework that reformulates offset prediction as recurrent scan-path prediction. Two components drive this design. First, an **Inside-Out Ring Scan** organizes visual tokens along a center-to-periphery path, providing a stable propagation prior that does not assume object centering. Second, a lightweight Mamba module embedded within the offset prediction network forms the **Recurrent Scan-Path Offset Prediction Network (RS-OPN)**, which predicts offsets from hidden states accumulated along the ring path. RS-OPN tightly couples deformable sampling with recurrent state evolution and produces coherent, object-aware sampling trajectories. Foveal-Mamba consistently outperforms strong baselines across multiple benchmarks. On ImageNet-1K, Foveal-Mamba-Tiny achieves 84.3\% Top-1 accuracy, surpassing DAMamba-Tiny by 0.5\% with fewer parameters and FLOPs. On ADE20K semantic segmentation, Tiny/Small variants reach 50.8/51.8 mIoU. On COCO, it achieves 48.9/50.2 box AP and 43.7/44.9 mask AP for object detection and instance segmentation, respectively. Qualitative results further demonstrate earlier object focus and more stable activations in multi-object scenes.

Machine unlearning aims to remove the influence of selected training examples without full retraining. Standard evaluations often summarize unlearning quality with aggregate metrics, such as accuracy- and forgetting-based scores, which can hide localized failures. We study this failure mode at the example level by comparing the predictions of an unlearned model to those of the model retrained after deletion. We show that this pointwise discrepancy can be highly non-uniform: for gradient-ascent and random-labeling methods, with and without retain-set fine-tuning, it grows with geometric proximity to the forget set. We call this phenomenon *localized collateral forgetting*. Our analysis identifies a mechanism behind the effect: surrogate targets used during unlearning can be inconsistent with the local prediction structure induced by retraining, and this inconsistency propagates through shared representations to nearby examples. Motivated by this mechanism, we propose *Local Teacher Distillation*, a simple mitigation strategy that replaces random targets with soft labels from a small teacher trained only on retained neighbors of the forget set. On CIFAR-100 partial-class deletion, this local teacher brings the unlearned model substantially closer to retraining, especially near the forget set, while maintaining competitive aggregate unlearning metrics.