NeurIPS 2026 Papers

Open-world semi-supervised learning aims to use limited labeled data from known classes to classify unlabeled samples that contain both known and novel classes. Most existing methods over-rely on known classes and tend to generalize incorrectly in complex open-world settings. This leads to degraded pseudo-label quality and the model is often overconfident in wrong predictions. To address this, this paper proposes a **M**itigating **C**onfidence **M**iscalibration Method in **Open**-World Semi-Supervised Learning (**OpenMCM**). By leveraging discriminative knowledge from known classes to guide learning in the novel class space, our method effectively mitigates overconfident misclassifications and improves discrimination accuracy. Furthermore, we propose an adaptive threshold calibration strategy to independently determine optimal decision boundary for each class. By integrating a high-reliable pseudo-label fusion mechanism, we enhance recognition stability for novel classes. Experiments on three benchmark datasets ($i.e.$, CIFAR-10, CIFAR-100, and ImageNet-100) show that our framework achieves performance improvements over existing state-of-the-art methods.

Learning solution operators for partial differential equations (PDEs) requires models that can represent spatial structure across grids and scales. This becomes especially important in time-dependent problems, where the learned operator is often used autoregressively: each predicted state is fed back as input to predict future states. In this setting, small spatial errors can be repeatedly propagated through the learned dynamics, leading to instability and drift, vitiating long-term solution fidelity. We propose Mesh-Free Convolutions (MFCs), filters defined in continuous space and discretized only at evaluation time. This allows the same learned filters to be applied across resolutions. Rather than imposing a grid-dependent stencil or a fixed spectral cutoff, MFCs learn smooth scale-dependent filtering and include directional terms for transport-like behavior. Across fluid-dynamics and weather-forecasting benchmarks, MFC-based models produce stable long-horizon rollouts, transfer reliably across resolutions, and better preserve long-time statistics. These results suggest that continuous, mesh-free filtering is a useful inductive bias for PDE operator learning, especially in autoregressive prediction.

Low-budget molecular optimization is often framed as Bayesian optimization in a learned latent space, but standard benchmarks typically allow hundreds or thousands of evaluations, whereas a realistic wet-lab round may allow only a few dozen. We revisit this regime using a SELFIES molecular variational autoencoder and GuacaMol objectives at $N=30$ oracle calls. The main finding is not a new surrogate model, but a geometric mismatch in candidate generation. The training latents of the VAE concentrate near a narrow spherical shell, while many high-dimensional optimizers propose candidates from boxes or box-shaped trust regions. Such candidates can leave the decoder's training distribution even when their coordinates look individually plausible. We study a simple geometric constraint, the on-sphere scaffold, that restricts candidate generation to the empirical shell occupied by the VAE training latents. Uniform random search on this shell already outperforms most of the classical box-supported Bayesian optimization baselines. We then introduce two minimal optimizers to probe what remains useful once the support is fixed. GeoWalk performs geodesic local search on the empirical sphere and outperforms every published Bayesian optimization baseline tested. GeoCMA keeps the scaffold but adds covariance learning in a random subspace, becoming preferable when the search representation contains learnable directional or semantic structure, as in LassoBench, NAS-Bench-101 graph-VAE optimization, and GSM8K text-latent prompt optimization. A controlled ablation of three recent Gaussian-process recipes, changing only whether candidates are proposed on the empirical sphere or on the published box-supported domain, shows that the scaffold rather than the surrogate is the load-bearing component. The practical lesson is simple: first match the empirical search support; add a surrogate or covariance learner only when the remaining search space contains directional structure that can be learned from the available budget.

Retrieval-Augmented Generation (RAG) has become a standard approach for grounding large language models in external knowledge, yet its performance depends critically on a large number of interacting design choices. Systematic optimization of these choices remains costly, as each configuration must be evaluated through full pipeline execution. In this work, we propose to treat RAG optimization as a prediction problem, using surrogate models to approximate the functional dependencies between configuration parameters and performance metrics. Using a stage-wise configurable RAG framework, we conduct large-scale experiments across three chemistry QA domains, systematically evaluating 648 configurations per domain and publicly releasing the resulting datasets of configurations and evaluation metrics. Surrogate models trained on this data achieve strong predictive accuracy ($R^2 \geq 0.84$ across all metrics and domains), and SHAP-IQ-based analysis of the learned models reveals consistent parameter interaction patterns across domains — despite substantial differences in corpus characteristics. We exploit this cross-domain regularity in transfer experiments: surrogate predictions generalize to unseen domains without target-domain training data on two transfer-robust metrics, and achieve full transfer across all six metrics with as little as 10\% of target-domain data. Finally, we apply surrogate models to optimize RAG configurations over an expanded search space of 61,152 candidates, recovering top-performing configurations in seconds rather than days, with gains of up to +0.09 in key retrieval and generation evaluation metrics. Together, these results demonstrate that RAG design spaces are structured, transferable, and efficiently optimizable via surrogate modeling.

Neural surrogate models for physical simulations are trained on discretized samples of continuous domains, where the induced empirical measure leads to uneven supervision, biasing optimization and causing spatial inconsistencies in physical fidelity. To mitigate this measure-induced bias, we propose M$^3$ (Multi-scale Morton Measure), a scalable framework that balances training measures by partitioning space according to physical variation and allocating supervision across multiple scales. Applied to three industrial-scale datasets with diverse discretizations, M$^3$ consistently improves predictions in the continuous physical domain, achieving up to 4.7$\times$ lower error in large-scale volumetric cases. These gains persist under aggressive subsampling (160M $\rightarrow$ 16M $\rightarrow$ 1.6M points), where M$^3$-trained models outperform those trained on higher-resolution data, reducing physics-weighted relative $L_2$ error by 3--4$\times$ and the corresponding MSE by up to 13$\times$. These results highlight data distribution as a key factor in operator learning and position M$^3$ as a scalable, data-efficient approach for physically consistent modeling.

Continual learning studies how models can adapt to new tasks while retaining previously acquired knowledge. Although a broad spectrum of methods has been proposed to mitigate catastrophic forgetting, the field remains predominantly performance-driven, with limited insight into what forgetting actually corresponds to within the vision model's representation space. Prior work has primarily analyzed forgetting through task-level performance or coarse measures of representational drift, without disentangling output-level accessibility from changes in finer-grained internal structure.To this end, we propose a diagnostic framework that leverages Sparse Autoencoders (SAEs) to define a task-anchored latent feature space, enabling analysis of how task-specific information evolves at a finer granularity, where individual SAE latents are treated as concept proxies for recurring and relatively disentangled visual patterns in the model’s internal computations. Within this framework, we decompose forgetting into apparent concept deletion, recoverability, and decodability. We show that a large portion of seemingly lost concept-level information can often be recovered under linearity assumption, with concept decodability degrading as more tasks are introduced. Overall, our findings suggest that a significant part of concept-level forgetting can be attributed to changes in the representational accessibility rather than complete information erasure.

Fine-tuned LLMs can learn complex and subtle behaviors. However, it can be difficult to ascertain what behaviors the model acquired during fine-tuning. We introduce LoRAcles: fine-tuned language models that take LoRA adapter weights as input, and answer natural language questions about them. We introduce a self-supervised pipeline for training LoRAcles: we first train LoRAs on small sets of pre-training documents, and then train LoRAcles on these LoRAs to answer questions about the documents. LoRAcles generalize far beyond their training data: they are state-of-the-art on AuditBench, a benchmark of models with sophisticated hidden behaviors; they can detect subtle changes in model organisms (fine-tunes deliberately constructed to encode a hidden behavior for auditing research), such as subliminal learning; and they are the first tool to achieve non-trivial performance at verbalizing semantic backdoor triggers. LoRAcles can also describe some learned behaviors from the weights of a full-parameter fine-tune. LoRAcles are a highly scalable method: we train LoRAcles for Qwen-3-14B and Llama-3.3-70B, and observe that performance scales smoothly with the size of our training dataset up to 100K LoRAs, though our method may need refinement in order to scale further. We note, though, that LoRAcles are prone to hallucinations and currently only elicit the most salient behavioral changes, limiting their utility beyond narrow fine-tunes.

Many real-world applications require highly interpretable machine learning systems, particularly in high-stakes domains such as healthcare. A promising recent direction is the paradigm of Maximal Coding Rate Reduction (MCR$^2$), which seeks to characterize and preserve the low-dimensional structure underlying each class. However, we find that exhaustively preserving structural information is often unnecessary and can even be detrimental in supervised learning, especially under noisy and uncertain real-world conditions, as it may overly constrain the fitting flexibility of the model. In contrast, we propose a simple yet effective framework, termed SimCoding, which captures only minimal structural information for each class. Despite relying on substantially less structural information, SimCoding significantly improves the generalization ability and robustness of deep models. Extensive experiments across diverse benchmark datasets demonstrate the effectiveness of SimCoding. We further validate SimCoding on a challenging real-world application, Parkinson’s disease severity assessment from free-living human activity signals, where it consistently achieves strong robustness and outperforms competing methods. Moreover, SimCoding can be naturally extended to incremental learning scenarios, where it substantially alleviates catastrophic forgetting and consistently surpasses strong baselines.

Self-supervised monocular depth estimation is appealing for its potential to scale with unlabeled data, yet existing methods often degrade when trained on diverse scenes. This study finds that, under self-supervision, models tend to encode dataset-specific structural biases rather than transferable scene geometry, leading to cross-domain optimization conflicts. To address this issue, this study proposes the MaSS framework, aiming to make self-supervised depth learning scalable to diverse data. MaSS introduces a dedicated structural representation pathway explicitly decoupled from contextual features, which is guided by a novel scene-structure discrimination objective to learn generalized, layout-aware structural representations for depth decoding. Trained and evaluated on four diverse datasets, both individually and jointly, MaSS substantially outperforms prior self-supervised methods and is the first to consistently benefit from mixed-domain training. Zero-shot evaluation on eight additional datasets further demonstrates its remarkable generalizability, particularly on those entirely out-of-distribution scenes. The source code will be publicly available upon publication.

A point source in a PDE does not occupy an entire grid. A weather station reports one value at one site; it does not observe the temperature between stations. A pickup event in a city occurs at a coordinate and a time; it is not born as an image. Yet many neural-operator pipelines begin by turning such sparse observations into dense surrogate fields before learning. This interpolation step is often treated as harmless preprocessing, but in sparse regimes it changes the object on which the operator is asked to act. We introduce the Dirac Neural Operator (DIRAC-NO), a measure-native neural operator for sparse event-to-field learning without interpolation. DIRAC-NO keeps observations as atomic event measures until query time and constructs the output field through adaptive Dirac-kernel aggregation. Because this aggregation is implemented by direct summation, the front end preserves source additivity at the aggregation stage and admits a learned Green-kernel view of source-to-field response. Across controlled point-source PDE benchmarks and real sparse reconstruction tasks, DIRAC-NO exposes the cost of interpolation-first learning. On MeasureBench-1D, it achieves relative $L^2$ error 0.015 at $n=128$, compared with 0.49 for FNO, and it better preserves superposition structure in trained models. In 2D, it is strongest under parameter and density shift, while ERA5 reconstruction shows that the representation advantage persists beyond analytic PDEs at high observation budgets. Boundary cases and negative-control event streams further show that the benefit appears where the representation argument predicts: sparse, event-native tasks with source-response structure. These results suggest that sparse operator learning should be organized not only by the neural backbone, but by the native space in which the input is represented.

Understanding the internal “thinking” process of Large Language Models (LLMs) and the cause of hallucinations remains a key challenge. To this end, we introduce latent debate, a structured surrogate framework for interpreting model outputs on True/False prediction tasks through the lens of internal latent arguments and interactions amongst them. Unlike human debates, latent debate captures the hidden supporting and attacking signals that arise within a model during a single inference. We first present a model- and task-agnostic conceptual framework, and then instantiate it symbolically to approximate the thinking process of LLMs towards binary decisions. Empirical studies demonstrate that our latent debate is a faithful structured surrogate model that has highly consistent predictions with the original LLM, while providing a form of interpretability. We also demonstrate that our latent debate provides a strong baseline for hallucination detection. Specifically, we identify strong correlations between debate patterns and hallucinations, such as a high degree of disagreement in the middle layers of the latent debate surrogate is linked to a higher risk of hallucinations. Our findings suggest that latent debate shows potential to analyze internal signals in LLMs for binary decision settings.

Modern Bayesian optimization and adaptive sampling methods increasingly rely on nonlinear parametric models, yet theoretical guarantees for such models under adaptive data collection remain limited. Existing analyses largely focus on Gaussian processes, kernel machines, linear models, or linearized neural approximations, leaving a gap between theory and the nonlinear models used in practice. We develop a kernel-based framework for analyzing regularized nonlinear parametric models trained on adaptively collected data. Our approach uses kernels over the parameter space to induce reproducing-kernel Hilbert space structures over the corresponding model class, yielding confidence bounds for models trained with broad classes of regularized convex losses. We show how these bounds can support convergence guarantees for nonlinear acquisition and surrogate models, including randomized regularized policies that select points by maximizing a trained random model. These results provide a unified route to analyzing nonlinear parametric models in Bayesian optimization and related adaptive optimization settings.

Operator learning has experienced a shift toward attention-based architectures. This trend has more recently extended to physics-informed operator learning, where stacked cross-attention blocks add considerable computational cost relative to the foundational PI-DeepONet. We ask whether this complexity is necessary. Starting from the DeepONet branch-trunk decomposition, we replace the bilinear readout with a single softmax-weighted readout in which the trunk emits a query vector and the branch emits a set of key-value pairs—a structure we show admits universal approximation via reduction to a simplex-bilinear form. The resulting architecture, KerONet, uses no iterated attention, layer normalization, or explicit learnable projection matrices. Under a unified hyperparameter optimization protocol with a strict $200{,}000$-parameter budget, we evaluate KerONet alongside iterated-attention architectures (PIT, PINTO) and PI-DeepONet on four canonical 1D PDEs whose solutions range from globally smooth to sharply localized. KerONet outperforms PIT and PINTO on every PDE, across all five seeds, in accuracy (approximately $2$–$4\times$ lower rel-$L^2$ error), reproducibility ($2$–$31\times$ lower standard deviation), and speed ($1.8$–$4.4\times$ faster training time). Against PI-DeepONet, KerONet shows substantial gains where input-dependent localized features dominate ($2.5$–$8\times$ lower error on Burgers and Allen-Cahn) while exhibiting comparable accuracy on smoother operators (advection and diffusion-reaction)–a balance the iterated-attention architectures do not achieve. Our findings demonstrate that iterated cross-attention is not necessary to realize the benefits of query-dependent feature mixing in physics-informed operator learning.

Neural surrogate models for computational fluid dynamics (CFD) are typically trained as forward operators that map explicit problem specifications, such as geometry and boundary conditions, to solution fields. This ties the model to the conditioning variables seen during training and limits reuse under boundary-condition shifts or local geometry changes. We propose to reformulate steady CFD inference as an inpainting problem: instead of training on explicit boundary conditions, we learn a self-supervised prior over velocity fields and impose boundary constraints only during inference by fixing known regions such as inlet, outlet or unchanged regions from previous simulations. To scale this idea to large 3D meshes, we introduce a local neighbourhood tokeniser that represents high-resolution velocity fields as compact spatial latent tokens and train latent flow-matching and masked-autoencoder models on these tokens. On intracranial aneurysm hemodynamics, our method reconstructs full velocity fields from sparse boundary context, outperforms supervised neural surrogates under boundary-condition and dataset shift and enables local geometry editing by reusing unchanged simulation context. These results suggest that viewing CFD inference as context-conditioned inpainting can turn neural surrogates from task-specific predictors into reusable flow priors.

Neural operators have emerged as powerful surrogates for scientific computing, enabling rapid reconstruction of complex physical fields from sparse sensor observations. Despite their strong predictive capability, their reliability often degrades under noisy or corrupted measurements, limiting practical deployment in real-world sensing environments. We show that enforcing statistical independence in latent representations provides a simple yet effective mechanism for improving neural operator robustness. To this end, we introduce Neural Operator Independence Regularization (NOIR), an end-to-end training framework that promotes statistically independent latent features during operator learning. Unlike structured basis approaches such as POD-DeepONet and PCA-Net, which impose orthogonality primarily as an offline compression constraint, NOIR directly shapes latent representations during training to enhance resilience against input perturbations. Across three canonical PDE benchmarks and five additive sensor corruption settings, NOIR consistently improves reconstruction accuracy while preserving clean-data performance. Beyond predictive gains, the induced latent representations exhibit distinctive statistical signatures that reveal architecture-specific robustness characteristics under corruption. These findings identify latent statistical independence as a principled inductive bias for robust neural operator design and provide new insight into the internal structure of operator learning systems. All models, datasets, checkpoints, and training code are publicly available at \url{https://github.com/t55176853-cmyk/NOIR/}

We study signSGD in linear regression with diagonal features and power-law spectra, a toy but expressive setting where the effect of coordinate-wise sign normalization can be analyzed sharply. Specifically, we establish a functional scaling law (FSL) for signSGD under general learning rate schedules, which decomposes the loss dynamics into two components: learning of the target signal and noise accumulation. Compared with SGD, signSGD learns the signal faster but exhibits slower noise forgetting. This signal-noise tradeoff yields a regime-dependent comparison between signSGD and SGD: signSGD achieves better data-scaling efficiency in hard-task regimes, while they perform similarly in easy-task regimes. The key mechanism is that gradient noise is {\it curvature-aligned}: its directional variance is of the same order as the directional curvature. As a result, sign operation effectively acts as a \(\diag(\bH)^{-1/2}\) preconditioner in the noise-dominated regime, where $\bH$ is the Hessian matrix. Moreover, we demonstrate that the same square-root preconditioning mechanism can arise in RMSprop and Adam, where coordinate-wise normalizers essentially track the coordinate curvatures. Synthetic experiments validate the proposed mechanism and the predicted scaling behavior. Language model experiments further suggest that, beyond the controlled setting, the resulting FSL can serve as a surrogate model for fitting and predicting practical training loss curves.

Understanding the behavior of stochastic gradient methods is a central problem in modern machine learning. Recent work has highlighted diagonal linear networks as a simplified yet expressive setting for analyzing the optimization and generalization properties of neural models. In this work, we show that in the high-dimensional regime, stochastic gradient descent on diagonal linear networks is well-approximated by continuous dynamics governed by a stochastic differential equation (SDE), which explicitly decouples the drift from the gradient noise. We further derive a deterministic partial differential equation whose solution propagates the relevant state of the iterates and characterizes the time evolution of a broad class of observable statistics, including the risk, curvature, and other metrics for optimality. To the best of our knowledge, this is the first deterministic high-dimensional description of SGD for a genuinely nonlinearly parametrized model. Finally, we show that, under a suitable parametrization, the stochastic dynamics are globally well posed and converge exponentially fast to zero risk with high probability, yielding a fully explicit non-asymptotic description of their long-time behavior. Numerical simulations corroborate our theoretical findings.

Multi-objective learning endeavors to concurrently optimize multiple objectives using a single model, aiming to achieve high and balanced performance across diverse objectives. However, this often entails a complex optimization problem to balance the learning of potentially conflicting objectives, leading to solutions with higher memory requirements and computational complexity. This paper introduces a Multi-Objective Goal-Conditioned Supervised Learning (MOGCSL) framework for automatically learning to achieve multiple objectives from offline sequential data. MOGCSL extends the conventional GCSL method to multi-objective scenarios by redefining goals from one-dimensional scalars to multi-dimensional vectors. It benefits from naturally eliminating the need for complex architectures and optimization constraints. Moreover, MOGCSL inherently disentangles uninformative or noisy training instances that fail to achieve desirable long-term rewards across multiple objectives. We also introduce a novel goal-selection algorithm for MOGCSL to model and identify desired and achievable goals for inference. In this paper, we focus on its application to the next action prediction problem in commercial-grade recommender systems. In this context, any viable solution needs to be reasonably scalable and also be robust to large amounts of noisy data that is characteristic of this application space. We show that MOGCSL performs admirably on both counts by extensive experiments. Also, analysis and experiments are included to explain its strength in discounting the noisier portions of training data in recommender systems with multiple objectives. Code is available at: https://anonymous.4open.science/r/MOGCSL-D7A2.

Neural operators provide fast surrogates for PDEs but their deterministic predictions limit their use in tasks requiring uncertainty quantification (UQ), especially under geometric variability. Existing approaches primarily model uncertainty in network parameters, largely overlooking the geometry-aware representations learned by the operator itself. We propose REEF-GP (Residual on Embedded Features Gaussian Process), a post-hoc UQ framework that fits a GP to the residuals of a frozen neural operator whose internal embeddings define the kernel feature space. Rather than learning a separate feature map, REEF-GP adapts the operator’s intrinsic coordinate-feature representations to construct geometry-aware uncertainties. To ensure stability and scalability on unstructured domains, REEF-GP incorporates spectral-normalized projections, heteroscedastic geometry-aware noise, and efficient subset-based training that avoids restrictive low-rank approximations. Across five PDE benchmarks with varying geometries, REEF-GP preserves predictive accuracy while achieving calibrated uncertainty estimates competitive with deep ensembles but at a fraction of their cost. Our approach remains robust under geometric distribution shift, with uncertainty concentrating in physically meaningful regions (e.g., shock fronts). Our results demonstrate that accurate and scalable post-hoc UQ for neural operators can be achieved directly in their learned feature space, offering a practical alternative to parameter-centric approaches.