NeurIPS 2026 Papers

AI evaluation results are produced at scale but reported inconsistently across leaderboards, model cards, benchmark papers, and company blogs. This fragmentation is borne at the level of interpretation: readers cannot reliably compare results across sources, identify what is missing from a given report, or trace an aggregate claim to the evidence behind it. Recent efforts address isolated components of this problem but leave three gaps unresolved: They cover only narrow slices of the evaluation lifecycle and do not compose into a single interpretable record. They specify static representations that do not differentiate between the questions technical and policy readers bring to the same evidence. And they remain proposals on paper, without the extraction infrastructure required for adoption at scale. We present EVALUATION CARDS, an operational reporting layer that composes benchmark metadata, evaluation run data, and model metadata into a unified record. We (1) derive a reporting schema from a structured literature review of 52 papers and 10 stakeholder interviews, (2) implement four interpretive signals (reproducibility, documentation completeness, provenance and risk, and score comparability), rendered through reader modes calibrated to research and policy audiences, and (3) provide a deployed monitoring tool that applies EVALUATION CARDS across 5,498 models, 635 benchmarks, and 101,843 results, surfacing systematic gaps in current reporting practice.

Imaging of dynamic cellular responses to perturbations requires live-cell measurements, posing a challenge for fluorescence-based assays that can visualize molecular phenotypes by specific labeling but often affect cell health. Virtual staining helps by predicting fluorescence-like signals from label-free microscopy. Although many models have been developed for this task, their performance on volumetric time-lapse data remains poorly characterized. We address this gap by introducing DynaCell, an evaluation framework for 3D time-lapse virtual staining of live cells, comprising a new paired label-free and fluorescence 3D time-lapse imaging dataset, a suite of baseline models, and a three-tier metric panel measuring pixel-level fidelity, organelle segmentation performance, and single-cell phenotypic similarity. Using DynaCell, we study model behavior across 2 cell types and microscopes, 4 organelles, and 3 perturbation states. We find that regression baselines better predict the spatial localization of target organelles and are more robust to label-free input distribution shifts, while the generative baseline better captures population-level phenotype distribution across cells and organelles. These differences expose trade-offs between preservation of structural and phenotypic fidelity that are obscured by single-metric evaluation, and suggest how virtual staining can support downstream tasks such as localization, segmentation, tracking, and phenotypic profiling. DynaCell provides reusable data, code, checkpoints, and documentation for assessing virtual stains for live-cell biological measurements.

Autoregressive language models are widely used for text evaluation, however, their left-to-right factorization introduces positional bias, i.e., early tokens are scored with only leftward context, conflating architectural asymmetry with true text quality. We propose masked reconstruction as an alternative paradigm, where every token is scored using full bidirectional context. We introduce DiffScore, an evaluation framework built on Masked Large Diffusion Language Models. By measuring text recoverability across continuous masking rates, DiffScore eliminates positional bias and naturally establishes an evaluation hierarchy from local fluency to global coherence. We further provide diagnostic tools unavailable to autoregressive frameworks: multi-timestep quality profiles that decompose scores across masking rates, and bidirectional Pointwise Mutual Information (PMI) decomposition that disentangles fluency from faithfulness. Experiments across ten benchmarks show that DiffScore consistently outperforms autoregressive baselines in both zero-shot and fine-tuned settings.

Evaluating LLM-generated interactive software requires execution in addition to static analysis. The key difficulty is that correctness is a graph-level reachability property over latent UI state-transition graphs, whereas a GUI evaluator observes only a single execution trajectory. A failed rollout therefore rules out only one realized path, leaving failure attribution ambiguous between evaluator-side execution error and genuine software defect. We present DIAGEVAL, a trajectory-conditioned diagnostic evaluation protocol for post-failure GUI-agent evaluation of interactive software. Rather than blindly retrying from scratch,DIAGEVAL reuses the failed trajectory to choose targeted diagnostic probes and aggregates their outcomes into an internal attribution signal. The latent-graph view motivates the diagnostic problem; DIAGEVAL does not reconstruct the graph or estimate calibrated posterior probabilities. We evaluate DIAGEVALon WebDevJudge-Unit and RealDevBench across multiple GUI-agent evaluators and LLM backbones. On false-negative cases, DIAGEVAL recovers 45.6--62.1\% of failures that were initially attributed to software defects, outperforming retry-based baselines with 34.4--160.6\% relative gains. On the full evaluation sets, this recovery improves accuracy from 69.9\% to 78.3\% on WebDevJudge-Unit and from 65.0\% to 81.6\% on RealDevBench. These results suggest that reliable GUI-agent evaluation requires not only stronger execution, but also active failure diagnosis to disambiguate evaluator-side errors from genuine software defects.

Dataset distillation has provided a reliable solution for data compression, where models trained on the resulting smaller synthetic datasets achieve performance comparable to those trained on the original datasets. To further improve the performance of synthetic datasets, various training pipelines and optimization objectives have been proposed, greatly advancing the field of dataset distillation. Recent decoupled dataset distillation methods introduce soft labels and stronger data augmentation during the post-evaluation phase and scale dataset distillation up to larger datasets. However, this raises a question: Is accuracy still a reliable metric to fairly evaluate dataset distillation methods? Our empirical findings suggest that the performance improvements of these methods often stem from additional techniques rather than the inherent quality of the images themselves, with even randomly sampled images achieving superior results. Such misaligned evaluation settings severely hinder the development of DD. Therefore, we propose DD-Ranking, a unified evaluation framework, along with new general evaluation metrics to uncover the true performance improvements achieved by different methods. By refocusing on the actual information enhancement of distilled dataset, DD-Ranking provides a more comprehensive and fair evaluation standard for future research advancements.

Training implicit neural representations (INRs) to capture fine-scale details typically relies on iterative backpropagation and is often hindered by spectral bias when the target exhibits highly non-uniform frequency content. We propose ELM-INR, a backpropagation-free INR that decomposes the domain into overlapping subdomains and fits each local problem using an Extreme Learning Machine (ELM) in closed form, replacing iterative optimization with stable linear least-squares solutions. This design yields fast and numerically robust reconstruction by combining local predictors through a partition of unity. To understand where approximation becomes difficult under fixed local capacity, we analyze the method from a spectral Barron norm perspective, which reveals that global reconstruction error is dominated by regions with high spectral complexity. Building on this insight, we introduce BEAM, an adaptive mesh refinement strategy that balances spectral complexity across subdomains to improve reconstruction quality in capacity-constrained regimes.

Foundation models for predictive machine learning on tabular data have recently gained significant traction in academia and industry. Research communities across disciplines are increasingly evaluating tabular foundation models on diverse datasets and tasks. However, these task- and discipline-specific evaluations remain largely inaccessible to model researchers because benchmark software and evaluation protocols are fragmented. As a result, model researchers rely on standard benchmarks, which are mostly defined for tasks where tabular foundation models already excel. The most challenging scenarios are excluded, limiting meaningful progress in the field by focusing on marginal improvements on IID data rather than on broader, more demanding challenges. To overcome this, we introduce BeyondArena, the first unified holistic benchmark for tabular data that supports diverse task types (IID, temporal, grouped), across sample size and feature dimensionality scales, with diverse feature types (with text, with high cardinality) from a broad range of disciplines. To enable unified benchmarking beyond standard benchmarks, we introduce DataFoundry, a Python framework and metadata schema for curating tabular datasets for predictive machine learning. Our results across 11 models and 142 curated datasets show that existing tabular foundation models excel on tiny- to medium-sized IID data, while traditional tree-based and deep learning models still dominate on non-IID, large, and high-dimensional datasets. BeyondArena guides model research for the most demanding challenges in tabular data, enabling progress towards truly foundational tabular models.

Adaptive immune receptor repertoire sequencing data has emerged as a promising potential modality for disease diagnosis, relying on computational methods to analyze T-cell receptor (TCR) sequences from an individual’s blood sample. Published methods rely on different cohorts, data preprocessing pipelines, and evaluation metrics, making direct comparison across methods challenging. We have developed BenchRep-T, a unified benchmark that standardizes multiple publicly available TCR repertoire datasets and evaluates nine computational approaches, spanning statistical enrichment of shared sequences, feature-engineered ensembles, deep learning, and graph-based clustering. BenchRep-T evaluates methods on four tasks: disease classification across conditions, performance scaling under restricted sequence-sampling depth, recovery of known antigen-specific driver sequences, and evaluation of sensitivity to demographic confounding. Under controlled evaluation, simple baselines prove competitive, with tree-based models trained on V- and J-gene usage and short sequence motifs approaching the classification performance of more complex methods. Our findings underscore the complexity of modeling TCR repertoire data, and show that no single method dominates across all tasks. BenchRep-T provides a framework for rigorous and reproducible evaluation of TCR repertoire classification methods to accelerate the development of immune repertoire-based diagnostics.

Modern transformation-aware self-supervised representations are routinely scored under protocols that grant the evaluator paired views, augmentation labels, or teacher anchors --- none of which the deployed predictor ever sees. We audit the gap that opens once these privileges are removed and only a single query remains. The finding is a systematic evaluation illusion: on 3DIEBench-Honest, honest single-query rotation errors are typically $2{-}3\times$ as large as native transformation scores, consistently across comparators and matched-utility thresholds. To make this gap measurable, we introduce a deployment-honest evaluation contract specifying a one-query test interface, matched semantic utility, a strongest-eligible-neighbor headline with tail summaries, and explicit privilege accounting. Theory and experiments are in service of the contract: lower-bound analysis grounds the contract at the interface level, while a positive control supplies a non-vacuous upper certificate under declared privileges. We exercise the contract on a generated law, on 3DIEBench-Honest, and on PTB-XL. The operational conclusion is that native transformation-aware success is not, on its own, a single-query deployment certificate.

AI agent research spans a wide spectrum: from RL agents that learn from scratch to foundation model agents that leverage pre-trained knowledge, yet no unified benchmark enables fair comparison across these approaches. We present **Agentick**, a benchmark for sequential decision-making agents designed to evaluate RL, LLM, VLM, hybrid, and human agents on common ground and to power research on the fundamental challenges of sequential decision-making. Agentick provides 37 procedurally generated tasks across six capability categories, four difficulty levels, and five observation modalities, all exposed through a single Gymnasium-compatible interface. The benchmark ships with a Coding API, oracle reference policies for all tasks, pre-built SFT datasets, a composable agent harness, and a live leaderboard. An evaluation spanning 27 configurations and over 90,000 episodes reveals that no single approach dominates: GPT-5 mini leads overall at 0.309 oracle-normalized score while PPO dominates planning and multi-agent tasks; the reasoning harness multiplies LLM performance by 3--10x; and ASCII observations consistently outperform natural language. These findings highlight the substantial room for improvement that remains across all agent paradigms. Agentick's capability-decomposed, multi-modal design provides the empirical infrastructure needed to drive progress toward general autonomous agents, both as an evaluation framework and as a training ground for RL post-training of foundation models in truly sequential environments.

Modern deep learning commonly relies on AdamW with prescribed learning rate schedules, but recent works challenge both components: Schedule-Free optimization removes explicit schedules via iterate averaging, and Muon improves the update geometry by orthogonalizing momentum for matrix parameters. Despite Muon's strong empirical performance, its underlying mechanism remains partially understood. We study Muon through the river-valley loss landscape, where useful training progress occurs along a flat, low-curvature bulk subspace (the river), while high-curvature dominant directions form steep valley walls that induce oscillations. We empirically show that while Muon's orthogonalization accelerates river progress by increasing the bulk component, it also amplifies dominant-direction noise, causing oscillatory trajectories. Building on this, we propose Anytime MUon with Stable gradient Evaluation (AMUSE), which integrates Muon's rapid bulk progress with the stabilizing effect of Schedule-Free averaging. AMUSE uses a time-varying interpolation coefficient that initially evaluates gradients near the fast Muon sequence for rapid adaptation, then gradually shifts toward the stable averaged sequence to suppress valley-wall oscillations. As a result, AMUSE requires no learning rate schedules and supports anytime training. Across vision tasks and large language model pretraining, AMUSE consistently improves the performance-iteration Pareto frontier over (Schedule-Free) AdamW and Muon.

There are now thousands of measurement instruments used to evaluate language models, and the resulting scores are routinely used to support consequential claims about capabilities and risks. Yet, the evidence linking benchmark scores to constructs (i.e., abstract concepts such as reasoning ability or safety) is often implicit, incomplete, or absent. We introduce AI Construct Lexis, a versioned ontology and interactive artifact that extracts and organizes relationships among constructs, behaviors, and measurement instruments as claimed in the literature for LLM evaluation. Drawing on over 8,337 published papers from selected top AI venues, we use language models to extract constructs, definitions, behavioral indicators, and measurement instruments, yielding 977 distinct constructs, 1,694 measurement instruments, 830 behavioral indicators, and 5,900 relationships. The Lexis captures relationships claimed in the published literature, not necessarily validated relationships; valid score interpretation still requires empirical evidence that the instruments capture the constructs they are used to support. By exposing the field’s implicit measurement assumptions, the Lexis enables their inspection, testing, falsification, and refinement into evidence-based relationships. On expert-annotated samples, the extraction pipeline achieves high agreement on extracted concepts while revealing substantial ambiguity in the boundaries of constructs and behaviors. By making these relationships explicit, AI Construct Lexis provides infrastructure for a more rigorous and cumulative science of AI evaluation.

Vision-language models (VLMs) are increasingly proposed as general-purpose tools for scientific data interpretation, yet their reliability on real astronomical observations across diverse modalities remains untested. We present AstroVLBench, a benchmark of over 4,100 expert-verified instances across five tasks spanning optical imaging, radio interferometry, multi-wavelength photometry, time-domain light curves, and optical spectroscopy. We systematically evaluate six state-of-the-art frontier models and find that performance is strongly modality-dependent and uncorrelated with general benchmark rankings. Mechanistic ablations reveal three concrete bottlenecks. First, prompts that ground attention in physical mechanisms (why a feature matters) yield more balanced classifications than phenomenological prompts (what to look for). Second, presenting one-dimensional measurements as numerical tables rather than rendered plots improves accuracy by up to 13 percentage points, indicating that plot rendering can obscure physically relevant signal. Third, a qualitative reasoning analysis reveals a pervasive “right-answer-wrong-reason” phenomenon: models routinely reach correct predictions through physically invalid logic, showing that accuracy alone is insufficient for trustworthy scientific deployment. AstroVLBench provides the first systematic, multi-modal baselines for VLMs in observational astronomy and identifies the representation, grounding, and reasoning bottlenecks that must be addressed before VLMs can be trusted as autonomous scientific interpreters.

Kernel methods, and Gaussian Processes (GPs) in particular, require a conditionally negative-definite (CND) distance measure to guarantee positive semi-definiteness (PSD) of the kernel matrix — a condition that fails for many natural input spaces, including smooth manifolds and spaces of probability distributions. We propose the Sparse Landmark Embedding (SLE) kernel, which eliminates this requirement entirely. Each input is embedded into a sparse feature vector via compactly supported bump functions centered at all training points $|\mathcal{D}|$; applying any standard PSD kernel in this embedding space yields a kernel that is provably PSD for arbitrary distance measures. The compact support automatically controls embedding sparsity, keeping kernel matrices well-conditioned and computationally tractable despite the high ambient dimension. We provide theoretical guarantees on PSD, sparsity, stability, and universal approximation, and demonstrate, using geodesic and Wasserstein distances, that the SLE kernel matches or substantially exceeds domain-specific baselines in both predictive accuracy and uncertainty quantification.

Small-molecule foundation models are typically pretrained on standalone molecular data, unlike vision and language models that often benefit from cross-modal or relational supervision. Protein-ligand co-folding provides a molecular analogue of such supervision by exposing models to atom-level ligand-protein interactions, raising the question of whether co-folding models can yield strong small-molecule representations. We study this question using Boltz2, a modern co-folding model, by transferring its atom-level ligand representations to standalone small-molecule tasks. Through systematic probing and distillation, we show that Boltz2 representations match or outperform existing models on the ADMET benchmark, accelerate molecular generative modeling, and improve sample efficiency in structure-guided ligand optimization. We further find that Boltz2 representations are complementary to those learned from conventional standalone molecular supervision, including 3D conformers, bioassay labels, and quantum-chemical properties. Finally, we extend representation alignment to reinforcement learning, showing that dense representation-level supervision can complement scalar rewards in molecular discovery. These results identify protein-ligand co-folding as a promising pretraining paradigm for small-molecule representation learning and position Boltz2 as a strong, off-the-shelf molecular foundation model.

Variational quantum models are linear maps of the input state: any useful non-linearity must be engineered through the data encoding, entanglement pattern, ancilla tracing, classical readouts, or explicit classical layers. Although many such mechanisms have been proposed, there is no standard protocol that scores them on a common footing. Most published comparisons reduce non-linearity to either the accuracy on a single downstream benchmark or a single abstract expressibility number, and the two are rarely measured under matched training budgets, circuit widths, and noise assumptions. We introduce a reproducible evaluation protocol that decomposes non-linearity quality into four complementary scalar scores covering frequency reach, gradient-variance scaling, trainability efficiency on a chaotic dynamical task, and stability under depolarizing noise. In addition, we define a composite index with published weights, enabling methods to be ranked under a single criterion while preserving full access to the underlying components. We instantiate the protocol with six variational ansatzes spanning the standard taxonomy of quantum non-linearity sources (data reuploading, ancilla tracing, their combination, strongly entangling layers, a quantum convolutional network, and a quantum reservoir), together with a classical multilayer perceptron and a hybrid quantum-classical network as non-quantum reference points. The evaluation surfaces a central mismatch that single-number reporting hides: in our reference run, the composite leader (data reuploading) is near the bottom on held-out Lorenz one-step $R^2$, while the Lorenz leader (ancilla reuploading) is only third on the composite. The main outcome is a practical benchmark that makes these trade-offs visible and testable: with fixed tasks, budgets, and seeds, new methods can be added by registering a circuit builder and rerunning the same protocol, and multi-seed aggregates can be updated without changing metric definitions.