NeurIPS 2026 论文列表

Reinforcement learning (RL) is now standard for post-training large language models. The same reward optimization that elicits useful capabilities, however, can also reinforce latent backdoors planted earlier in the pipeline: attack success rates under 0.2% after SFT rise to 40--98% on training-distribution inputs after RL, and generalize to 13--30\% on held-out evaluation, with no modification to the RL data, reward function, or training loop. We study this phenomenon in the context of agent models, where SFT poisoning teaches the victim model (Qwen3-8B and a small frontier model) to call an attacker-controlled oracle. Because the oracle can be set up to always return correct answers, calling it earns higher reward than the model's own attempts, and RL reinforces the behavior. We further show that the oracle's RL-time responses can instill persistent biases, such as brand preferences, that survive into deployment even when no tool calls happen. Such patterns appear difficult to detect with current tools: traditional guard models do not flag this mechanism, and LLM-based auditors remain unreliable, achieving only 4.5\% precision even after iterative prompt refinement. Our findings point to the value of post-RL safety evaluation, in particular scrutinizing tool-call patterns such as unnecessary external invocations.

AI progress is increasingly tracked through evaluations, such as AI benchmarks, yet small perturbations in evaluation pipelines can produce unstable scores and model rankings. We recast benchmark reliability as a measurement problem and formalize it as a signal-to-noise ratio derived from a crossed random-effects decomposition. This framework yields a direct mapping between variance components and expected rank stability, linking theoretical reliability to observable leaderboard concordance. Modern AI benchmarks operate in a high-dimensional regime with many items and relatively few evaluated models, where classical item-level reliability measures are ill-posed. We define a lower bound for benchmark item-level reliability classically, and we further introduce a tractable proxy, $\lambda^\bigstar_6$, that preserves the variance structure underlying reliability without modifying the estimand through sparsity or dimensionality reduction. Across diverse benchmarks, we show that selecting items via $\lambda^\bigstar_6$ produces smaller subsets that achieve higher ranking reliability than random subsampling at fixed evaluation budgets. Finally, we analyze reliability under positive distribution shift--as often observed in the current AI ecosystem--by partitioning models into lower- and higher-performing cohorts. We show that reliability is population-dependent and frequently declines as between-model variance shifts among stronger systems. Our results establish a principled framework for diagnosing and improving benchmark reliability and demonstrate that ranking stability is neither intrinsic nor static, but a measurable and optimizable property of the benchmark–population pairing.

Biosynthetic gene clusters (BGCs) are co-located genes that encode the biosynthesis of secondary metabolites, a major source of clinically used antibiotics and anticancer agents. Recent advances in deep learning, particularly self-supervised foundation models, have spurred growing interest in BGC sequence modeling, but evaluation infrastructure has lagged behind. Current practice often relies on coarse classification accuracy over small, experimentally curated datasets, making it difficult to discriminate model capabilities or assess downstream utility. We introduce BGC-Bench, an evaluation suite designed to resolve this limitation by systematically incorporating biochemical knowledge in the benchmark design. In terms of generalization, we test whether representations transfer to genetically and chemically novel samples; for functionality, we examine application-relevant tasks, including retrieval, prediction, and active learning. BGC-Bench reveals that no single model class dominates: pretrained foundation models perform strongly under out-of-domain generalization but remain sensitive to training conditions; compositional baselines are competitive and lead in cross-modal retrieval; and active-learning gains vary substantially across tasks, strategies, and model families. Beyond these BGC-specific findings, BGC-Bench highlights a broader principle: domain-informed benchmark design over existing datasets enables better evaluation in data-scarce scientific domains.

LLM benchmarks are often treated as coherent measurement units, yet they are heterogeneous collections of instances spanning diverse domains, skills, formats, and contexts. As a result, aggregate benchmark scores can conflate model capability with benchmark composition, obscuring what existing benchmarks actually cover. We introduce BenchHub, a composition-aware evaluation framework that represents benchmarks as distributions over instance-level attributes. BenchHub integrates 54 benchmarks comprising 839K samples across 10 languages. It enables researchers to inspect benchmark contents, uncover reusable coverage hidden behind benchmark names, transparently compare benchmarks, and construct controllable evaluation sets for application-aligned model selection. Using BenchHub, we show that similarly motivated benchmarks can differ substantially in internal composition, existing benchmarks often contain reusable coverage beyond their stated purposes, models exhibit fine-grained category-level performance variation hidden by aggregate scores, and model rankings can shift under different reweighting and resampling configurations. Our results motivate evaluation practices that make benchmark composition explicit, inspectable, and controllable.

Structured generation remains a critical bottleneck for Multimodal Large Language Models (MLLMs). Current post-training paradigms struggle to resolve this: Supervised Fine-Tuning (SFT) hits a rigid ceiling, while scalar-reward Reinforcement Learning (e.g., DPO, GRPO, SimPO) suffers from a policy-level signal gap. Compressing long routing trajectories into sparse signals induces a short-completion bias and traps RL near the SFT baseline, whereas vanilla self-distillation actively regresses. To systematically quantify this joint visual-structural-relational reasoning, we introduce OracleGraph, a 10,000-page 2.5D historical document benchmark featuring a 12-dimensional diagnostic VQA suite and a structured-hallucination taxonomy. To overcome these optimization pathologies, we propose PRISM, a Policy-Reward Integrated Self-distillation framework. PRISM leverages multimodal information asymmetry via a Hindsight Teacher guided by a Diagnostic Graph Report (DGR). By synergizing Softmax Reward-Weighted Aggregation (preserving Graph-F1 rankings) with a Length-Normalized Auxiliary Preference Loss, PRISM explicitly neutralizes catastrophic truncation, yielding a $+3.81$pp Graph Reward gain and a $+2.3$pp schema-parsing improvement over SDPO ($p < 0.001$). While PRISM's nominal $+1.02$pp mean gain over SFT is not statistically significant ($p = 0.12$), it drastically stabilizes optimization, achieving $2.4\times$ lower cross-seed variance. Crucially, on a cross-book out-of-distribution (OOD) probe, PRISM's stability advantage amplifies to a $14\times$ variance reduction over SFT, confirming robust structural stabilization. Code and datasets are available at https://anonymous.4open.science/r/PRISM2026.

Many safety and alignment failures of large language models (LLMs) occur due to out-of-distribution (OOD) situations: unusual prompt or response patterns that are unforeseen by model developers. We systematically study whether LLM monitoring pipelines can detect these **OOD alignment failures** by introducing a benchmark called Misalignment Out of Distribution Benchmark (**MOOD**). It is difficult to find failures that are truly OOD for off-the-shelf models trained on vast safety datasets. We sidestep this by including a restricted training set in MOOD that we use to train our own monitors, as well as seven test sets with diverse alignment failures that are outside the training distribution. Using MOOD, we find that guard models (safety classifiers) often fail to generalize OOD. To fix this, we propose combining guard models with OOD detectors. We test four types of OOD detectors and find that a combination of a guard model and Mahalanobis distance-based OOD detector can improve recall from 39% to 47%. We also establish positive scaling trends across model scales for monitors that combine a guard model and OOD detector; we find that incorporating OOD detection into monitoring achieves a higher recall gain than using a guard model with 20 times more parameters. Our work suggests that OOD detection should be a crucial component of LLM monitoring and provides a foundation for further work on this important problem.

The rapid advancement of large language models (LLMs) has expanded their potential in the legal domain. However, existing legal benchmarks remain largely English-centric and oriented toward common law, leaving a critical gap in evaluating LLMs for civil law systems that govern most jurisdictions worldwide. To address this gap, we introduce Vietnamese Legal Benchmark VietLegal, a cognitively grounded benchmark designed for the hierarchical and codified structure of Vietnamese law. Although instantiated in Vietnamese legislation, VietLegal provides a replicable evaluation framework for civil law systems characterized by complex statutory hierarchies and frequent amendments. Inspired by Bloom’s taxonomy, VietLegal assesses multiple levels of legal understanding through tasks that mirror real-world legal assistant use cases, including legal question answering, multi-step reasoning, and scenario-based problem solving. The benchmark contains 10,450 expert-annotated samples, each cross-validated against authoritative legal sources to ensure fidelity to practical legal workflows. By offering the first standardized legal benchmark for Vietnamese, VietLegal enables systematic assessment of LLMs in civil law contexts and supports the development of more reliable and interpretable AI-assisted legal systems.

Cyber-physical system (CPS) forecasting models depend on sensor streams with noisy, biased, missing, or temporally misaligned readings, yet standard forecasting evaluation often selects models by nominal error without showing whether they remain robust under such faults. We introduce SensorFault-Bench, a shared CPS-grounded sensor-fault stress-test protocol for evaluating forecasting architectures and robustness-improvement methods, together with an operational taxonomy that organizes the method comparison. Across four real-world datasets and eight scored scenarios governed by a standardized severity model, it reports worst-scenario degradation, clean mean squared error (MSE), and worst-scenario fault-time MSE, separating relative robustness from absolute error. A disjoint fault-transfer split lets explicit fault-training methods train on adjacent fault families while reported scores use separate benchmark scenarios. Empirically, forecasting architectures favored by clean MSE can degrade sharply under faults, and clean-MSE rankings can disagree with worst-scenario fault-time error rankings. Chronos-2, the evaluated zero-shot foundation-model representative, matches or trails the last-value naive forecaster in clean MSE on the two single-target datasets and has the largest worst-scenario degradation on ETTh1 and Traffic, where all channels are forecast targets. For the evaluated robustness-improvement method set, paired deltas show selective degradation reductions: projected gradient descent adversarial training and randomized training lead where value faults dominate observed degradation, while fault augmentation leads where availability faults dominate. SensorFault-Bench provides open-source code, documented data access, and reproduction and extension guides, so new datasets, architectures, and robustness-improvement methods can be evaluated under the same CPS sensor-fault robustness protocol.

As AI systems are deployed in open-ended, high-stakes domains, a critical behavioral dimension remains unmeasured: whether models differ systematically in how they translate perceived risk into action. We introduce a cross-domain evaluation framework that isolates and quantifies \emph{risk attitude} in large language models (LLMs) by decoupling contextual belief ($B_C$, a model's perceived danger level given situational context) from categorical risk decision ($R_D$, the action chosen in response to that belief). Unlike existing capability benchmarks, which assess factual accuracy but do not probe the belief-to-decision mapping, this framework directly targets the disposition governing whether an agent acts cautiously or aggressively under identical levels of perceived danger. We applied the framework to six frontier LLMs and 100 human participants across spatial navigation, clinical triage, and financial allocation tasks, and identified two qualitatively distinct behavioral profiles. Five models satisfied both reliability criteria (Contextual Belief Consistency and Risk Decision Consistency) across all conditions and tasks. These models showed strong measurement reliability with stable belief formation and belief-to-decision mappings, and exhibited stable, trait-level cross-domain risk attitudes that were preserved across all three tasks (Kendall's $W=1.00$, $p=0.017$). One model, Grok~4, failed both criteria: its belief formation was unstable in Drone Navigation Control task and its decision mapping was inconsistent in Clinical Triage Decision task, yielding task-specific rather than trait-level risk behavior. Across all six LLMs, risk profiles converged toward a restricted region of the broader human risk-attitude distribution, suggesting that alignment training constrains models toward a narrow behavioral consensus. These results demonstrate that the framework reliably measures risk attitude as a stable, model-specific behavioral dimension and can serve as a diagnostic tool for pre-deployment assessment by distinguishing models with coherent risk personalities from those with domain-contingent risk postures.

The recent development of Large Language Models (LLMs) has been accompanied by an effervescence of novel ideas and methods to better optimize the loss of deep learning models. Claims from those methods are myriad: from faster convergence to removing reliance on certain hyperparameters. However, the diverse experimental protocols used to validate these claims make direct comparisons between methods challenging. This study presents a comprehensive evaluation of recent optimization techniques across standardized LLM pretraining scenarios, systematically varying model size, batch size, and training duration. Through careful tuning of each method, we provide guidance to practitioners on which optimizer is best suited for each scenario. For researchers, our work highlights promising directions for future optimization research. Finally, by releasing our code and making all experiments fully reproducible, we hope our efforts can help the development and rigorous benchmarking of future methods.

As language models are increasingly deployed as autonomous agents, they will need to coordinate with others in long-horizon, open-ended interactive tasks. Yet current evaluations rarely test these demands together, focusing instead on short interactions, single-agent open-ended tasks, or highly structured multi-agent settings. We introduce $alem$, a JAX-based, procedurally generated open-ended benchmark for multi-agent coordination built on Craftax-like dynamics. Alem embeds procedurally generated coordination tasks, soft specialisation, communication, and controllable coordination difficulty into a long-horizon survival world with exploration, crafting, trading, and combat. We evaluate $13$ modern LLMs zero-shot within homogeneous teams, with trained MARL agents as reference points. Most LLM agents struggle, averaging only ~6% of maximum reward, although performance varies widely. Gemini-3.1-Pro-High approaches MARL agents trained for one billion steps on Hard coordination ($17.5\%$ vs. $17.6\%$ Coord.\%), Gemma-4-31B-it is the strongest tested open-weight model, and GPT-5.4-High makes strong base progress, while achieving much lower coordination reward. We find that base task competence does not imply coordination competence, communication helps agents share intent, memory and reasoning help with multi-step planning, and heterogeneous teams regress toward average member performance rather than matching their strongest teammate. These results identify coordination as a distinct bottleneck for current LLM agents, separate from single-agent capabilities. Alem makes this bottleneck measurable, providing a controlled setting for developing agents that can communicate, allocate roles, and execute shared plans.

Recent advancements in Large Vision-Language Models (LVLMs) have significantly enhanced their ability to integrate visual and linguistic information, achieving near-human proficiency in tasks like object recognition, captioning, and visual question answering. However, current benchmarks typically focus on knowledge-centric evaluations that assess domain-specific expertise, often neglecting the core ability to reason about fundamental mathematical elements and visual concepts. We identify a gap in evaluating elementary-level math problems, which rely on explicit visual dependencies-requiring models to discern, integrate, and reason across multiple images while incorporating commonsense knowledge, all o which are crucial for advancing toward broader AGI capabilities. To address this gap, we introduce VCBench, a comprehensive benchmark for multimodal mathematical reasoning with explicit visual dependencies. VCBench includes 1,720 problems across six cognitive domains, featuring 6,697 images (averaging 3.9 per question) to ensure multi-image reasoning. We evaluate 26 state-of-the-art LVLMs on VCBench, revealing substantial performance disparities, with even the top models unable to exceed 50% accuracy. Our findings highlight the ongoing challenges in visual-mathematical integration and suggest avenues for future LVLM advancements.

Modern language models are commonly adapted after pretraining to follow instructions, align with user preferences, and improve deployment behavior. Such post-training often relies on preference data from users, annotators, or model interactions, which may contain sensitive prompts, private responses, proprietary tasks, or confidential judgments. Understanding the privacy implications of this data is therefore essential. Membership inference attacks (MIAs), which test whether a record was used for training, are the de facto standard for empirical privacy auditing in machine learning. However, preference-based post-training changes the unit of membership: a training record contains a prompt, a preferred response, and a dispreferred response, rather than a single input-output sequence. Audits that ignore this structure can under-report leakage. We therefore introduce a benchmark protocol that adapts strong reference-model MIAs to complete preference records. Our protocol asks whether membership is exposed by either response alone, by the model’s preference between them, or by the two responses jointly. We find that auditing the response pair can reveal membership evidence missed when the record is reduced to a single score. Across three preference datasets, two model families, and seven post-training objectives spanning three post-training families, measured leakage depends strongly on both the audited record statistic and the training objective, with imbalanced risk between preferred and dispreferred responses. We further evaluate the impact of parameter-efficient fine-tuning (PEFT) and differential privacy (DP), finding widely varying privacy-utility trade-offs. Overall, preference-based post-training leaks in ways that standard single-response audits can miss, motivating formal privacy methods that protect complete preference records while preserving post-training utility.

Self-supervised learning (SSL) has received notable attention in the graph machine learning community, enabling the efficient use of unlabeled data. However, evaluation setups vary significantly across studies, and baseline tuning often receives limited attention, making it difficult to draw reliable conclusions about model performance. In this paper, we reevaluate four representative graph SSL methods within a unified setup across a diverse set of datasets. We focus on rigorously tuning both SSL and supervised methods by employing enhanced GNN architectures and comprehensive hyperparameter optimization. We observe that, contrary to prior literature, two popular generative methods, MaskGAE and GraphMAE, regularly fail to outperform well-tuned supervised baselines. In contrast, the contrastive methods BGRL and GRACE consistently perform better than both the generative methods and supervised baselines. We hypothesize that this discrepancy arises because BGRL and GRACE capture information from both graph structure and node features, whereas MaskGAE and GraphMAE focus on a single source of information. We support this hypothesis through an analysis on carefully designed synthetic data. Motivated by our observations, we advocate for designing SSL objectives that capture both feature and structural information. To verify the effectiveness of this approach, we propose a simple generative method, GrASP, which reconstructs both graph structure and node features. Despite its simplicity, GrASP outperforms all other evaluated approaches and can be considered as a strong baseline in future studies.

Large Vision-Language Models (LVLMs) have shown remarkable potential for embodied AI, yet a critical bottleneck persists: the lack of fine-grained spatio-temporal alignment between high-level textual reasoning and low-level visual grounding. Existing benchmarks fail to recognize the critical importance of such fine-grained alignment for embodied tasks. To address this gap, we introduce ECLBench, a comprehensive benchmark designed to systematically assess spatio-temporal embodied understanding across two primary capabilities: Embodied Cognition (Object Cognition and Spatial Cognition) and Embodied Localization (Grounding and Pointing). Together, these four pillars encompass 21 specialized sub-capabilities, supported by 3,616 egocentric video clips (577,998 frames) and 12,000 meticulously curated open-ended questions. Extensive evaluations on ECLBench reveal that general LVLMs excel in semantic understanding but struggle with precise spatio-temporal grounding, while existing embodied agents often sacrifice generalization for specialization. Guided by the benchmark's findings, we develop ECLBrain, a baseline model that discretizes continuous spatial coordinates into integer tokens. This simple yet effective design bridges textual semantics and spatio-temporal vision, demonstrating a promising path toward unified embodied reasoning. ECLBench and ECLBrain together establish a rigorous, multi-dimensional standard for diagnosing current limitations and guiding future research toward spatially-aware and temporally-consistent embodied intelligence.

Machine Learning Interatomic Potentials play a fundamental role in computational chemistry and materials science, enabling applications from molecular dynamics simulations to drug design and materials discovery. While recent approaches can estimate inter-atomic forces with high precision, it remains unclear to what extent they can generalise to previously unseen molecules. Do they learn the compositional structure of chemistry, capturing how molecular fragments and their combinations determine properties, or do they primarily learn to interpolate patterns that are specific to the training examples? To address this question, we propose a benchmark consisting of four tasks that require some form of compositional generalisation. In each task, models are tested on molecules that were unseen during training, but the training data is chosen such that generalisation to the test examples should be feasible for models that learn the underlying physical principles. Our empirical analysis shows that the considered tasks are highly challenging for state-of-the-art models, with errors on out-of-distribution examples often an order of magnitude higher than on in-distribution examples, even when using foundation models that have been pre-trained on millions of molecules.

Here we introduce the Olfactory Perception (OP) benchmark, designed to assess the capability of large language models (LLMs) to reason about smell. The benchmark contains 1,010 questions across eight task categories spanning odor classification, odor primary descriptor identification, intensity and pleasantness judgments, multi-descriptor prediction, mixture similarity, olfactory receptor activation, and smell identification from real-world odor sources. Each question is presented in two prompt formats, compound names and isomeric SMILES, to evaluate the effect of molecular representations. Evaluating 21 model configurations across major model families, we find that compound-name prompts consistently outperform isomeric SMILES, with gains ranging from +3.1 to +18.9 percentage points (mean $\approx$ +7 points), suggesting current LLMs access olfactory knowledge primarily through lexical associations rather than structural molecular reasoning. The best-performing model reaches 64.4\% overall accuracy, which highlights both emerging capabilities and substantial remaining gaps in olfactory reasoning. We further evaluate a subset of the OP across 21 languages and find that aggregating predictions across languages improves olfactory prediction, with AUROC=0.86 for the best performing language ensemble model. LLMs should be able to handle olfactory and not just visual or auditory information.

Large language models can achieve strong benchmark scores without proportional gains in broader capability, but diagnosing when this occurs remains difficult. We study this gap through benchmark shadows: support-concentrated data regimes that may concentrate learning around narrow evaluation-relevant patterns while limiting broader representational development. Under fixed architecture, tokenizer, optimizer family, and training budget, we compare a coverage-expanding baseline with two support-concentrated regimes: redundant repetition and frequency-concentrated rewriting. Spectral, rank-based, and layer-wise update diagnostics reveal distinct parameter-space signatures: repetition-concentrated training is largely recoverable after later diverse training, whereas frequency-concentrated support collapse leaves more persistent footprints. Correlational analyses of open-source multimodal model families reveal analogous structural patterns that co-occur with asymmetric benchmark profiles, while a prompt de-duplication case study shows that surface redundancy alone does not induce the same regime-level effects. Benchmark scores alone are therefore insufficient to characterize capability; parameter-space diagnostics provide complementary signals about training quality, coverage, and generalization-related regime effects.

Benchmark recovery is often treated as a deployment certificate for low-bit LLMs, but thresholded controller stacks ship a different object: a frozen score-to-action contract. This paper asks a narrower acceptance question: after quantization, does that interface survive independent threshold retuning against frozen FP16 labels, the strongest cheap repair an operator would plausibly try first? We study one fixed four-action controller, one frozen train/cal/report protocol, and four instruction-tuned checkpoints: Llama-3.1-8B/70B-Instruct and Qwen-2.5-7B/72B-Instruct. In the only score-carrying regime, W4A16, GPTQ + -search stays near FP16 on MMLU yet still leaves 11.9/10.9 percentage-point false tool-trigger rates and 16.1/15.4 aggregate FCR on the two large models. Because BFCL directly labels the tool frontier, that failure already establishes the paper's main acceptance claim even if FAQ and the preservation-only retrieve/stop frontiers are ignored. Aggregate FCR is reported only as interface-wide preservation context because retrieve/stop remain preservation-only. FAQ is included only as a bounded reducibility probe under the same frozen audit target, not as a generic new quantizer or the central novelty claim. FAQ lowers tool FTR and aggregate FCR without obvious MMLU or throughput collapse, but that is secondary non-collapse evidence. The contribution is therefore a single-controller deployment-acceptance counterexample under a frozen permission set: on the retained large-model W4A16 rows, benchmark-close MMLU recovery does not certify preserved tool-trigger behavior. The accompanying contribution is the freeze-and- retune audit protocol that makes that failure measurable.