NeurIPS 2026 Papers

Outcome homogenization---models producing similar outputs---has drawn increasing attention in the large language model (LLM) community as researchers and practitioners have come to recognize that alignment practices can reduce model diversity and reproduce dominant narratives. However, most approaches address outcome homogenization exclusively through a technical lens. In this position paper, we argue that outcome homogenization is best understood as a value-chain phenomenon shaped by stakeholders, governance structures, resource allocation, model development, and organizational practices. We identify key drivers of homogenization in the AI ecosystem, showing how current practices reinforce dominant paradigms while marginalizing alternative perspectives. We argue that addressing homogenization requires shifting attention from model-level decisions to the broader power dynamics and organizational practices. Through a case study of LLM vendors’ safety alignment practices, we illustrate how these ecosystem-level forces shape model development and deployment. We conclude with a call to action, offering targeted recommendations for the research community.

Large language model (LLM) agents are increasingly used to operate browsers, files, code and tools, making personal assistants a natural deployment target. Yet personal agents face a privacy-cost-capability tension: cloud models execute multi-step workflows well but expose sensitive intermediate context to external APIs, while local models preserve privacy but remain less reliable. Both settings also pay repeatedly for long skill prompts and growing histories. We propose constant-context skill learning, a context-to-weights framework for recurring agent workflows: reusable procedures are learned in lightweight task-family modules, while inference conditions only on the current observation and a compact state block. A deterministic tracker renders this state block from task progress and supplies aligned subgoal rewards, so each module can be trained with step-level SFT and refined through online RL. Across ALFWorld, WebShop, and SciWorld, our agents achieve strong performance across Qwen3-4B, Qwen3-8B and Llama-3.1-8B. With Qwen3-8B, SFT+RL reaches 89.6\% unseen success on ALFWorld, 76.8\% success on WebShop, and 66.4\% unseen success on SciWorld. They match or exceed strong published agent-training results while reducing prompt tokens per turn by 2-7$\times$ relative to controlled ReAct prompting baselines, showing that procedural context can be moved from prompts into weights.

Agents based on large language models are increasingly expected to autonomously handle negotiation and transactions. However, existing benchmarks predominantly focus on text-only, bilateral, zero-sum price haggling, failing to capture the complexity of real-world commerce. To address this gap, we introduce Agenticpay, a unified framework and benchmark for evaluating how well multimodal agents reach high-welfare, multi-dimensional agreements in realistic markets. Built around four core components ( Environments, Tasks, Agents, and Metrics), Agenticpay comprises 160 multimodal tasks spanning 4 real-world business scenarios (E-commerce, Food Delivery, Ride-hailing, and Apartment Rental) and 8 market topologies, scaling from 1-to-1 bargaining to many-to-many (N-to-N) competitive markets. Beyond price haggling, agents must read product images, infer each opponent's hidden preferences, and trade off multiple binding contract terms (e.g., price, lease duration, return policy, delivery speed). Agents communicate through multi-round natural-language dialogue, with each turn proposing or revising a full contract, and outcomes are scored by a utility-based framework that rewards agreements maximizing joint welfare. Evaluations on state-of-the-art proprietary (GPT-5.4, Claude Sonnet 4.6, Gemini 3.1 Pro Preview) and open-weight (Qwen3-VL-32B-Instruct, InternVL3-38B) multimodal models reveal substantial gaps in non-zero-sum value creation and market reasoning: even the strongest agent (Gemini 3.1 Pro Preview) reaches a GlobalScore of only 42.3/100, and performance consistently degrades as markets scale from bilateral to multi-sided, with the average GlobalScore dropping by 5.7 points and open-weight models suffering the largest declines (up to 11.5 points). These findings establish Agenticpay as a foundational testbed for multimodal agentic commerce. Code and dataset are available at: https://anonymous.4open.science/r/AgenticPay-4BBB/README.md.

LLM benchmarks are usually consumed as scalar leaderboards, where adjacent ranks are read as adjacent capability differences. We argue that this display should be complemented rather than simply trusted or discarded: benchmark scores are useful summaries, but a sorted score table can overstate which model pairs are actually distinguishable. This limitation is especially clear for accuracy- and success-rate leaderboards, whose averages hide task-level structure, and it also appears in pairwise systems such as Elo-style ratings when pairwise evidence is compressed into a total order. We propose LLM tier ranking as a task-level reporting framework for benchmarks. Given benchmark outputs, we construct pairwise wins, losses, and ties between models on each task and aggregate them into ordered tiers using Selective Preference Aggregation (SPA). Cross-tier comparisons are reported only when supported by sufficient task-level agreement; models in the same tier are left unresolved, not declared equal. Across AIME 2025, $\tau$-bench airline, xbench DeepSearch, GDPVal, SWE-Bench Verified, and Chatbot Arena, tier ranking often preserves useful broad capability classes while abstaining from unsupported fine-grained distinctions among frontier models. In Chatbot Arena perturbation studies, tier outputs are more stable than Elo-style total orders, expressing weaker evidence through later tier separation rather than rank reshuffling. We recommend reporting tiered leaderboards, solution paths, top-tier size, singleton-winner thresholds, and pairwise resolution statistics alongside scalar scores.

While multimodal large language models (MLLMs) have shown strong visual reasoning abilities, serving a large model for every query is computationally expensive. MLLM cascades mitigate this cost by first querying a weak but cheaper model and deferring to a strong model when the weak model's output is unconfident. However, since the weak model's confidence directly controls compute allocation, these systems expose a new attack surface: an adversary can manipulate confidence so that their queries are consistently deferred to the strong model. Motivated by this vulnerability, we introduce the Forced Deferral Attack (FDA), an adversarial image attack that lowers the weak model's confidence and causes cascades to route queries to the strong model. FDA learns a universal border trigger by optimizing a temperature-flattened objective. This objective pushes the weak model's token distribution on triggered inputs toward less concentrated targets constructed from its clean responses. Across datasets, model families, and deferral metrics, FDA consistently increases strong-model routing while outperforming image-perturbation and prompt-injection baselines. These results show that MLLM cascades are vulnerable to attacks that manipulate compute allocation, forcing unintended strong-model usage without directly targeting answer correctness.

Reliable hypothesis testing is the foundation of many empirical scientific claims. Large language model (LLM) agents are increasingly used to automate this process, as they can inspect datasets, generate code, and produce analyses end-to-end. However, we show that they frequently make subtle inferential errors that lead to incorrect conclusions despite correctly executed analyses. Existing benchmarks fail to capture this failure mode, as they rarely assess whether a reported p-value is statistically valid given the assumptions underlying the data. We address this gap by building P-Bench, a benchmark comprising 425 open-ended, realistic hypothesis-testing tasks spanning economics, biology, and medicine. Each task requires an agent to select a statistical method, compute a p-value, and draw a conclusion given only a scientific hypothesis and a dataset. We further introduce Fisher-R1, an open-weight LLM agent trained for rigorous hypothesis testing using synthetic tasks and reinforcement learning. On P-Bench, Fisher-R1-14B substantially improves over its backbone and outperforms strong proprietary and open-source baselines, including GPT-5.4 and DeepSeek-V4-Pro, achieving a 21% average relative improvement in single-trial success over DeepSeek-V4-Pro, with gains up to 26% on the most challenging tasks. Our results demonstrate that current LLM agents lack reliable statistical reasoning for hypothesis testing and that reinforcement learning on tasks with verified statistical reward substantially improves reliability.

Diffusion large language models promise parallel token generation, yet inference remains bottlenecked by deciding which masked tokens can be safely committed together. Fast-dLLM addressed this with KV caching and confidence-guided parallel decoding, but its decoding theory uses a homogeneous high-confidence assumption that effectively reduces each candidate set to its weakest selected token. We argue that this leaves speed on the table because real decoding steps exhibit heterogeneous confidence profiles. We propose **Fast-dLLM++**, a training-free extension that introduces *Fr\'{e}chet profile decoding*: selecting parallel commit sets from the full sorted confidence profile rather than a single worst-case confidence. The resulting rule is a heterogeneous-confidence generalization of Fast-dLLM's factor selector and it recovers the previous rule exactly in the equal-confidence case and adds a provable *heterogeneity bonus* when the selected tokens have uneven confidences. Fast-dLLM++ leaves the model, diffusion process, and cache implementation entirely unchanged, making it a drop-in replacement for existing Fast-dLLM decoding. Experiments on GSM8K, MATH, HumanEval, and MBPP with the LLaDA-8B model show that the theoretical improvement translates directly into empirical gains: profile-aware selection improves the accuracy-throughput frontier by exploiting safe parallelism that weakest-token rules miss, achieving up to 37\% higher throughput at comparable accuracy. Our anonymous code release is at https://anonymous.4open.science/r/fast-dllm-plus-plus/.

Enabling language model agents to autonomously adapt to new environments remains a fundamental challenge. While recent memory-augmented approaches have achieved promising results, they are largely heuristic in design and treat task execution and knowledge construction as decoupled processes. Inspired by how biological systems rapidly adapt through principled acting and learning, we propose \textbf{FEP-Agent}, a framework that grounds LLM agent self-evolution in the \textit{Free Energy Principle (FEP)}. Our system instantiates the two components of Active Inference into: a Worker agent that minimizes \textit{Expected Free Energy (EFE)} during online interaction, balancing goal-directed exploitation with curiosity-driven exploration to probe uncertain environmental dynamics; and a Builder agent that minimizes \textit{Variational Free Energy (VFE)} post-execution, updating a knowledge base while controlling complexity through counterfactual validation. To support scalable knowledge in open-ended environments, we introduce a structured semantic memory that employs associative linking and overlapping communities for efficient retrieval. The excellent results across multiple LLM-agent benchmarks demonstrate that FEP-Agent achieves principled, self-evolving systems.

With LLM watermarking already being deployed commercially, practical applications increasingly require *multibit* watermarks that encode more complex payloads, such as user IDs or timestamps, into the generated text. In this work, we propose a fundamentally new approach for multibit watermarking: introducing binomial encoding to directly encode every bit of the payload at every token position. We complement our approach with a *stateful encoder* that during generation dynamically redirects encoding pressure toward underencoded bits. Our evaluation against 8 baselines on up to 64-bit payloads shows that our scheme achieves superior message accuracy and robustness, with the gap to baseline methods widening in more relevant settings (i.e., large payloads and low-distortion regimes). At the same time, we challenge prior works’ evaluation metrics, highlighting their lack of practical insights, and introduce *per-bit confidence scoring* as a practically relevant metric for evaluating multibit LLM watermarks.

Physical reasoning is central to agents that step into the physical world. Models now excel at code and math, but their physical reasoning capabilities remain largely untested. Existing physical benchmarks evaluate goal-state matching rather than functional performance. This position paper argues that evaluating physical reasoning in LLM agents requires construction benchmarks, where physics verifies functional outcomes. To operationalize physical reasoning evaluation, we identify five capabilities spanning the full agentic loop from goal to verified artifact. Our audit of eleven physical benchmarks finds only two that combine realistic physics with functional verification, the minimum requirements for testing physical reasoning; both are construction benchmarks. Neither covers all five capabilities, yet frontier models already achieve near-zero success as task complexity increases. Progress depends on the breadth of benchmarks we build. We call on academia, game studios, and industry labs to expand construction benchmarks along both axes, widening capability coverage and deepening physical challenges. Their breadth determines how fully agents move from code and math to reshaping the physical world.

Asynchronous reinforcement learning (RL) is an effective paradigm for improving the training efficiency of large language model (LLM) post-training by decoupling rollout generation from policy optimization. However, this decoupling introduces stale off-policy samples generated by earlier behavior policies, leading to distribution shift and unstable policy updates. Recent replay selection methods such as D-ARL mitigate this issue by selecting variance-aware samples, but require expensive current-policy evaluation over replay responses and mainly rely on variance reduction or current-policy matching as the selection criterion. To address these limitations, we propose GA-ARL, an efficient Gradient-Aware Asynchronous Reinforcement Learning framework for LLM post-training. GA-ARL derives an analytic target distribution from the KL-regularized RL objective and introduces an advantage-weighted variant to account for both policy preference and gradient strength. During training, GA-ARL maintains a replay buffer containing samples from recent behavior policies and selects low-discrepancy samples according to the advantage-weighted analytic target, avoiding additional current-policy evaluation during replay scoring. GA-ARL outperforms SOTA asynchronous methods across mathematical, logical, and code reasoning benchmarks, achieving the best average accuracy on both Qwen3-1.7B and Qwen3-4B, with improvements of up to 6.1%. Meanwhile, compared with the SOTA D-ARL, GA-ARL reduces wall-clock training time by 25.3% on average by avoiding additional current-policy evaluation during replay selection.

Exhaustively evaluating many large language models (LLMs) on a large suite of benchmarks is expensive. We cast benchmarking as finite-population inference and, under a fixed query budget, seek tight confidence intervals (CIs) for model accuracy with valid frequentist coverage. We propose *Factorized Active Querying* (FAQ), which (a) leverages historical information through a Bayesian factor model; (b) adaptively selects questions using a hybrid variance-reduction/active-learning sampling policy; and (c) maintains validity through *Pro-Active Inference*---a finite-population extension of active inference (Zrnic & Candès, 2024) that enables direct question selection while preserving coverage. With negligible overhead cost, FAQ delivers up to $5\times$ effective sample size gains over strong baselines on two benchmark suites, across varying historical-data missingness levels: this means that it matches the CI width of uniform sampling while using up to $5\times$ fewer queries. We release our source code and our curated datasets to support reproducible evaluation and future research.

Zeroth-order (ZO) optimization enables memory-efficient large language model (LLM) fine-tuning, with recent methods employing curvature-aware strategies. However, in the massive LLM parameter space, these methods inherently suffer from exploding gradient variance and prohibitive costs. Conversely, confining updates to a low-dimensional subspace reduces variance but discards critical high-dimensional information, degrading performance. This creates a fundamental dilemma: full-space optimization is informative but high-variance, while subspace optimization is low-variance but blind. To break this trade-off, we propose \textbf{\Engine} (\textbf{En}do\textbf{g}enous Var\textbf{i}atio\textbf{n}al MultiScal\textbf{e}). Inspired by computational mechanics, we adapt static Variational MultiScale theory to dynamic optimization, treating low- and high-dimensional spaces as resolved coarse and unresolved fine scales. Through a multigrid-style V-cycle, \Engine\ alternates between spaces, dynamically probing fine-grained information in the high-dimensional space and injecting it to correct the efficient low-dimensional sampling process. Evaluations on standard LLM benchmarks show that \Engine\ achieves higher accuracy and faster convergence than existing ZO methods, achieving up to a $\mathbf{2.61\times}$ \textbf{speed-up in forward passes} with zero additional peak memory overhead. Code is available at \url{https://anonymous.4open.science/r/ENGINE-D0A4}.

Large language models (LLMs) are often ensembled together to improve overall reliability and robustness, but in practice models are strongly correlated. This raises a fundamental question: which models should be selected when forming an LLM ensemble? We formulate budgeted ensemble selection as maximizing the mutual information between the true label and predictions of the selected models. Furthermore, to explain why performance can saturate even with many models, we model the correlated errors of the models using a Gaussian-copula and show an information-theoretic error floor for the performance of the ensemble. Motivated by these, we propose a simple greedy mutual-information selection algorithm that estimates the required information terms directly from data and iteratively builds an ensemble under a query budget. We test our approach on four datasets spanning binary and multi-class classification and sentiment analysis: MEDMCQA, MMLU, AG News, and IMDB. Across all datasets, we observe that our method consistently outperforms strong baselines under the same query budget.

LLM agents show strong potential in software engineering tasks such as competitive programming and code synthesis. Existing benchmarks, however, largely focus on small-scale settings such as self-contained Python modules, leaving distributed systems, an essential class of production software, under-examined. We introduce DistDebug-Bench, a benchmark for evaluating agents’ ability to diagnose complex-system failures at the code level. It curates 120 high-quality real-world bugs from 11 widely used distributed systems, spanning 5 programming languages and 10 failure patterns. Through an extensive evaluation of state-of-the-art LLMs and agents, we identify substantial gaps between current agents and real-world debugging tasks. A recurring gap is tunnel vision due to a lack of explicit reasoning structure and nondeterminism from inconsistent tool-use scope.

Comparative claims about large language model (LLM) performance are typically based on aggregate benchmark scores. This treats models with similar scores as comparable on the items the benchmark contains, an assumption that can fail if models from different populations succeed on systematically different items. We test this assumption using Differential Item Functioning (DIF), a psychometric diagnostic that flags items whose relative difficulty differs across groups after conditioning on overall performance. Applying Mantel--Haenszel DIF analyses to six widely-used LLM benchmarks across two widely-used open-source model families (Llama and Qwen), we find that many items behave differently across families even among models with the same total benchmark score, with some items favoring Llama-family models and others favoring Qwen-family models. The flagged items concentrate in benchmark-provided categorical content (subjects, subtasks, problem types), and choosing items based on which family they favor can substantially amplify, attenuate, or reverse aggregate family-level comparisons. Our results show that aggregate benchmark scores can summarize different item-level performance patterns across model populations, and that DIF offers a useful diagnostic for surfacing this variation.

As Large Language Models (LLMs) deploy into mission-critical domains (e.g., finance, medicine, and law), output reproducibility has become a strict system requirement. While practitioners use greedy decoding to eliminate algorithmic stochasticity, empirical deployments with 16-bit precisions still exhibit catastrophic output divergence across heterogeneous GPUs. Through SASS-level profiling, we reveal that this inconsistency is fundamentally driven by truncation errors introduced during downcasting at kernel boundaries. However, achieving reproducibility via a global FP32 pipeline incurs prohibitive system penalties: bypassing 16-bit hardware accelerators hurts compute efficiency, while upcasting the KV cache doubles memory overhead. To bridge this gap, we propose Hybrid Error ALleviation (HEAL), a targeted intervention that approximates FP32 precision while resolving hardware constraints through two targeted mechanisms. First, recognizing that floating-point formats underutilize their bit-width for Q, K, V tensors, HEAL applies INT16 quantization that preserves numerical stability without expanding the KV cache footprint. Second, HEAL synthesizes high-precision matrix multiplications via an algebraic error compensation strategy, executing entirely on high-throughput 16-bit Tensor Cores. To evaluate our approach practically, we introduce MCR-Bench, a benchmark targeting reproducibility in mission-critical tasks. HEAL achieves the same level of reproducibility on downstream tasks as the FP32 baseline while reducing the performance overhead by up to 6.7$\times$.

We propose WALLA, a decentralized mechanism for aggregating probabilistic predictions from multiple LLMs using wagering mechanisms. Each model reports a prediction and a learned wager reflecting its expected score advantage; predictions are then aggregated using wagers as weights. Our mechanism introduces a leave-one-out baseline that yields three key properties: dominant-strategy incentive compatibility under general belief structures, a best-response wager proportional to expected score advantage, and decoupling of prediction and wager optimization. We instantiate two mechanism variants trading off normality and no-arbitrage, both with bounded worst-case deficit independent of the number of participants. Experiments on QA benchmarks and a forecasting benchmark show that WALLA matches centralized baselines while simultaneously achieving advantage-weighted aggregation, uncertainty-awareness, fully decentralized learning, and incentive-compatibility guarantees.

Fluent guidance is not the same as useful intervention. LLM tutors are typically trained to generate the next teacher utterance, implicitly assuming that every student turn warrants a response. However, our experiments indicate that this conflates tutoring capability (what to say) with intervention necessity (whether to say it). We introduce DICE, a framework that decouples intervention decisions from response generation by first selecting an explicit pedagogical action. To calibrate this action selection policy, we define Intervention Value (IV), a rollout-grounded counterfactual metric that compares each action against non-intervention. IV shows that many prescribed interventions provide little or no marginal benefit. We further introduce DICE-Bench, a multi-variant math tutoring benchmark with skill-preserving problem variants for session-level evaluation. Using IV-weighted and KL-regularized policy optimization, DICE learns to intervene selectively while preserving tutoring effectiveness. In simulated tutoring sessions, DICE reduces the over-intervention rate to near zero while guiding students to correct solutions in approximately 3-4 fewer turns on average than existing Socratic tutoring baselines.