NeurIPS 2026 Papers

Audits of human-AI complementarity target the wrong estimand. They ask whether the human knows more than the model; deployment asks whether that knowledge would change the action. Residual predictive expertise is not residual decision value. Deployment rewards actions, and a posterior movement that does not cross a reward-induced action boundary changes nothing; a signal can be informative under log-loss and leave the deployed action unchanged. We formalize the missing quantity as **boundary regret**: the regret of holding the model's action after observing the human. In any finite-action Bayesian decision problem, the value of consulting the human equals expected boundary regret, formally verified in Lean 4. The identity turns when humans help into a geometric question: which decision facets does the human signal move belief across? Complementarity is therefore reward-relative. A given human signal can be valuable under one reward matrix and worthless under another, the human's knowledge unchanged; the boundary moved, not the expertise. The gap is widest precisely where asymmetric costs dominate: high-stakes clinical and operational decisions. Audits built on residual predictive expertise overstate the value of human review and route scarce review toward cases where the human is informative but the action will not move. Boundary regret is the quantity to measure.

Scaling reinforcement learning (RL) to diverse multitask settings remains a central challenge. While recent advances in model-based RL achieve strong performance, they rely on planning and complex training pipelines, making it unclear which components are essential for scalability. We revisit this question and argue that the primary driver of scalable multitask RL is not model-based control, but _representation learning_. In particular, we show that combining predictive, model-based representations with high-capacity value function approximation is sufficient to achieve strong performance, without planning. We evaluate a simple model-free algorithm, MR.Q, that integrates auxiliary predictive objectives into a scalable actor-critic architecture. This approach outperforms a recent world-model-based method and a range of deep RL baselines across a diverse suite of multitask continuous control tasks, while significantly reducing computational overhead and improving wall-clock efficiency. We observe consistent improvements with increased model capacity and show through ablations that predictive representation learning is critical for performance.

Deep reinforcement learning (RL) for quantum circuit optimization faces three fundamental bottlenecks: replay buffers that ignore the reliability of temporal-difference (TD) targets, curriculum-based architecture search that triggers a full quantum-classical evaluation at every environment step, and the routine discard of noiseless trajectories when retraining under hardware noise. We address all three by treating the replay buffer as a primary algorithmic lever. We introduce ReaPER$+$, an annealed replay rule transitioning from TD error-driven prioritization to reliability-aware sampling as value estimates mature, achieving up to 4$\times$ improved sample efficiency over the strongest off-policy baselines (fixed PER, ReaPER, and uniform replay) and matching prior on-policy solution quality at up to 32$\times$ fewer interactions. At 12-qubit scale, ReaPER achieves the lowest energy error in the fewest steps which is consistent with ReaPER$+$'s annealing direction, while PER and Vanilla replay find more compact circuits at the cost of higher energy error. Validation on LunarLander-v3 confirms the annealing principle is domain-agnostic obtaining +9% AUC over PER and fixed ReaPER. We further introduce OptCRLQAS, which amortizes expensive quantum-classical evaluations over multiple architectural edits, cutting wall-clock time per episode by up to 67.5% on a 12-qubit task without degrading solution quality. Finally, a lightweight replay-buffer transfer scheme warm-starts noisy-setting learning from noiseless trajectories, without network-weight transfer or $\epsilon$-greedy pretraining, reducing steps to chemical accuracy by up to 85-90% and final energy error by up to 90% over from-scratch baselines, with transfer advantages that grow with system size. Together, these results establish that experience storage, sampling, and transfer are decisive levers for scalable, noise-robust quantum circuit optimization. The anonymized repository is available at : https://anonymous.4open.science/r/annealed-replay-buffer-transfer-for-quantum-optimization-6861/.

Reinforcement Learning has significantly advanced the reasoning capabilities of Multimodal Large Language Models (MLLMs), yet the resulting policies remain brittle against real-world visual degradations such as blur, compression artifacts, and low-resolution scans. Prior robustness techniques from vision and deep RL rely on static data augmentation or value-based regularization, neither of which transfers cleanly to critic-free RL fine-tuning of autoregressive MLLMs. Reinforcing reasoning against such corruptions is non-trivial: naively injecting degraded views during rollout induces reward poisoning, where perceptual occlusions trigger hallucinated trajectories and destabilize optimization. We propose ROMA, an RL fine-tuning framework that modifies the optimization dynamics to reinforce reasoning against visual degradation while preserving clean-input performance. A dual-forward-pass strategy uses teacher forcing to evaluate corrupted views against clean-image trajectories, avoiding new rollouts on degraded inputs. For distributional consistency, we apply a token-level surrogate KL penalty against the worst-case augmentation; to prevent policy collapse under regularization, an auxiliary policy gradient loss anchored to clean-image advantages preserves a reliable reward signal; and to avoid systematically incorrect invariance, correctness-conditioned regularization restricts enforcement to successful trajectories. On Qwen3-VL 4B/8B across seven multimodal reasoning benchmarks, our method improves robustness by +2.4% on seen and +2.3% on unseen corruptions over GRPO while matching clean accuracy.

The success of deep learning in medical image analysis is often hindered by the long-tail distribution of disease cases, where rare or early-stage tumors are critically under-represented. While controllable synthesis offers a potential solution, existing methods suffer from a \textit{specification bottleneck}, as low-dimensional signals like text or binary masks fail to capture complex textures and morphological variations. In this paper, we propose an exemplar-driven synthesis framework that utilizes real tumor samples as high-dimensional templates to enable precise, case-specific generation. To the best of our knowledge, this is the first work to introduce reinforcement learning (RL) into the field of 3D medical image generation, called **MedGRPO**. We formulate the unpaired tumor synthesis task as a constrained trajectory optimization problem and leverage Flow-GRPO to find an optimal generation path. By designing a multi-objective reward system encompassing texture, shape, and semantic consistency, our model effectively balances pathological fidelity with anatomical coherence without requiring paired supervision. Rigorous evaluations across four major tumor types and the external AbdomenAtlas 2.0 dataset demonstrate that our method produces high-fidelity synthetic data that significantly enhances the performance of downstream diagnostic models.

Partial differential equations (PDEs) are foundational to modeling in science and engineering, but constructing reliable numerical solvers remains labor-intensive, demanding expert knowledge of discretization schemes, stability conditions, and boundary treatments. Recent work has begun to frame PDE solving as a code-generation task for large language models (LLMs), yet existing approaches operate primarily at inference time: relying on prompting, debugging, self-refinement, and test-time scaling rather than adapting the model itself. In parallel, reinforcement learning with verifiable rewards has emerged as a powerful post-training paradigm for code and math reasoning, but its verifiers are typically binary: a compiler runs, or a test passes. Such signals discard the graded structure of scientific correctness, where two solvers may both execute and yet differ in solution accuracy by orders of magnitude. In this work, we introduce RLVP: Reinforcement Learning with Verifiable Physics, an RL post-training framework for multi-PDE solver code generation.RLVP addresses this verifiability gap with a hybrid verifier: hard program-validity checks ensure executability, while continuous physics rewards score function-space accuracy and PDE-residual consistency. A single policy is post-trained across diverse PDE families spanning hyperbolic, parabolic, elliptic, and incompressible-flow systems. RLVP improves over both pre-trained and supervised-only baselines on PDE benchmarks, and shows zero-shot improvement transfer to held-out PDEs. We show that a smaller LLM post-trained with RLVP can outperform prompting a frontier model on in-distribution PDE solver generation. The trained policy shows evidence of compositionality in numerical motifs: it recombines stencils, time-stepping schemes, and boundary-handling primitives learned from the PDEs used in training into generated solvers for unseen PDE problems.

We study tabular reinforcement learning problems with multiple steps of lookahead information. Before acting, the learner observes $\ell$ steps of future transition and reward realizations: the exact state the agent would reach and the rewards it would collect under any possible course of action. While it has been shown that such information can drastically boost the value, finding the optimal policy is NP-hard, and it is common to apply one of two tractable heuristics: processing the lookahead in chunks of predefined sizes ('fixed batching policies'), and model predictive control. We first illustrate the problems with these two approaches and propose utilizing the lookahead in adaptive (state-dependent) batches; we refer to such policies as adaptive batching policies (ABPs). We derive the optimal Bellman equations for these strategies and design an optimistic regret-minimizing algorithm that enables learning the optimal ABP when interacting with unknown environments. Our regret bounds are order-optimal up to a potential factor of the lookahead horizon $\ell$, which can usually be considered a small constant.

3D Gaussian Splatting enables real-time novel view synthesis with high fidelity, yet its sparse and unorganized Gaussian anchors make compression challenging without harming structure and cross-view consistency. A key challenge is that effective compression depends not only on reducing parameter redundancy, but also on identifying where the compressed model is most fragile and how limited representation capacity should be allocated accordingly. We present an RL-guided distillation framework for 3DGS compression (RLGS). RLGS first uses a lightweight reinforcement learning policy to predict informative viewpoint offsets for adaptive distillation. It then distills a compact student 3DGS from a high-quality teacher under multi-view rendering supervision. Finally, it introduces an Unbalanced Optimal Transport based structural selection module to align teacher-student anchor distributions in voxelized space, thereby retaining the most informative anchors during pruning and allocation. Under the same bitrate constraints, our method consistently outperforms strong 3DGS compression baselines in rendering quality. At matched rendering quality, it further reduces model size while preserving structural fidelity, achieving up to about 15% additional size reduction at comparable rendering quality over prior methods.

Reinforcement learning (RL) for exponential-utility optimization in discounted Markov decision processes (MDPs) lacks principled value-based algorithms. We address this gap in the fixed risk-aversion setting. Building on the Bellman-type equation for exponential utility studied in Porteus [1975], we derive two Q-value-style extensions and show that the associated operators are contractions in the $L_\infty$ and sup-log/Thompson metrics, respectively. We characterize their fixed points and prove that the induced greedy stationary policy is optimal for the exponential-utility objective among stationary policies. These structural results lead to two model-free algorithms: a two-timescale Q-learning--style algorithm, for which we establish almost-sure convergence and provide finite-time convergence rates via timescale separation, and a one-timescale algorithm governed by a sublinear power-law operator. Since the latter does not admit a global contraction in standard metrics, we prove its convergence using delicate arguments based on local Lipschitzness, monotonicity, homogeneity, and Dini derivatives, and provide a scalar finite-time analysis that highlights the challenges in obtaining convergence rates in the vector case. Our work provides a foundation for value-based RL under exponential-utility objectives.

Reinforcement learning (RL) has been effective for post-training autoregressive (AR) language models, but extending these methods to diffusion language models (DLMs) is challenging due to intractable sequence-level likelihoods. Existing approaches therefore rely on surrogate likelihoods or heuristic approximations, which can introduce bias and obscure the sequential structure of denoising. We formulate diffusion-based sequence generation as a finite-horizon Markov decision process over the denoising trajectory and derive the policy gradient that decomposes over denoising steps in terms of stepwise advantages, without requiring explicit evaluation of the sequence likelihood. Grounded in this theorem, we develop tractable approximations for large-scale training: (i) denoising steps are selected for policy updates via an entropy-guided approximation bound, and (ii) stepwise advantages are estimated using a one-step denoising completion naturally provided by the diffusion model, avoiding costly multi-step rollouts or auxiliary value networks. Experiments demonstrate state-of-the-art results on nearly all benchmarks spanning coding, logical reasoning, and mathematical reasoning, outperforming existing RL post-training approaches for DLMs.

RL for code correctness is now established: have the model generate a program, run it against hidden test cases, and reward solutions that pass. Extending this to code optimization seems straightforward: just add execution time to the reward. But in practice, once timing drives the reward, small problems in measurement noise, reward sparsity, or GRPO instability overwhelm the signal and make RL fail: generated solutions are barely faster, and more of them can fail. We make execution time learnable through three stages: (1) how code is tested, by building DMC-Optim with large optimization tests and a calibrated sandbox; (2) how speed is turned into reward, by composing correctness and speed in the RL environment and using an offline simulator to predict the most promising configurations; and (3) how the model learns from that reward, by adapting GRPO and evaluation to the sparser, noisier timed-execution setting. On DMC-Optim, the strongest optimization-aware configurations improve strict top-decile pass@1 from 18.1% to 27.7% on Qwen 2.5 7B and from 28.7% to 42.4% on CWM 32B, while preserving most of the pure-correctness score. When the timing sandbox is degraded, robust optimization RL reaches up to roughly 130% improvement over standard RLVR. On LCB, CWM 32B wins up to 70.3% of best-sample speed comparisons against standard RLVR; relative to the fastest correct human submissions per problem, it reaches about half the human rate of complexity-class improvements (14% vs. 28%).

To date, the field of reinforcement learning (RL) has primarily focused on developing discounted methods, which are designed to optimize the short-term or transient performance of RL agents. Conversely, the field has been reluctant to explore and invest in methods that are designed to optimize the long-term or steady-state performance of RL agents. In this position paper, we challenge this reluctance under the framing that RL agents should be viewed as competitive swimmers. That is, we argue that, like competitive swimmers, RL agents cannot hope to win all races, or solve all tasks, based on transient performance alone. To this end, we first motivate the need for RL agents that can adequately optimize both the transient and steady-state performance. Then, we summarize recent works which show that discounted methods have deficiencies when it comes to optimizing the steady-state performance. Finally, we argue in favor of more attention and investment from the community towards average-reward methods, which are designed to optimize the steady-state performance. Namely, we challenge common misconceptions associated with average-reward methods, and highlight recent advancements which suggest that such methods present a viable and potentially superior alternative.

Existing LLM-based policy optimizers see only scalar rewards: that a policy scored 0.45, but not whether the agent got stuck in a loop, fell into a hole on the third step, or performed well on 19 out of 20 rollouts and failed catastrophically on one. We propose Reflective Prompted Policy Optimization (R2PO), a two-stage LLM framework for policy search over compact policy classes that augments scalar reward feedback with trajectory-level behavioral evidence. A Search-LLM acts as a global policy optimizer and proposes candidate policy parameters; the environment executes them; a Critic-LLM then inspects the resulting rollouts and proposes targeted parameter revisions grounded in observed states, actions, and rewards. Across ten environments, ablations show R2PO's gains arise from a design that explicitly separates global search from behavior-grounded revision and uses selection to filter high-variance edits. We further identify a dominant failure mode, salience bias. When presented with multiple rollouts, the Critic-LLM fixates on improving a single failure even when most trajectories succeed. In a three-trajectory variant, where the Critic-LLM is shown the best, worst, and median rollout from each evaluation, this behavior explains 76.6\% of regressions on CartPole. R2PO mitigates this by reasoning over aggregate rollout statistics, median-trajectory selection, and a revision rule. Using a relatively small open-weight 20B-parameter model, R2PO achieves the highest mean best reward across all ten environments, while reaching near-optimal performance substantially earlier in training (e.g., near-maximum CartPole reward within $\sim$500 episodes), and training far more stably than both deep RL and prior LLM-based methods. Together, these results show that treating trajectories as first-class in-context evidence, rather than background artifacts reduced to scalar returns, fundamentally changes how even comparatively small LLMs can search over policy spaces, enabling them to learn faster, diagnose more precisely, and reliably improve external controllers rather than tune them by trial and error.

Recent years have witnessed growing interest in applying Large Reasoning Models (LRMs) to Machine Translation (MT). While most approaches adopt a "pre-thinking" paradigm and benefit from explicit reasoning trajectories, they suffer from substantial inference cost and latency. To address these limitations, we propose ReflectMT, a two-stage reflection internalization framework for machine translation that employs a "post-thinking" paradigm. Our approach develops the model's "translate–reflect–refine" capability through reinforcement learning. In the first stage, we cultivate the model's capacity for high-quality reflection and refinement, thereby enhancing its semantic comprehension and task-specific knowledge. In the second stage, we train the model to internalize the knowledge acquired during reflection. As a result, during inference, ReflectMT operates in a direct translation mode, producing high-quality translations on the first attempt without any explicit reasoning steps. Experimental results on benchmarks such as WMT24 demonstrate that our model’s first-pass translations during inference outperform multi-step reasoning LRMs (e.g., DeepSeek-R1) in both automatic metrics and GPT-based evaluation, achieving a 2.16-point improvement in GPT-based translation quality evaluation while reducing token consumption by 94.33%.

We introduce ReflectDrive-2, a masked discrete diffusion planner with a separate action expert for autonomous driving that represents plans as discrete trajectory tokens and generates them through parallel masked decoding. This discrete token space enables in-place trajectory revision: AutoEdit rewrites selected tokens using the same model, without requiring an auxiliary refinement network. To train this capability, we use a two-stage procedure. First, we construct structure-aware perturbations of expert trajectories along longitudinal progress and lateral heading directions and supervise the model to recover the original expert trajectory. We then fine-tune the full decision--draft--reflect rollout with reinforcement learning (RL), assigning terminal driving reward to the final post-edit trajectory and propagating policy-gradient credit through full-rollout transitions. Full-rollout RL proves crucial for coupling drafting and editing: under supervised training alone, inference-time AutoEdit improves PDMS by at most $0.3$, whereas RL increases its gain to $1.9$. We also co-design an efficient reflective decoding stack for the decision--draft--reflect pipeline, combining shared-prefix KV reuse, Alternating Step Decode, and fused on-device unmasking. On NAVSIM, ReflectDrive-2 achieves $91.0$ PDMS with camera-only input and $94.8$ PDMS in a best-of-6 oracle setting, while running at $31.8$ ms average latency on NVIDIA Thor.

In the field of Neural Combinatorial Optimization (NCO), Reinforcement Learning (RL) stands out for its ability to naturally enforce complex constraints via its auto-regressive decision pattern. To mitigate the longstanding challenges, e.g., reward sparsity and sample inefficiency in the vast combinatorial action space which causes ineffective training, subpar performance, and poor scalability, we propose CORectifier, a novel NCO solver learned under the hierarchical gated rectification mechanism to regularize the arbitrary exploration: partial policy-predicted actions in a trajectory are probabilistically replaced with high-quality segments from reference solutions. CORectifier prompts the model with tri-level optimal signals, operating at the batch, instance, and sub-instance levels with fragments of diverse lengths injected at random decision steps. This Rectified RL (RRL) paradigm helps develop optimality-aware and sample-efficient RL learners while maintaining their sequential-decision manner for constraint satisfaction, delivering a new perspective to hybridize RL and SL/IL for NCO with improved utility rate of limited supervision. Sufficiently extensive experiments on Traveling Salesman Problem (TSP), Asymmetric TSP (ATSP), Prize-Collecting TSP (PCTSP), Capacitated Vehicle Routing Problem (CVRP), Knapsack Problem (KP), Job-Shop Scheduling Problem (JSSP), and Single Machine Total Weighted Tardiness Problem (SMTWTP), across synthetic and real-world benchmarks, show superior quality than RL baselines by up to 59.7%.

Ensuring high returns while providing strong safety guarantees in offline imitation learning (IL) is a fundamental challenge: existing approaches either rely on explicitly specified cost functions, which are difficult to design for complex real-world safety constraints, or require expert demonstrations that are both safe and high-performing, which are often too costly to collect. We introduce *dual-expert imitation learning*, which leverages two complementary data sources: (i) safety-compliant demonstrations that faithfully follow safety guidelines but may be suboptimal, and (ii) performance-oriented demonstrations that achieve high returns but may violate safety guidelines. To address this problem, we propose DexDICE (**D**ual-**ex**pert offline imitation learning via stationary **DI**stribution **C**orrection **E**stimation). Instead of conventional distribution constraints, DexDICE combines DICE-style offline learning with safe support constraints, enabling agents to exploit high-return behaviors while remaining within safety-compliant regions. We evaluate DexDICE on a re-curated DSRL benchmark and validate it on a real-world mobile-robot navigation task trained directly from human teleoperation demonstrations, where it significantly outperforms offline IL baselines and remains robust to data scarcity and quality degradation.

Deep decarbonization is transforming power grids into safety-critical, low-carbon cyber-physical systems, where AI controllers must jointly manage economic efficiency, carbon reduction, and operational security. This operational energy trilemma is challenging for reinforcement learning: cost and carbon objectives are naturally optimized in expectation, whereas security violations are rare, heavy-tailed events whose consequences can be catastrophic and therefore require explicit tail-risk control. Existing safe reinforcement learning methods either constrain safety only in expectation or rely on light-tailed approximations, which can underestimate rare but severe grid violations. We propose a heterogeneous risk-constrained reinforcement learning framework for low-carbon grid operation. The key idea is to assign different risk semantics to different objectives: safety-critical security constraints are enforced through Conditional Value-at-Risk (CVaR), while economic and carbon-related objectives are optimized in expectation to preserve operational flexibility. To better capture extreme scenarios, a mixture distribution model is introduced to characterize heavy-tailed constraint violations. We further develop a residual policy learning scheme built on a distributionally robust chance-constrained reference module: the robust module provides a feasible and economically efficient baseline policy, and the learned residual refines it toward lower-carbon operation while respecting tail-risk bounds. Experiments on power-system operation tasks show that the proposed framework reduces economic cost by over 25% compared with expectation-based CMDP baselines and improves safety by two orders of magnitude over Gaussian-tail methods. These results suggest that risk-aware reinforcement learning can support reliable decarbonization by reconciling efficiency, sustainability, and security within a unified decision-making framework.

Frontier reasoning models are produced by post-training base language models with reinforcement learning. Recent work has challenged this by showing that sampling from a sharpened version of the base model's distribution, a so-called *power distribution,* elicits comparable reasoning without additional training, curated datasets, or verifiers. However, making this method practical requires *efficiently sampling* from the power distribution. A sampler needs to "mix" to the power distribution which necessitates moving between modes of the target distribution; intuitively, e.g., trying different reasoning strategies. The samplers proposed in prior works repeatedly select a "cut" position in the current reasoning trace uniformly at random and resamples the suffix from that position onwards. However, reasoning traces typically contain a few consequential decisions (e.g., the choice of proof strategy or algorithm), and we observe that a uniformly chosen cut tends to rewrites local details rather than revisiting decision points. We introduce an algorithm (Entropy-Cut Metropolis--Hastings) which uses the base model's next-token entropy as a proxy to identify key decision points and re-sample from these positions. We empirically verify that entropy jumps are a useful proxy for decision points, and in a stylized model of reasoning prove that our method's mixing time scales with the number of decisions in a trace rather than with the number of tokens, which can be much larger. Across MATH500, HumanEval, GPQA, and AIME on several models, our method consistently improves over baselines and RL-trained models.