NeurIPS 2026 Papers

Modern off-policy reinforcement learning algorithms often rely on simple uniform replay sampling and it remains unclear when and why non-uniform replay improves over this strong baseline. Across diverse RL settings, we show that the effectiveness of non-uniform replay is governed by three factors: replay volume, the number of replayed transitions per environment step; expected recency, how recent sampled transitions are; and the entropy of the replay sampling distribution. Our main contribution is clarifying when non-uniform replay is beneficial and providing practical guidance for replay design in modern off-policy RL. Namely, we find that non-uniform replay is most beneficial when replay volume is low, and that high-entropy sampling is important even at comparable expected recency. Motivated by these findings, we adopt a simple Truncated Geometric replay that biases sampling toward recent experience while preserving high entropy and incurring negligible computational overhead. Across large-scale parallel simulation, single-task, and multi-task settings, including three modern algorithms evaluated on five RL benchmark suites, this replay sampling strategy improves sample efficiency in low-volume regimes while remaining competitive when replay volume is high.

Large language models reliably follow complex instructions in a single turn, yet across long multi-turn interactions they start strong then gradually lose the thread of the instructions, persona, and rules they were given. This degradation has been measured behaviorally but not mechanistically explained. We trace this failure to a transition between two information channels: accessibility to goal-defining tokens through attention, and a residual channel that carries goal information forward through hidden states. We introduce the Goal Accessibility Ratio (GAR), measuring attention from generated tokens to task-defining goal tokens, and combine it with sliding-window ablations and residual-stream probes. When attention to instructions closes, what survives reveals architecture. Across the architectures we test, this transition produces qualitatively different failure modes: some models preserve substantial goal-conditioned behavior at vanishing attention, others fail despite carrying decodable goal information in their residual stream, and the depth at which this encoding emerges varies dramatically by architecture (from layer 2 to layer 27). A within-model causal ablation that closes the attention channel by force on Mistral collapses recall from near-perfect to eleven percent on a 20-fact retention task and raises persona-constraint violations to levels exceeding the adversarial-pressure baseline despite no user pressure, with both effects emerging at the predictable crossover turn. Linear probes on residual representations recover per-episode recall outcomes with AUC up to 0.99 across all four primary architectures (input embedding: chance), evidencing the second channel and showing its depth profile is architecture-specific. Across multiple model architectures and model scales, we show that the attention channel and the residual channel are separable, and that the gap between attention loss and residual capacity determines whether goal-conditioned behavior survives a long conversation. We provide GAR as a metric, the channel transition framework as a mechanism, and a parametric prediction of when multi-turn instruction-following will fail.

Multi-task scene understanding models typically require modifications to the vision encoder or complex decoders for cross-task interaction, which limits their flexibility and compatibility with off-the-shelf vision foundation models (VFMs). This paper introduces Virtual Task Prompting (VTP), which frames multi-task prompting as retrieving task-adaptive subsets from a shared virtual state, where both the state and accessing process itself are optimizable in an end-to-end manner. Concretely, VTP maintains a set of virtual tasks alongside the model, and at each layer, learnable routing matrices read an actual task composition as input for interaction with image tokens, then write the processed results back, forming explicit learning pathways that capture the intricacies among tasks before and after the computation of each encoder block. Since VTP keeps the backbone clean, it integrates seamlessly with recent VFMs like DINOv3, advancing the state of the art on Pascal-Context and NYUD-v2. Stress test on Taskonomy with 13 heterogeneous tasks further shows VTP's robustness in extreme settings. Ablations and analysis confirm that VTP's gains stem mainly from its prompting design rather than merely increased capacity.

Cross-view Referring Multi-Object Tracking (CRMOT) aims to track multiple objects specified by natural language across multiple camera views, with globally consistent identities. Despite recent progress, existing methods rely heavily on costly frame-level spatial annotations and cross-view identity supervision. To reduce such reliance, we explore CRMOT under weak supervision by leveraging the capabilities of foundation models. Our empirical study shows that directly applying foundation models such as SAM2 and SAM3, even with task-specific modifications, fails to accurately understand referring expressions and maintain consistent identities across views. Yet, such models remain effective at producing reliable object tracklets that can serve as supervision signal. We therefore repurpose foundation models as pseudo-label generators and propose a two-stage framework for weakly supervised CRMOT, using only object category labels as coarse-grained supervision. In the first stage, we design an Affinity-guided Cross-view Re-prompting strategy to refine and associate SAM3-generated tracklets across cameras, producing reliable cross-view pseudo labels for subsequent training. In the second stage, we introduce ViewSAM, a CRMOT model built upon SAM2 that explicitly models view-aware cross-modal semantics. By formulating view-induced variations as learnable conditions, ViewSAM bridges the gap between view-variant visual observations and view-invariant textual expressions, enabling robust cross-view referring tracking with only approximately 10% additional parameters. Extensive experiments demonstrate that ViewSAM achieves SOTA performance under weak supervision and remains competitive with fully supervised methods.

Multi-agent language-model systems typically rely on a human designer or external orchestrator to fix the agent population, roles, and workflow—a single point of failure that places coordination decisions outside the team executing them. Existing approaches address fragments of this problem: communication standards lack coordination logic, automated workflow-search systems concentrate authority in a single planner, and decentralized peers lack explicit governance, handoff validation, and bounded repair. To close this gap, we introduce the Unified Resource-Grounded Coordination Protocol (URCP), an orchestrator-free protocol in which a fixed pool of black-box models and executable resources collectively decides task framing, role formation, workflow topology, assignment, checkpoint validation, and final completion through supermajority voting under a single decision rule Φq. A five-level repair hierarchy reopens only the affected part of the workflow on failure; under standard timing and communication assumptions, every run terminates with an explicit verdict. Empirically, with twelve heterogeneous models against eight baselines on MultiAgentBench, τ-bench, MemoryAgentBench, GAIA, and REALM-Bench, URCP exceeds the strongest baseline by +9.7 / +7.9 / +10.6 points on the three primary benchmarks, terminates deadlock-free in 100% of runs, and remains stable up to 20% agent failure. Trace analyses show that the gains come from voted role concentration on frontier models (73.9% of critical-path roles), frequent critic/verifier insertion (63.5% of accepted schemas with at least three roles), and bounded repair rather than from model strength alone.

Genomic events span multiple spatial scales, from single-nucleotide substitutions to large structural variants, yet existing foundation models process only reference DNA sequences, leaving methylation, short variants, and structural-variant boundaries as downstream labels or auxiliary inputs. We propose the Unified Genomic Model (UGM), an encoder-only genomic foundation model that learns multi-scale genomic events as native tokens within a unified 43-token vocabulary. UGM introduces three algorithmic components: (i) a unified event tokenizer that maps reference bases, SNPs, CpG methylation, short INDELs, and structural-variant boundaries into a single event-state space; (ii) event-balanced pretraining, which promotes adequate gradient coverage for rare but biologically important events; and (iii) Site-Level Joint Prediction (SJP), a masked-language-modeling objective that recovers the complete genomic state at a masked position as a single token rather than predicting base, methylation, and variant labels independently. We evaluate UGM against specialized and generalist baselines across functional variant classification, regulatory benchmarks (NT and GUE), structural-variant detection, and cross-event attention analysis. UGM achieves competitive performance, particularly on tasks where explicit event-state information is central. It obtains the highest score on all seven functional variant tasks, reaches competitive splicing prediction on NT, and notably outperforms DNA-only baselines on SV breakpoint detection. These results suggest that native multi-scale event pretraining is a promising direction for genomic representation learning.

Recent long-tailed recognition methods increasingly adopt two-stage fine-tuning, where a generic representation is first learned and imbalance-aware adaptation is then performed in a lightweight second stage. However, existing second-stage designs often allocate objective emphasis using delayed empirical summaries, such as historical class-wise performance or validation feedback, and commonly restrict adaptation to classifier heads. These choices may make routing unreliable when empirical feedback is noisy or skewed, and may also limit the feature plasticity required for rare categories. In this paper, we propose U-MOF, an uncertainty-guided parameter-efficient multi-objective fine-tuning framework for long-tailed recognition. U-MOF introduces lightweight residual adapters into a shared-backbone expert committee, uses entropy-based committee statistics as diagnostic routing proxies to modulate tail-oriented and smoothing-oriented objectives, and applies conflict-aware orthogonal projection to coordinate heterogeneous objective gradients. Experiments on CIFAR-100-LT, ImageNet-LT, and iNaturalist 2018 show that U-MOF improves rare-category and overall performance in controlled stage-two comparisons and remains competitive with strong published long-tailed recognition baselines, while preserving the efficiency advantages of decoupled adaptation. These results indicate that reliable stage-two long-tailed fine-tuning benefits not only from selecting appropriate objectives, but also from diagnosing when each objective should dominate and from retaining limited feature-level plasticity. An anonymized implementation is included in the Supplementary Material.

Calibrated predictions are useful because their numerical values can be interpreted as probabilities. Calibration errors are therefore widely used to evaluate and compare probabilistic predictors. Recently, Haghtalab et al. [2024] introduced truthfulness as an additional requirement for such measures. A calibration measure is truthful if a predictor minimizes its expected measured error by reporting the true conditional label distribution. Many standard empirical calibration errors are non-truthful: a predictor may appear better calibrated by distorting its probabilities. We study the implications of truthful and non-truthful calibration errors for practice. First, we introduce perfectly truthful calibration errors for full multiclass calibration and classwise calibration, two standard multiclass notions. More generally, our construction applies to any linear property of the label distribution, generalizing the truthful calibration error for binary predictions in Hartline et al. [2025]. We also identify a truthful correction for confidence calibration. Second, we characterize the decision-theoretic implications of these truthful errors. For calibrated predictors, truthful calibration errors preserve the Blackwell dominance: a more informative calibrated predictor receives no larger expected error. Third, we show that this decision-theoretic interpretation explains and mitigates the well-observed ranking robustness problem of binned calibration errors. Empirically, non-truthful confidence-based errors can reverse model rankings when the number of bins changes, while our truthful classwise error gives more stable rankings across binning choices.

Recent progress in computer vision has produced a wide range of powerful specialized models for detection, segmentation, counting, and other visual tasks. However, these models are usually optimized for isolated task formulations, making it difficult to directly support general-purpose visual intelligence, especially when a task requires complex language understanding and dense small-object perception. In this paper, we propose VisHarness, a trainable visual agent that decouples high-level perception, reasoning, and decision-making from low-level task execution. Instead of training a model to solve a specific visual task, VisHarness learns to harness a set of carefully designed heterogeneous visual experts. This paradigm preserves the general intelligence of the agent while fully leveraging the precision advantages of specialized visual models in concrete visual tasks. With only lightweight training, VisHarness learns a generalizable visual expert-harnessing policy and can solve common fundamental vision tasks under various complex conditions through multi-turn interactions with visual expert models. To enable efficient on-policy reinforcement learning training in a live environment, we introduce dynamic visual memory archiving, which mitigates the rapidly accumulating visual-token overhead caused by multi-turn interactions with visual expert models. Experiments on four representative benchmarks covering reasoning segmentation, generalized referring segmentation, dense small-object detection, and referring counting demonstrate that VisHarness substantially outperforms existing general-purpose models and achieves competitive or superior performance compared with task-specific models.

Semantic fidelity is essential for multi-agent communication under non-ideal channels, comprising transmission fidelity and conversion fidelity. Yet existing numerical and neural message formats cannot simultaneously achieve both. To address this, we propose a novel message format-\textbf{H}yperdimensional \textbf{S}ymbolic \textbf{M}essages (HSM), and its corresponding communication framework \textbf{Herm}. For transmission fidelity, Herm suppresses semantic entanglement via an entity-attribute-value structured encoding pipeline. For conversion fidelity, Herm first leverages a receiver-centric ego-binding mechanism to align semantics. Then, a similarity-aware dual-branch fusion module is proposed to prevent critical semantic dilution. To realize explicit message-semantic conversion, we devise an iterative decoding scheme via algebraic inverse operation. Experiments exhibit that Herm outperforms baselines across various noisy scenarios while maintaining superior structural readability.

AI alignment and participatory design motivate a new democratic design problem: how to collectively choose a _decision rule to use repeatedly_. We study this problem for _linear ranking rules_, which repeatedly rank items $x_j$ within batches $X=(x_1,\dots,x_m)\in(\mathbb{R}^d)^m$, where each item's ranking is dictated by its score $\langle \theta^{\ast},x_j\rangle$ according to a fixed scoring vector $\theta^{\ast}$. Given voters' preferred scoring vectors $\theta^{(1)},\dots,\theta^{(n)}$ and their population fractions $\alpha^{(1)},\dots,\alpha^{(n)}$, we ask how to choose a collective vector $\theta^{\ast}$ satisfying _individual proportionality (IP)_: every voter type $i$ should agree with the resulting rankings to an $\alpha^{(i)}$-proportional degree, either on average over time (_long-run IP_) or even within each batch (_per-batch IP_). The default rule, the arithmetic mean of the $\theta^{(i)}$, has been shown to be severely majoritarian; more generally, it is not clear that _any_ fixed linear rule can balance many voters' disparate opinions. Our main result is that, surprisingly, there _is_ a simple rule that does satisfy long-run IP: the _angular mean_, the spherical analog of the arithmetic mean. We then show that exact per-batch IP is impossible for fixed linear rules, but that the gap between per-batch and long-run IP shrinks quickly with batch size. Experiments on three real-world preference datasets show that all rules perform similarly when voters' preferences are homogeneous, while the angular mean substantially improves proportionality in high-disagreement regimes.

KV-cache compression for long-horizon LLM agents is often guided by accumulated attention, yet attention frequency can retain repetitive error traces while discarding decisive instructions. We propose TaskKV, a streaming step-level retention policy with three contributions: (1) a signed directional utility label from the first-order change in action likelihood, distinguishing helpful steps from confusing distractors unlike unsigned saliency; (2) a structured residual scorer that treats attention mass and intent alignment as fixed priors and learns only a denoising correction, converging with 500 trajectories; and (3) an entropy-based regime classifier that falls back to contiguous retention when the scorer cannot confidently rank steps. With a 20% KV budget on AgentBench and AlfWorld, TaskKV preserves $\sim$87--89% of full-cache success rate, improving over SnapKV by 5.0--7.9 SR points and over a strong heuristic by 2.9--7.3 points; on dense tasks, gains come from detecting when utility pruning should be disabled. Code is available at https://anonymous.4open.science/r/taskkv-F0D5.

Soft Actor-Critic (SAC) and its variants dominate Multi-Task Reinforcement Learning (MTRL) due to their off-policy sample efficiency, while on-policy methods such as Proximal Policy Optimization (PPO) remain underexplored. We diagnose that PPO in MTRL suffers from a previously overlooked issue: critic-side gradient ill-conditioning, which may cause tail tasks to stall while easy tasks dominate the value function's updates. To address this, we propose TOPPO (Tail-Optimized PPO), a reformulation of PPO via Critic Balancing---a set of modules that improve gradient conditioning and balance learning dynamics across tasks. Unlike prior approaches that rely on modular architectures or large models, TOPPO targets the optimization bottleneck within PPO itself. Empirically, TOPPO achieves stronger mean and tail-task performance than published SAC-family and ARS-family baselines while using substantially fewer parameters and environment steps on Meta-World+ benchmark. Notably, TOPPO matches or surpasses strong SAC baselines early in training and maintains superior performance at full budget. Ablations confirm the effectiveness of each module in TOPPO and provide insights into their interactions. Our results demonstrate that, with proper optimization, on-policy methods can rival or exceed off-policy approaches in MTRL, challenging the prevailing reliance on SAC and highlighting critic-side gradient conditioning as the central bottleneck.

Recent work has promoted task-aware layer pruning as a way to improve model performance on particular tasks, as shown by TALE. In this paper, we investigate when such improvements occur and why. We show first that, across controlled polynomial regression tasks and large language models, such pruning yields no benefit on in-distribution (ID) data but consistently improves out-of-distribution (OOD) accuracy. We further show empirically that OOD inputs induce layerwise norm and pairwise-distance profiles that deviate from the corresponding ID profiles. This leads to a geometric explanation of task-aware pruning: each task induces a task-adapted geometry, characterized empirically by the representation profiles observed on ID inputs. OOD inputs can introduce a distorted version of the task-adapted geometry. Task-aware pruning identifies layers that create or amplify this distortion; by removing them, it shifts OOD representational norms and pairwise distances toward those observed on the adapted distribution. This realigns OOD inputs with the model’s task-adapted geometry and improves performance. We provide causal evidence through controlled distribution shifts and residual-scaling interventions, and demonstrate consistent behavior across model scales.

We present SyncLight, a method to enable consistent, parametric control over light sources across multiple uncalibrated views of a static scene conditioning in a single view. While single-view relighting has advanced significantly, existing generative approaches struggle to maintain the rigorous lighting consistency essential for multi-camera broadcasts, stereoscopic cinema, and virtual production. SyncLight addresses this by enabling precise control over light intensity and color across a multi-view capture of a scene, conditioned on a single reference edit. Our method leverages a multi-view diffusion transformer trained using a latent bridge matching formulation, achieving high-fidelity relighting of the entire image set in a single inference step. To facilitate training, we introduce a large-scale hybrid dataset comprising diverse synthetic environments-curated from existing sources and newly designed scenes-alongside high-fidelity, real-world multi-view captures under calibrated illumination. Surprisingly, though trained only on image pairs, SyncLight generalizes zero-shot to an arbitrary number of viewpoints, effectively propagating lighting changes across all views, without requiring camera pose information. SyncLight enables practical relighting workflows for multi-view capture systems. Dataset, code, and models will be released upon acceptance.

This position paper argues that environmental cost should be part of the objectives used to train, select, and deploy AI models. Today, sustainability is mainly measured after design choices have been made, while architectures, data, hyperparameters, and serving strategies are chosen through accuracy, latency, and other proxies. This separation weakens model development. In training, newer models can move toward higher emissions without a comparable gain in performance. In inference, small differences in energy per query can become large lifecycle costs at scale. FLOPs, parameter counts, and latency do not capture these effects, because the real cost depends on hardware, datacenter efficiency, energy mix, water demand, and deployment volume. We call for multiobjective model development where performance and lifecycle environmental cost guide decisions together. The paper explains how to distinguish sustainable progress from costlier forms of progress, addresses alternative views, and outlines research directions for objectives, benchmarks, and model selection practices that make sustainability part of optimization.

Interactive language models should respond to user feedback without blindly conforming to incorrect pressure. Existing evaluations of sycophancy typically measure whether models flip their answers under pressure and treat apparent recovery after pressure removal as robustness. We show that this recovery is often superficial. We introduce the **Pressure--Recovery--Relapse (PRR) protocol**, a multi-turn evaluation framework that disentangles pressure-induced flipping, surface recovery, and relapse under renewed weak pressure. Across factual QA benchmarks and instruction-tuned models, PRR reveals three findings: (1) models can appear to recover at the output level while remaining vulnerable to renewed pressure; (2) surface answer correction can be dissociated from repair of the latent answer state; and (3) different internal components support distinct recovery pathways. Through causal interventions, we show that output-level steering can enforce correct answers without repairing corrupted latent readouts, whereas restoring clean hidden states repairs both surface behavior and latent states. Further component-level restoration shows that residual-stream states enable full repair, while mid-layer attention outputs provide partial latent repair. These results highlight that robust multi-turn reliability requires evaluating recovery stability and latent state repair, not merely apparent answer correctness.

Continuous DAG learners typically optimize a single weighted adjacency matrix, forcing each entry to represent both direct-edge support and signed effect magnitude. As a result, data fitting, sparsity, acyclicity, and any auxiliary structural information are all imposed on the same variables, even when the available information concerns higher-order relations such as reachability or ancestral order. We propose a structure-separated continuous DAG-learning framework that gives graph support its own soft representation. The effective adjacency is \( A=(W_f(M)\circ G(Z))\circ(\mathbf{1}-I), \qquad W_f(M)=\beta\tanh(M), \) where \(W_f(M)\) is a bounded signed-magnitude branch and \(G(Z)\) is a structural gate. Data fitting, sparsity, and the DAGMA barrier are evaluated on \(A\), while cycle and relation constraints are applied to closure scores computed from \(G\). The bounded magnitude branch prevents strong effective edges from being hidden behind vanishing structural gates, making closure-side structural regularization meaningful. On this separated structural gate, we introduce SMNC, a scan-efficient multi-scale normalized closure module. SMNC computes dyadic closure scores over \(G\) and normalizes each scale by its path-length budget, producing scale-normalized matrix-valued logits for self-reachability and off-diagonal reachability. This design addresses a key limitation of unnormalized product-style closure scores, whose path evidence scales as \(p^L\) on a length-\(L\) path with gate value \(p\), causing long-cycle signals to decay exponentially. At the appropriate covering scale, SMNC routes gradients through an active length-\(L\) cycle with a polynomial \(1/L\)-type scale factor; in the all-scale soft readout used in practice, this should be interpreted as a local active-scale effect. The same relation space supports diagonal self-reachability losses for cycle suppression and masked off-diagonal losses for optional ancestry, backward-exclusion, and incomparability cues, without treating positive reachability labels as direct-edge labels. Across linear and nonlinear SEM benchmarks, with ablations over factorization and closure normalization, we show that structure-separated SMNC improves DAG recovery and provides a principled closure-space interface for higher-order structural supervision.

Pretraining for electroencephalogram (EEG) foundation models has predominantly relied on self-supervised masked reconstruction, a paradigm largely adapted from and inspired by the success of vision and language foundation models. However, unlike images and text, EEG datasets are notoriously expensive to collect and characterized by low signal-to-noise ratio. These challenges introduce difficulties in scaling the EEG foundation models and capturing the underlying neural semantics through reconstruction. In this work, we ask the question: can we stand on the shoulders of well-established foundation models from well-represented modalities to bootstrap the pretraining of EEG foundation models? We first demonstrate that mainstream foundation models, such as those from vision and time series, transfer surprisingly well to EEG domain. Motivated by this observation, we propose the multi-teacher distillation pretraining (MTDP) framework for pretraining EEG foundation models via two-stage multi-teacher distillation. In the first stage, we introduce a learnable gating network to fuse representations from diverse teachers (e.g., DINOv3 and Chronos) via a masked latent denoising objective. In the second stage, we distill the fused representation into an EEG foundation model. Extensive evaluations across 2 backbone architectures, 9 downstream tasks and 12 datasets demonstrate that multi-teacher distillation pretraining consistently outperforms self-supervised pretraining for EEG foundation model and remains competitive even when the pretraining data is reduced to 25%.