NeurIPS 2026 论文列表

Standard Flow Matching (FM) models typically employ a pointwise loss function, effectively assuming that the velocity vector at each spatial position is pointwise decomposed supervision given the conditioning. We argue that this Pointwise Independence Assumption is suboptimal for image generation, where strong local correlations exist across space. Neglecting these dependencies leads to artifacts such as structural inconsistency. In this paper, we introduce Contextual Flow Matching (CFM), a generalized framework that enforces consistency not just on individual pixels, but on the contextual relationships within the velocity field. We formulate the training objective using a set of generalized Context Operators---akin to convolution kernels---that abstract relationship constraints. Specifically, we instantiate a Differential Operator to capture high-frequency motion dynamics and an Aggregate Operator to stabilize low-frequency structural evolution. Extensive experiments on ImageNet demonstrate that CFM significantly accelerates convergence and enhances generation fidelity by explicitly modeling spatial dependencies.

Modern object detectors achieve strong performance on standard benchmarks, yet their robustness to contextual variation remains insufficiently investigated. Prior evaluations largely rely on aggregate metrics such as AP on uncontrolled distribution shifts, which conflate multiple factors and can obscure how performance degrades under context change. We introduce \textsc{ContextShift}, a controlled benchmark for evaluating context dependence by systematically manipulating object--context relationships while preserving object appearance. Built on COCO 2017, it isolates context as an independent variable both implicitly, through geometric transformations and explicitly via synthetic and natural background substitutions - with a compatibility axis based on normalized pointwise mutual information (NPMI) for continuous evaluation. Across diverse detector architectures, we observe a consistent degradation pattern: false negatives increase up to 227\% and prediction volume decreases down to -44\%, while false positives remain relatively stable or decline. This observed suppression behavior is not revealed by aggregate metrics such as AP, which can mask substantial recall loss and changes in prediction dynamics. Further analysis suggests that degradation is not primarily explained by reduced confidence, but is instead consistent with a reduced formation of valid detection candidates. Moreover, we find that performance along the statistical compatibility axis is non-monotonic, peaking at intermediate NPMI and degrading toward both extremes, indicating that statistical co-occurrence does not correlate linearly with effective visual context. Finally, we show that a training methodology based on context-aware augmentations rather than the dataset alone, which yields overall better models: every augmented variant outperforms the dataset-only baseline on both original (unmanipulated) test images as well as manipulated images, indicating that data-oriented augmentation can partially recover performance lost to prediction-suppression failures by exposing the model to object--context decoupling during training.

Autoregressive image generation has recently emerged as a competitive visual generation paradigm, yet its generation order is often predefined or heuristic, such as next-patch prediction, random-order prediction, or one-step generation for specific scales. We observe that these strategies can cause models to overlook important structural dependencies in images, thereby limiting both structural robustness and reasoning capability. In this work, we propose a context-aware autoregressive generation framework, where the model dynamically determines the next token location based on global context and current local information. Rather than following a fixed or random order, our method allows the generation process to adapt to image structure and semantic dependencies. Extensive experiments show that context-aware generation significantly improves structural robustness across different image AR paradigms, including masked autoregressive and scale-wise autoregressive models. Beyond quantitative improvements, we further observe emerging reasoning properties: the model can better understand complex semantic instructions and generate images that more faithfully follow the given guidance. These findings suggest that generation order is a critical yet underexplored factor in image autoregressive modeling, and that context-aware decoding can unlock stronger reasoning-aware visual generation capabilities.

Transformers rely on a growing key–value (KV) cache to store context, causing linear memory growth and limited generalization beyond the training window. We show that contextual influence can instead be viewed as updates to a low-dimensional subspace in the model’s weight space, meaning context can be represented as a bounded, low-rank parameter update rather than token-level storage. Motivated by this, we propose PARADE, which internalizes context as parametric memory. It compresses past inputs into a fixed set of weight-space basis vectors and composes them via query-dependent routing into per-query weight updates, enabling streaming inference with constant memory and effectively unbounded context. Experiments show that PARADE matches full attention on long-context tasks and outperforms efficient baselines, especially when relevant information lies far beyond the attention window, while improving as context exceeds the model’s training length.

In-context reinforcement learning (ICRL) has emerged as an effective paradigm for test time adaptation to unseen tasks without parameter updates. However, existing ICRL methods can exhibit brittle and unstable adaptions, and the mechanisms underlying such adaptions remain poorly understood. We provide a policy-gradient view of ICRL and argue that relying on trajectory-level return feedback can lead to high-variance updates when test time rollout is limited. Motivated by the actor--critic principle, we propose **Context Value Informed ICRL (CV-ICRL)**, which equips the in-context policy with an explicit *context value*: the expected discounted return conditioned on the current state and accumulated interaction context. CV-ICRL trains a value head and writes its prediction back into the context as a value token, enabling TD-style return targets for lower-variance policy updates. Experiments on the Dark Room, Minigrid, and Procgen testbeds show that CV-ICRL substantially improves the stability of test time adaptation and achieves higher returns across tasks and environments. The source code and data of this paper are available at https://anonymous.4open.science/r/CV-ICRL-D161.

Real-world generative problems in biology, medical imaging, or robotics, rarely come with perfectly paired source and target distributions. While unbalanced Optimal Transport (uOT) provides a robust framework to bridge unpaired datasets via minimum-length paths in probability space, existing formulations based on the Wasserstein–Fisher–Rao (WFR) geometry introduce a structural bias toward high density regions, often triggering mode collapse. We introduce Contact unbalanced Optimal Transport (CuOT), a novel uOT formulation grounded in contact geometry that decouples density transport from mass growth to eliminate structural bias. This allows CuOT to adapt sampling steps to the local data structure, naturally reducing step sizes in high-density regions to prevent overshooting and ensuring stable convergence. The result is a stable, scalable path-length minimization solver that consistently outperforms WFR-based methods on biological dynamics reconstruction, unpaired image-to-image translation, and video generation.

Motivated by agentic markets -- two-sided markets in which AI tools facilitate search -- we propose a model that incorporates individual consumer search and market-wide learning, solve it to understand the long-run market outcomes, and then study the impact of improved search technologies on these outcomes. A sequence of consumers engage in costly search to acquire progressively refined signals of product fit prior to purchase. The market observes acquired signals and post-transaction feedback thereby impacting future searches. We solve for the per-consumer optimal search policy and characterize the steady-state of the learning dynamics. As search technologies reduce the cost and/or increase the informativeness of search, they cause an improvement in learning and consumer surplus. Using numerical simulations, we also find that these technologies improve the rate of learning and the trajectory of consumer utility. Notably, we show that if the market is unable to observe acquired signals, search technologies can degrade market outcomes, highlighting the importance of transparency in agentic markets.

The quadratic assignment problem (QAP) is widely considered one of the most difficult NP-hard combinatorial optimization problems, and one that remains challenging for neural constructive methods. We introduce CIMP, a Constructive policy trained by Imitation from Multi-expert teachers and built around an explicit Pairwise pre-conditioning block. CIMP reaches a 11.02% mean optimality gap on the 133 QAPLIB instances with published reference values, across 5 fixed same-config seeds, in a single constructive pass — without local search or per-instance refinement — with inference times between 0.1 and 3.1 seconds per instance. The model is trained on 400 synthetic instances with n ≤ 25. The most recent published neural reference reporting full QAPLIB results is SAWT [Tan and Mu, ICML 2024], a learn-to-improve (L2I) method that iteratively refines an assignment per instance after training on 5,120 synthetic instances, and reports a 26.8% mean gap. CIMP operates in a strictly more constrained learn-to-construct (L2C) regime and from a much smaller training budget, yet attains a substantially lower mean gap. The pre-block builds a state-dependent facility–location representation before scoring, a hybrid context-biased decoder mixes learned content similarity with QAP-specific algorithmic bias terms, and a sequence-distribution distillation objective is derived from a diversity-filtered pool of heuristic teachers. Family-wise and size-conditioned analyses, supported by a secondary normalized score, indicate that the gain is broadly consistent across QAPLIB families and across instance sizes beyond the training range.

Constrained learning has been increasingly applied to various domains to ensure explicit feasibility requirements due to fairness, safety, robustness, regularization, and physics or logic constraints. Understanding how training samples influence the solution (e.g., learned parameters) of constrained learning is crucial for interpretability and robustness. The classical influence function may become unreliable in constrained settings: data perturbations can reshape both the objective and the feasible region, leading to estimates that violate feasibility. In response, we propose the Directional Influence Function (DIF), a new estimator that explicitly incorporates the constraints into influence estimation. DIF formulates the optimality conditions of constrained learning as a variational inequality (VI) and analyzes how perturbing training data affects this VI. We validate DIF in constrained linear regression and demonstrate that it recovers leave-one-out retraining results, whereas IF and penalty-based IF exhibit significant bias. We further apply DIF to fairness-constrained CNNs, where DIF accurately predicts test loss changes under data removal and aligns closely with actual retraining. Our results establish DIF as an efficient and reliable tool for data attribution in constrained learning.

Prompt-level instructions are the main way users give language models preferences, safety boundaries, and content rules. But a model can remember a constraint and still violate it when generating text. This points to a gap between two abilities that are often treated as one: retrieving a constraint from context, and enforcing it during token selection. Across five model families (0.5B--32B parameters), we find a clear behavioral split: constraints that admit an alternative generation plan are largely followed, while token-level prohibitions fail whenever the prohibited continuation is locally dominant. Mechanistic analyses show that retrieval heads still attend to the constraint at the violation step, but the resulting signal either paradoxically enhances the prohibited token's logit or suppresses it too weakly to change the argmax. Decoding-time interventions can eliminate surface-form violations, but semantic compliance still requires higher-level verification. These results establish that constraint retrieval and constraint enforcement are separable capabilities, and that reliable token-level compliance requires explicit veto mechanisms beyond prompting.

In this work, we develop proximal preconditioned gradient methods with a focus on spectral gradient methods providing a proximal extension to the Muon and Scion optimizers. We introduce a family of stochastic algorithms that can handle a wide variety of convex and nonconvex constraints and study its convergence under heavy-tailed noise, through a novel analysis tailored to the geometry of the proposed methods. We further propose a variance-reduced version, which achieves faster convergence under standard noise assumptions. Finally, we show that the polynomial iterations used in Muon are more accurately captured by a nonlinear preconditioner than by the ideal matrix sign, leading to a convergence analysis that more faithfully reflects practical implementations.

Biological recurrent circuits can reuse a common substrate by changing its operating regime rather than replacing the circuit itself. This principle is difficult to isolate in fully adaptive neural models, where recurrence, controllers, readouts, and internal representations can all absorb task structure. As a result, successful multi-context behavior does not reveal which route made the computation possible. We introduce Constrained Modulatory Reservoirs (CMR), a fixed-substrate framework for studying such reconfiguration with the main bypasses removed. The primary dynamics and low-rank perturbation basis are fixed, and the readout is denied direct access to the cue, forcing input-dependent adaptation to act through a limited set of recurrent connectivity changes. In transient-cue channel-selection tasks, this restriction makes a capacity transition visible. Low-loss behavior appears only when the available perturbation directions are sufficient, the selected change is retained across the cue-to-signal gap, and the primary dynamics express it during the response window. Controls show that this effect is not explained by low-dimensional cue representation alone. Allowing the substrate to learn hides the bottleneck, while activity- and gain-level alternatives fail to reproduce the same regime. CMR therefore reframes flexible recurrent computation as a question of where adaptive influence is allowed to enter a fixed dynamical system, making capacity, memory, and expression experimentally separable.

We introduce a framework that models constrained MDPs subject to trajectory-wise safety constraints. These constraints require that expected future costs remain below zero conditioned to any possible trajectory---not merely in expectation from the initial state. This captures stronger and more realistic safety guarantees, which are crucial in high-stakes applications. We characterize the structure of optimal policies, and show that Markovian policies are in general suboptimal. Moreover, we show that computing approximately-safe optimal policies is NP-hard when the number of constraints can be arbitrarily large. Despite all these challenges, we provide an exact algorithmic characterization of optimal history-dependent (non-Markovian) policies, by exploiting a suitable recursive formulation of the sets of reachable reward-cost values. Then, we employ it to design a polynomial-time approximation algorithm that computes approximately-safe optimal policies by dynamic programming, when assuming a constant number of constraints.

Diffusion and flow models enable powerful generative modeling, but controlled editing remains challenging due to the need to balance source fidelity and edit compliance along the generative trajectory. We propose a principled, inversion-free framework for flow-based image editing from a trajectory centric perspective. Rather than relying on inversion or heuristic guidance, we introduce a one-step look-ahead mechanism that evaluates the effect of candidate updates on both edit and fidelity rewards estimated via denoising, and adjusts the current update based on this evaluation. At each step, our method applies reward-driven corrections guided by the look-ahead evaluation under an explicit fidelity constraint, resulting in stable and coherent trajectory evolution. This look-ahead design mitigates gradient interference between edit and fidelity objectives by anticipating their interaction in future states, leading to improved trade-offs. Experiments demonstrate that the proposed approach achieves stronger semantic edits while better preserving source content compared to existing strong baselines.

Planar graphs are central to applications across science and engineering, yet existing generators provide limited support for goal-directed generation under hard structural and geometric feasibility constraints. We propose a dataset-free method for generating planar graph embeddings by combining parametric graph grammars with safe reinforcement learning to optimize generic task-specific objectives while satisfying constraints during construction. We formulate the generation process as a constrained Markov decision process, where the graph grammar defines the state and action spaces. We further introduce an action projection that maps sampled actions toward state-dependent safe sets, improving constraint satisfaction during training. In contrast to classical graph generators and deep generative models, which typically offer limited goal-directed control or rely on weak constraint satisfaction, our method constructs feasible planar graph embeddings directly during generation. We also introduce a benchmark suite for constrained and goal-directed planar graph generation, together with classical and deep generative baselines. Across all benchmark tasks, our method consistently outperforms baselines while satisfying the formulated constraints.

Constrained decoding is essential for serving LLMs, ensuring that generated outputs follow specific structures such as JSON schema-formatted function calls. Existing systems are designed for autoregressive models and assume left-to-right generation, masking out invalid next tokens at each step. Diffusion language models, however, break this assumption: they sample multiple positions simultaneously from a fully-factorized mean-field distribution at each denoising step. In this paper, we present an exact and tractable algorithm for sampling from the constrained mean-field posterior under any constraint expressible as a finite automaton. Viewing finite automata as graphical models, we obtain tractable representations of the constrained distribution that enable efficient inference. The approach guarantees constraint satisfaction by construction, supports both greedy and sampling-based decoding, and is compatible with parallel and block-wise decoding under arbitrary remasking schedules. Applying depth-reduction techniques from arithmetic circuit theory, we further reduce sampling depth from linear to logarithmic in the sequence length. Empirical evaluations on Dream-7B and LLaDA-8B show substantial accuracy gains across various tasks including function calling (xLAM, BFCL), planning (Sudoku, Countdown), text-to-SQL (Spider), and math reasoning (GSM-Symbolic), with little inference overhead relative to unconstrained decoding. For example, on BFCL-Live, our approach improves Dream-7B's greedy decoding accuracy from 63.9% to 71.5%, and stochastic sampling accuracy from 22.3% to 69.0%, where the unconstrained baseline collapses, with under 5% wall-clock overhead.

Determinantal point processes allow efficient sampling of diverse subsets of data. However, (exact) sampling under additional constraints is often computationally intractable under complexity-theoretic assumptions. We take an algebraic approach to designing novel algorithms that allow exact constrained sampling with theoretical efficiency guarantees. Our algorithms apply to a general class of point processes based on Bombieri inner products of polynomials, which we call Bombieri $k$-point processes. This perspective recovers known algorithms for constrained $k$-DPPs and extends them to a broader class of simple $k$-point processes. As applications, we obtain exact samplers for matching-, path-, subgraph-, and budget-constrained processes, as well as distributions combining diversity with clustering. While common complexity-theoretic assumptions rule out polynomial-time algorithms in many of these settings, our algorithms are efficient with respect to the relaxed notion of fixed-parameter tractability when the sample size $k$ is kept bounded.

Post-training of language models is commonly framed as a sample-score-update loop implemented by gradient descent. A recent line of work, exemplified by RandOpt, relocates this loop to weight space, sampling Gaussian perturbations around a pretrained model and ensembling the top-$K$ rewarded specialists at inference. While competitive with PPO and GRPO under matched training compute, this prediction-level ensemble incurs $K$ forward passes per test example and does not extend cleanly to free-form generation. We ask whether the rewarded population can instead be folded into a single deployable model, replacing the inference-time ensemble with one consolidated update. A split-half analysis over 25 model-task pairs reveals reproducible low-rank structure in every case. We turn this geometry into CoRP (Consolidating Rewarded Perturbations), a gradient-free operator that combines reward-weighted aggregation, compatibility-aware reweighting, and a held-out validation gate, with no gradient flowing through the language model. Across five language models from $0.5$B to $8$B and five tasks covering math, code, and creative writing, CoRP improves the base model by $8.1$ points on average. Using one tenth of RandOpt's perturbation budget, CoRP exceeds single-inference RandOpt by $6.5$ points and recovers more than half of the gain of the 50-pass majority-vote ensemble, at one forward pass per test example.

Parallel thinking scales inference-time compute by generating multiple reasoning traces for a query, and aggregating them to reach a final answer. However, most aggregation algorithms treat each reasoning trajectory independently, ignoring the complementary reasoning sub-processes across the full ensemble of trajectories. To learn from these parallel trajectories, we propose CORAL (Consolidating Reasoning with Test-Time Learning), a novel framework for aggregating trajectories by jointly consolidating them into parametric memory using test-time training. Our algorithm meta-learns how to consolidate using nested optimization at training time, where an inner loop encodes reasoning trajectories into a lightweight LoRA adapter, and an outer loop learns meta-parameters for the inner loop such that the consolidation is helpful for the model to synthesize a final solution. As no standard datasets exist for training aggregation methods, we construct a novel OpenParallelThinking corpus composed of 12,810 math problems paired with a mixture of diverse trajectories from six LLMs. Evaluated on challenging mathematics and STEM benchmarks, CORAL consistently outperforms best aggregation baselines by 20% overall. Importantly, we curate reasoning trajectories at multiple quality levels to simulate the reality of noisy parallel reasoning and demonstrate that CORAL is especially robust to the quality of thinking trajectories and able to learn useful knowledge from entirely wrong solutions. Additionally, this weak-to-strong generalization feature scales with the number of incorrect trajectories available for aggregation, while RL can further incentivize such reasoning consolidation capability. We will release CORAL and OpenParallelThinking for use in future work on reasoning aggregation.