NeurIPS 2026 Papers

Online Convex Optimization with memory and time-varying constraints (COCO-M) is a recently-introduced framework that captures the dynamics of stateful online learning and non-stochastic control with budget constraints. Despite its expressive power, COCO-M remains largely understudied. In this work, we tackle this problem in its full scale, providing an optimistic algorithm that incorporates untrusted predictions to tame universal dynamic regret and cumulative constraint violation. Our analysis leverages a potent reduction from memory to delay and treats state-dependent gradients as delayed feedback. We establish the first bounds that adapt to both environment non-stationarity (path length $P_T$) and prediction accuracy, while lifting previous restrictive assumptions such as functional separability. En route to these results, we derive findings of independent interest for the memory-to-delay reduction, and for delayed OCO with time-varying constraints. The proposed framework ensures sublinear worst-case guarantees that improve with prediction accuracy, reaching \(\mathcal O(1)\) regret under perfect predictions, with cumulative constraint violation (CCV) $\mathcal O(\sqrt{T})$ for unknown $P_T$ and $\mathcal O(\log T)$ for known $P_T$, while recovering the optimal COCO rates when memory disappears.

Depth-recurrence promises improved latent reasoning by sharing parameters across depths. However, prior work still relies on partially fixed layer stacks and overlooks the bottleneck of constant hidden-size; moreover, it largely lacks rigorously resource-matched ablations. To address this, we introduce depth-recurrent attention mixtures (Dreamer), a modular architecture with a single recurring layer. Concretely, we mix sequence attention, depth attention, and sparse expert attention. This alleviates the hidden-size bottleneck, decouples scaling dimensions, and achieves effective yet efficient depth-recurrence. Across natural language reasoning benchmarks at up to ~2B parameters, Dreamer requires on average ~3x fewer training tokens for the same accuracy as FLOP-, parameter-, and memory-matched modern Transformers, and outperforms ~2x larger Transformers given same training tokens. Analysis further reveals 2-11x greater expert selection diversity than conventional MoEs, highlighting the flexible knowledge sharing across depths.

Decoding cognitive functions and brain diseases from functional magnetic resonance imaging (fMRI) is a foundational problem in neuroscience advanced by recent deep learning across task and resting-state (rs) settings. Existing region-level decoders capture spatial structure through region-of-interest (ROI)-aware graph or Transformer architectures, but apply a uniform temporal operator across all regions. Consequently, they do not explicitly model how cortical regions integrate information over heterogeneous timescales. We propose CoRTeX, a region-resolved recurrent architecture for fMRI in which each cortical region carries its own learnable parameters and evolves on its own learned timescale. At its core, per-region differentiable temporal integration windows allow each ROI to learn how far back in time to integrate information through a soft mask over a private buffer. Cross-region information is restricted to a single identity-aware attention channel, additionally biased by a co-activation-driven memory whose learning and forgetting rates are themselves region-specific. Experiments on stimulus classification using task-fMRI (NSD) and on two disease diagnosis tasks using rs-fMRI (ABIDE, COBRE) demonstrate that CoRTeX achieves competitive performance with strong baselines while enabling region-level analyses that are not well-defined for models with exchangeable units: the learned per-region timescales exhibit interpretable spatial organization, and class-specific regional attribution aligns with category-selective cortex. Code is available at: https://anonymous.4open.science/r/CoRTeX-F.

Transformers rely on a growing key–value (KV) cache to store context, causing linear memory growth and limited generalization beyond the training window. We show that contextual influence can instead be viewed as updates to a low-dimensional subspace in the model’s weight space, meaning context can be represented as a bounded, low-rank parameter update rather than token-level storage. Motivated by this, we propose PARADE, which internalizes context as parametric memory. It compresses past inputs into a fixed set of weight-space basis vectors and composes them via query-dependent routing into per-query weight updates, enabling streaming inference with constant memory and effectively unbounded context. Experiments show that PARADE matches full attention on long-context tasks and outperforms efficient baselines, especially when relevant information lies far beyond the attention window, while improving as context exceeds the model’s training length.

Geometry is invariant to viewpoint, which makes any collection of images a redundant encoding of a single 3D state. Existing feed-forward reconstruction models fail to exploit this: per-view methods emit overlapping, unaligned pointmaps that grow linearly with input count, while global-latent methods commit to a fixed, low-resolution output. We introduce Surflo, which compresses a variable number of unposed RGB views into K latent tokens-one global state-and decodes oriented 3D surface points by independently transporting them from noise onto the surface via flow matching. This frees the output from any fixed grid or token budget: the same latent yields from a few thousand to a million points in a single forward pass. To suppress the local inconsistencies inherent to independent per-point decoding, an inference-time guidance term correlates nearby points by injecting a photometric gradient during ODE integration. Surflo matches or surpasses feed-forward baselines on surface metrics, run an order of magnitude faster than optimization-based methods that require hundreds of views, and is the only feed-forward approach to combine a global latent with arbitrary-resolution decoding.

Swapping two LLM post-training stages can change the final model, but an aggregate loss or benchmark delta only says that something moved, not where the order-dependent residue landed. We ask whether this residue leaves a memory trace: a structured signal that is localized in output space, changes held-out order-gap NLL under targeted interventions, and is assignable from paired endpoint weights. For a specified two-stage path $(A,B)$, we define commutator memory by projecting the Lie bracket $b_{AB} := H_B g_A - H_A g_B$ at the base model through the logits into token scores $\tau_k=\mathbb{E}[e_k\,\delta z_k]$, whose sum is the leading bracket-predicted order effect. The token readout is localized: endpoint and batch-resampled checks preserve bracket top-token supports far more than norm-matched random directions (82--99% vs. 35--49% top-20 overlap), and a separate endpoint-support prediction check recovers 53--57% of Qwen top-1% active-token supports. It has intervention leverage in the tested protocols: top-positive-$\tau_k$ token interventions close 32% (median) of the held-out Qwen-3-4B SFT ordering gap with tau-neutral controls near zero, with the same bracket-vs-control separation on a pre-specified above-floor Qwen-2.5-1.5B fp32 subset. It is assignable from paired endpoints: given a base model, candidate stages, and paired alternate-order endpoints, the statistic $\langle\theta_{AB}-\theta_{BA},\,b_{AB}\rangle$ distinguishes $k=1$ SGD training order in 66/72 pair-seed units across four LLMs. Matched-batch DPO and frozen-rollout reward-surrogate experiments are matched-path stress tests; AdamW is reported as an analogous lifted-state pilot. Commutator memory turns training order from a scalar nuisance into a localized, intervention-tested, and paired-endpoint-readable residue.

Maintaining character consistency over long-horizon gameplay is a critical bottleneck for deploying Large Language Models (LLMs) as Non-Player Characters. Prevalent discrete Affective State Machines (ASMs) fail due to two paradoxes: (1) the \textit{Cliff-edge Effect}, where threshold-based switching causes jarring narrative discontinuities; and (2) \textit{Auto-regressive Inertia}, where accumulated dialogue history dilutes current state instructions. We propose \textbf{C3 (Causal-Continuous Characterization)}, which couples a continuous state space with \textit{hysteresis-aware dynamics} to dampen emotional volatility, and a \textit{State-Conditioned Memory Rewriting} mechanism that re-contextualizes past events through the lens of the current relationship. On a 100-turn dynamic simulation (\textsc{Chronos-Sim}, 5{,}000 turns per condition), C3 achieves a \textbf{10.5\% absolute SAR improvement} over a strong smoothed-ASM baseline (Welch's $t$-test, $p < 0.0001$, Cohen's $d \approx 3.97$). Cross-model evaluations on \texttt{Llama-3-8B-Instruct} and \texttt{Mistral-v0.2-7B} reproduce the relative gain, indicating the mechanism transfers across backbones although absolute SAR remains backbone-dependent. A human study with 20 game designers (ICC$=0.74$) corroborates higher perceived consistency, and a separate factual audit (Fleiss' $\kappa=0.81$) confirms the Rewriter alters interpretation without corrupting facts. Deployment analysis shows a $\sim$60\% reduction in long-session token consumption.

Recent cryptographic results establish that neural networks can be backdoored such that no efficient algorithm can distinguish them from a clean model. These guarantees, however, have been confined to stylised architectures of limited practical relevance, leaving open whether comparable undetectability extends to modern, end-to-end trained networks. We construct such an attack mechanism for state-of-the-art architectures, closely aligned to the cryptographic notion of undetectability, by identifying backdoor channels as learned latent directions, and show that the question of undetectability reduces to a hypothesis test between two unknown distributions over model parameters, which we conjecture to be intractable in practice. The consequence of this reframing is significant: if exploitable channels within a network's latent space are statistically indistinguishable from naturally learned directions, an attacker need not introduce foreign structure but can instead exploit the geometry the network already possesses. Demonstrating the approach on ResNet and Vision Transformer architectures trained on standard image classification datasets, the attack achieves both consistently high success rates with negligible clean accuracy degradation, and resists a comprehensive suite of post-training defences, none of which neutralise the backdoor without rendering the model unusable. Our results establish that cryptographic backdoors need not be artefacts requiring exotic architectures or artificial constructions, but identifiable as latent properties inherent to the geometry of learned representations.

Autoregressive (AR) video diffusion is a powerful paradigm for streaming and interactive video generation. However, its reliance on softmax self-attention leads to quadratic compute complexity in sequence length and increasing memory usage due to key-value caching, which fundamentally limits its scalability to long video horizons. Existing remedies (e.g., sparse attention and KV-cache compression) reduce per-step cost but still rely on a linearly growing cache or irreversibly discard past context, and thus fail to simultaneously address linear memory growth and streaming context management. To address this scalability bottleneck, we propose ARL$^2$ (Attend Locally, Remember Linearly), a hybrid attention module that replaces quadratic cross-frame attention with a fixed-size recurrent state. Specifically, we decompose self-attention into two branches: an intra-frame softmax branch for spatial detail and local dependencies, and an inter-frame gated recurrent linear branch that maintains a fixed-size state for streaming context. Our key insight is that softmax attention captures fine-grained local interactions, while a recurrent state provides a controllable mechanism for long-range memory. This design achieves linear-time scaling with constant memory while improving temporal consistency over the full-softmax model. To prevent noisy intermediate states from corrupting long-range memory, we update the recurrent state only after the denoised pass. To avoid within-frame information asymmetry, all tokens share the same pre-update state rather than sequential updates. To the best of our knowledge, this is the first work to convert a pretrained AR video diffusion model into a hybrid linear attention architecture, through an efficient two-stage training scheme adapted to the AR video setting. With 75% of layers replaced by hybrid linear attention, the model achieves up to $2.26\times$ wall-clock speedup and 54% memory reduction, while maintaining comparable quality with improving temporal consistency.

Sequence models in machine learning compress long input histories into finite-dimensional states, from recurrent networks and reservoir computing to recent state space models for long-context data. This raises a basic expressivity question: how simple can the recurrent memory be before it can no longer approximate stable causal sequence-to-sequence maps? We answer this question for fading-memory operators, where the distant past has uniformly diminishing influence. We show that every continuous causal time-equivariant fading-memory operator on a compact input set can be uniformly approximated by a state space model whose recurrence is a fixed diagonal linear contraction. The dynamics are time-invariant, input-independent, and contain no nonlinear hidden-state update or delay-line structure. The construction uses a bank of short exponential traces; although such traces do not exactly store a recent input window, a singular Vandermonde extraction approximately reconstructs the window while the remaining tail becomes negligible as delta decreases. We also prove that exact delay recovery is impossible for any fixed finite trace bank, showing that the accuracy-dependent extraction is essential. Technically, the result separates recurrent expressivity from exact memory storage, while exposing a conditioning cost in the readout. More broadly, it clarifies why highly stable, structurally simple SSM cores can still be universal approximators for stable sequence behavior.

We derive closed-form ODEs for the learning dynamics of a linear recurrent policy network trained with REINFORCE on sparse-reward tasks in a high-dimensional teacher–student setting. Under alignment assumptions between student and teacher networks, the dynamics reduce to a finite system of coupled order-parameter ODEs whose reward terms are governed by trajectory-level orthant probabilities of a structured correlated Gaussian process. The theory predicts that recurrence can accelerate escape from sparse-reward plateaus by accumulating task-relevant memory, and that each episode length admits an optimal recurrent scale that minimizes learning time. Our results provide a quantitative theory of how recurrence, and sparse reward interact during learning.

Temporal sequences in working memory are hierarchically organized to support flexible behavior. Recent neural recordings suggest that this organization relies on a 1D-to-2D neural geometrical folding in working memory, whereby ordinal positions are re-encoded within an orthogonal scaffold, with two dimensions indexing global (chunk-level rank) and local ranks (within-chunk position), respectively. This findings implicate an important structural principle for memory, yet the computational mechanisms underlying this geometry remain unclear. Here, we study this question using a recurrent neural network with fast Hebbian plasticity (Hebb-RNN). When meta-trained on rank recalling tasks, the Hebb-RNN generalizes to novel item–rank bindings and hierarchical sequences, outperforming control models. Mechanistic analysis reveals that the network develops an orthogonal ordinal geometry consistent with empirical findings. This geometry transforms temporal inputs into abstract ordinal ranks that provide a memory scaffold, while Hebbian plasticity supports rapid conjunctive binding of item content to this structure. Furthermore, the pre-learned geometric scaffold significantly accelerates learning in novel tasks, supporting efficient transfer. By incorporating a predictive module, our model extends to real-world encodings and acquires hierarchical geometry without explicit chunking signals. Together, our results provide a biologically plausible framework for hierarchical sequence representation, offering new insights into structured memory organization for both neuroscience and machine learning.