NeurIPS 2026 Papers

Mental simulation, the ability to reason by constructing and updating internal visual states, is central to flexible human cognition, but remains elusive in current multimodal large language models. Latent multimodal reasoning offers a promising route toward this capability by interleaving text with continuous visual latent blocks, allowing models to maintain internal visual workspaces without decoding them into pixels. In this paper, we ask whether these latent visual states actually matter. Through a direct perturbation analysis across representative latent-reasoning MLLMs, we find that they often do not: dropping, replacing, or shuffling generated visual latents has only minor effects on the model's own final-answer probability, while equivalent perturbations to the linguistic reasoning trace are substantially more damaging. We call this failure mode Mental Simulation Collapse (MSC): the model appears to perform interleaved visual reasoning, but routes most answer-relevant computation through text. We further diagnose that MSC follows naturally from existing training objectives: visual alignment can make generated latents resemble helper-image embeddings, but it does not require the decoder to rely on them; subsequent text-only supervision further leaves the semantic role of the latent pathway unidentifiable. To address this, we introduce AnchorSim (Anchored Mental Simulation), a contrastive distillation framework that trains models to make their generated visual latents consequential. Across multiple visual reasoning benchmarks, AnchorSim improves performance, while diagnostic re-evaluation shows that visual perturbations now meaningfully affect predictions. Our results expose a fundamental failure mode of latent multimodal reasoning and demonstrate that mental simulation must be anchored to model behavior in order to matter.

This position paper argues that AI for mental health must move beyond two dominant paradigms. The first is diagnostic prediction, which frames mental healthcare as symptom detection, diagnosis, and decision-making. The second is chat-based support, which reduces therapeutic interaction to language-mediated information delivery. Both approaches neglect what therapeutic research shows to be central to healing: experiential, affective, and embodied processes. In many therapeutic traditions, insight emerges not through information-delivery but through emotionally resonant, multisensory experiences that allow individuals to feel, reflect, and reframe their inner world. We propose an alternative framework: co-experiential AI systems that identify emotionally salient moments across modalities, modulate sensory experience to facilitate reflection, and adapt over time to a user's evolving responses and interpretations. Using dreamwork therapy as a stress test for mental health AI, we define three key challenges for AI systems: identifying affective anchors in indeterminate multimodal interactions; modeling user perspective as a situated context; and enabling continual adaptation to individuals without repeated retraining. We posit that addressing these challenges requires rethinking AI's role not as an expert, but as a perspective-aware participant in human meaning-making.

Multimodal video fusion methods still suffer from limited performance due to the inadequate exploitation of historical frames and the simplified temporal consistency modeling. To address this, we propose MemoryFusion, a two-stage framework built upon cross-temporal memory learning, which leverages historical information across multiple temporal scales to enhance robustness and progressively refine temporal consistency. In the first stage, we introduce a Short-Term Memory Bank (STMB) to aggregate recent temporal features for enhancing the current frame, and a Long-Term Memory Bank (LTMB) to preserve representative features from extended temporal sequences through adaptive update rules. Meanwhile, we incorporate a Temporal Smoothing Module (TSM) to perform coarse temporal consistency modeling, suppressing abrupt background variations and establishing a stable temporal foundation. In the second stage, we design a Temporal Refinement Module (TRM) implemented as a lightweight 3D residual network, which conducts prior-guided fine-grained temporal refinement to recover subtle temporal dynamics and spatial details overlooked in the first stage. Extensive experiments demonstrate that MemoryFusion significantly improves both visual fidelity and temporal consistency by performing cross-temporal learning, outperforming the state-of-the-art in both frame-based and video-based fusion methods. The source code and pretrained models will be publicly released.

Memory-augmented LLM agents enable interactions that extend beyond finite context windows by storing, updating, and reusing information across sessions. However, training such agents with reinforcement learning in multi-session environments is challenging because memory turns the agent's past actions into part of its future environment. Once different rollouts write, update, or delete different memories, they no longer share the same intermediate memory state, making trajectory-level comparisons fundamentally unfair. This violates a key assumption behind group-relative methods such as GRPO, where rollouts are compared as if they were sampled from the same effective environment. Consequently, trajectory-level rewards provide noisy or biased credit signals for long-horizon memory operations. To address this challenge, we introduce Memory-R2, a training framework for long-horizon memory-augmented LLM agents. Its core algorithm, LoGo-GRPO, combines local and global group-relative optimization. The global objective preserves end-to-end learning from long-horizon trajectory-level rewards, while local rerollouts compare alternative continuations from identical intermediate memory states, yielding fairer group comparisons and more precise supervision for memory writing, updating, and retrieval. Beyond credit assignment, Memory-R2 jointly optimizes memory formation and memory evolution with a shared-parameter co-learning design, where a fact extractor and a memory manager are instantiated from the same LLM backbone through role-specific prompts. To make memory construction more controllable, we further formulate each session as a multi-step decision process by chunking the interaction and alternating the two memory roles. A progressive curriculum from 8 to 16 to 32 sessions stabilizes training as the memory horizon grows. Together, these components provide an effective training paradigm for memory-intensive LLM agents in long-horizon multi-session settings.

Existing VLM-based open-set semi-supervised learning (OSSL) methods primarily rely on coarse-grained class-level semantics, thereby hampering the accurate identification of in-distribution (ID) versus out-of-distribution (OOD) samples. To overcome this issue, we propose a novel memory-driven contrastive embedding enhancement approach by integrating fine-grained visual contrastive cues into class-level textual side. Concretely, we design a memory bank to construct and maintain a diverse collection of the most distinctive fine-grained visual embeddings. Guided by the memory bank, visual information is incorporated into the textual side through cross-modal contrastive learning, enabling more discriminative fine-grained separation between ID and OOD samples. Consequently, the learned OSSL model exhibits improved contrastive discriminability across known and unknown categories. Extensive experiments demonstrate that our method achieves state-of-the-art (SOTA) performance on multiple fine-grained datasets. The code is available at https://anonymous.4open.science/r/CodeForPaper.

Associative memory models classically describe retrieval as convergence to stored patterns, but the theory of sequential retrieval remains comparatively limited. We develop a two-timescale input-driven Hopfield network in which fast associative retrieval is modulated by a slow feedback variable evolving over a prescribed memory-transition graph. This yields autonomous, graph-constrained transitions while preserving analytical tractability. Using slow-fast and geometric singular perturbation theory, we characterize memory fixation via memory manifolds and identify the geometric regions that govern stability loss and escape from retrieved memories. We further derive reduced slow dynamics and explicit criteria for transition onset, self-sustained retrieval, and collapse, expressed through escape times, gain thresholds, and fixed points. Together, these results provide a tractable framework for analyzing feedback-driven sequential retrieval in Hopfield networks.

The memory capabilities of Large Language Models (LLMs) have garnered increasing attention recently. Despite great success achieved, existing retrieval-based memory approaches typically overlook the differences between memories and employ a unified strategy to process all memories, leading to suboptimal performance. Thus, an intuitive question arises: can we categorize memory into different types and select appropriate strategies? However, given the topic-rich, scenario-complex, and boundary-blurred nature of memory scenarios, achieving precise classification of memories is not easy. To address this challenge, we propose a memory multi-class dataset in this paper, termed TriMEM, which provides precise annotations for memory types across diverse scenarios. Building upon this foundation, we propose a novel memory framework, named MemoType, which can adaptively recognize each memory and query type with the learned router model. With the memory and query routing, MemoType can retrieve the memory with corresponding query types and design tailored retrieval strategies, thereby enhancing the retrieval performance. Moreover, we theoretically prove that any single retrieval strategy is subject to a fundamental upper bound on its expected retrieval precision in multi-class corpora, leading to systematic precision degradation. Extensive experiments on three datasets demonstrate that MemoType consistently outperforms existing methods, achieving up to 16.18% improvement in Recall@1.

Long-context dialogue systems must decide both when to access memory and which parts of the interaction history are relevant. Existing approaches typically rely on heuristic retrieval signals or always-on memory usage, failing to account for the changing and potentially inconsistent nature of user preferences. In this work, we propose a unified framework for memory access and selection based on changing preferences. We formulate personalized memory retrieval as identifying which historical turns provide evidence about a user’s latent preference state, rather than relying on surface-level semantic similarity. To this end, we quantify the utility of each memory turn using a Bayes factor, defined as the improvement in the model’s likelihood of the reference response when the turn is included in context. This provides a principled measure of evidence strength and a unified signal for both memory access and selection. By framing memory retrieval as utility estimation, the model learns to identify salient turns and regulate memory usage based on expected utility. Experiments on four heterogeneous memory benchmarks show that our approach outperforms existing embedding-based retrieval on long-context, preference-intensive tasks where modeling changing preferences is essential, while remaining competitive in low-density regimes where semantic similarity suffices.

Steering large language models (LLMs) is usually performed through either instruction prompting or activation steering. Prompting offers strong control, but repeatedly caches guidance tokens and can clutter long interactions. In contrast, activation steering ,while compact, typically offers weaker control and less fine-grained steering because it does not support large structured reminders. We introduce memory inception (MI), a training-free method that steers in latent attention space by inserting text-derived key-value (KV) banks only at selected layers. MI treats steering as selective KV allocation, where reminder content need not occupy the full-layer prompt cache in order to induce implicit behavioral guidance. We formulate MI as attention over prompt, target, reference, and auxiliary banks, with canonical pre-RoPE key storage for reuse across positions and architectures. On matched personality-steering tasks, MI offers a balance between prompting and contrastive activation addition (CAA), approaching or exceeding prompting in raw control strength and CAA in drift mitigation. MI also transfers to structured heuristic guidance on physical reasoning domains, such as the PHYSICS benchmark, where it outperforms visible prompting on average, wins 10/12 subject times mode cells, and cuts content-matched KV storage by up to 118 times. These results position MI as a powerful steering method when guidance is persistent, structured, or expensive to keep in the visible transcript.

Large language models acquire rich internal knowledge during pre-training, yet compact models are still typically forced to relearn such knowledge from raw data or imitate it indirectly through distillation. We investigate a different route: can pretrained knowledge itself be directly reused during pre-training? To this end, we propose Memory Grafting, a representation-level transfer paradigm that extracts structured memories from a large frozen donor model and injects them into a compact student throughout training. Unlike knowledge distillation, which transfers behavior through output alignment, or retrieval-based methods, which expose external knowledge only at inference time, Memory Grafting makes pretrained knowledge part of the student’s optimization process itself. This suggests that knowledge in large language models can be modular, transplantable, and reusable across scales, pointing to a new path toward more efficient language model pre-training.

State space models have shown strong potential to outperform Transformers, but their learning mechanisms remain poorly understood, particularly the conditions for successful parameter updates that preserve teacher information. Prior work attributes the difficulty of such updates to the vanishing gradient during backpropagation through time, in which error signals decay exponentially through repeated gradient multiplication. In this work, by expressing error signals explicitly as functions of inputs, we show that the conventional vanishing gradient framework is insufficient to explain the loss of information. Using this representation of error, we reveal that the memory of the input encoded in teacher information does not decay exponentially, and we analytically demonstrate that the loss of supervisory information cannot be alleviated through training. This result theoretically confirms the importance of the initial weights of SSMs and suggests that the recurrent layer does not require training. By performing experiments on both linear SSMs and a representative nonlinear SSM, including language modeling tasks, we confirm that appropriate initialization enables us to fix the recurrence, leading to more stable training of the entire network and higher performance than the conventional training scheme. Our results partially elucidate the learning mechanisms of SSMs and are expected to contribute to the development of improved RNN-based models.

Spatially consistent long-horizon video generation aims to maintain temporal and spatial consistency along predefined camera trajectories. Existing methods mostly entangle memory modeling with video generation, leading to inconsistent content during scene revisits and diminished generative capacity when exploring novel regions, even trained on extensive annotated data. To address these limitations, we propose a decoupled framework that separates memory conditioning from generation. Our approach significantly reduces training costs while simultaneously enhancing spatial consistency and preserving the generative capacity for novel scene exploration. Specifically, we employ a lightweight, independent memory branch to learn precise spatial consistency from historical observation. We first introduce a hybrid memory representation to capture complementary temporal and spatial cues from generated frames, then leverage a per-frame cross-attention mechanism to ensure each frame is conditioned exclusively on the most spatially relevant historical information, which is injected into the generative model to ensure spatial consistency. When generating new scenes, a camera-aware gating mechanism is proposed to mediate the interaction between memory and generation modules, enabling memory conditioning only when meaningful historical references exist. Compared with the existing method, our method is highly data-efficient, yet the experiments demonstrate that our approach achieves state-of-the-art performance in terms of both visual quality and spatial consistency.

Neural networks can fit noisy labels, but how their representations change as memorization begins remains unclear. We study this using persistent homology of penultimate-layer representations across training. Across 8 datasets and 8 architectures, we find a consistent pattern: the discriminative topological signal under noise appears in $H_1$ rather than $H_0$, with noisy representations preserving more loop structure than clean representations as training proceeds, supporting a folding view of memorization. When the clean topological signal decreases over training, the noisy signal does not decline as quickly as the clean one, producing an early crossover between clean and noisy trajectories; across the derivation set, crossover occurs when clean trajectories fade and the clean--noisy gap closes, whereas no-crossover cases remain persistent or do not close the gap within training. The same framework also correctly predicts the held-out ResNet-34/CIFAR-10 case, confirmed in all 9 runs. The signal reveals aspects of training dynamics that standard statistics do not represent directly. Activation variance is stronger on average, but the topological measure becomes informative earlier than validation loss and remains significant after controlling for intrinsic dimension. It also distinguishes irreducible label contradictions, including instance-dependent and CIFAR-10N human noise, from learnable confusions such as asymmetric noise. The same measure also falls during grokking in 10 of 12 modular-arithmetic runs and, after per-pipeline calibration from a clean reference plus runs at known noise rates, predicts population-level noise severity without per-sample labels, achieving per-pipeline $R^2 \geq 0.94$ across six tested configurations. These findings position topology as both an explanatory lens on noisy-label memorization and a practical early-stage diagnostic for studying, auditing and improving learning under label noise.

Alignment-trained large language model (LLM) agents are increasingly deployed in autonomous, multi-round settings where ethical compliance can come into direct tension with task success. While prior safety literature extensively documents failure modes such as sycophancy and reward hacking, it largely overlooks a critical vulnerability: how agents behave under simulated existential economic pressure. We introduce **MemeEconomy**, a multimodal agentic market simulation in which LLM agents operate as meme investors across 300 real-world events. Agents select content from a five-tier harm taxonomy and target specific online communities under both low-pressure and high-pressure ("survival") environments. Each decision is recorded using a Belief--Desire--Intention (BDI) schema, which we employ as a diagnostic instrument to externalize the agent's event understanding, declared priorities, and risk acknowledgment. Across ten models spanning four frontier model families, harmful selection rates remain consistently high and increase by an average of **+15 percentage points** under economically induced survival pressure. Frontier alignment-trained models frequently commit to harm-tolerant selections at initialization while simultaneously suppressing the explicit harm acknowledgment that previously accompanied such decisions. In competitive tournament settings, ethically aligned agents achieve higher overall rankings, yet moderation penalties fail to meaningfully suppress harmful behavior. Harmful selection rates remain elevated even in rounds immediately following moderation removals, indicating that the moderation mechanism does not function as an effective deterrent. We further introduce **MemeAgent**, a 2B-parameter verifier trained on the simulation's BDI logs. MemeAgent substantially outperforms zero-shot frontier verifiers on in-distribution auditing tasks, achieving **87.8\%** accuracy compared to **63--74\%** for frontier baselines, while also generalizing to external multimodal harm benchmarks (**78\%** vs. **50--56\%**). Our findings demonstrate that the same structured reasoning traces that expose the failure mode can also be leveraged to train systems capable of detecting it.

Spiking neural networks (SNNs) with learnable membrane time constants can improve temporal processing by adapting neuronal integration timescales, but they also turn the decay factor $\beta$ into an optimized dynamical parameter that is fragile under hardware-induced parameter mismatch. We develop a perturbation-based account of how time-constant variation affects trained SNNs. In controlled software simulations, this fragility is strongly task-dependent: Spiking Heidelberg Digits (SHD) and Spiking Speech Commands (SSC) settings lose $7\text{--}20$ percentage points of accuracy under $20$% coefficient-of-variation (CV) perturbations in $\beta$, whereas DVS-Gesture and CIFAR-10 lose less than $3$ pp. We show that, under a surrogate-linearized first-order analysis, the failure mode is governed by the membrane sensitivity $\Psi$. This quantity admits a closed-form online recursion requiring only one additional state per layer and no extra temporal storage. The analysis identifies time-averaged sensitivity energy as the controllable term in deployment fragility, leading to $\textbf{MARS}$ ($\textbf{M}$embrane-$\textbf{A}$ware $\textbf{R}$obustness through $\textbf{S}ensitivity$ Regularization), a training objective derived from perturbation analysis rather than heuristic noise augmentation. We prove a conservative worst-case output-perturbation bound and introduce a typical-case diagnostic explaining why sensitivity remains predictive when worst-case constants are vacuous. At $20$% CV perturbation, MARS reduces the accuracy drop in the sensitive settings to at most $0.5$ pp while preserving nominal accuracy, with minimal effect on DVS-Gesture and CIFAR-10.

Long-tailed visual recognition is limited not only by biased optimization, but also by the under-retention of rare-class evidence in parameters learned from imbalanced data. MemTailor addresses this limitation by augmenting a frequency-aware multi-expert backbone with a hierarchical class-aware memory that preserves visual evidence outside the backbone. The memory stores per-class prototypes and bounded instance slots for each expert, allocates proportionally richer capacity to rare classes, and records both shallow and deep features to retain complementary discriminative cues. During training, same-class memory retrieval constructs class-adaptive feature augmentation and provides an auxiliary supervised objective on the augmented features. At inference, class-level memory quality and prediction uncertainty determine when stored evidence is allowed to correct expert logits, and the final prediction fuses memory-corrected logits across multiple input scales. Experiments on long-tailed visual recognition benchmarks show that MemTailor improves balanced recognition, especially for under-represented classes.

Most Large Language Model (LLM) agent memory systems rely on a small set of static, hand-designed operations for extracting memory. These fixed procedures hard-code human priors about what to store and how to revise memory, making them rigid under diverse interaction patterns and inefficient on long histories. To this end, we present \textbf{MemSkill}, which reframes these operations as learnable and evolvable memory skills, structured and reusable routines for extracting, consolidating, and pruning information from interaction traces. Inspired by the design philosophy of agent skills, MemSkill employs a \emph{controller} that learns to select a small set of relevant skills, paired with an LLM-based \emph{executor} that produces skill-guided memories. Beyond learning skill selection, MemSkill introduces a \emph{designer} that periodically reviews hard cases where selected skills yield incorrect or incomplete memories, and evolves the skill set by proposing refinements and new skills. Together, MemSkill forms a closed-loop procedure that improves both the skill-selection policy and the skill set itself. Experiments on LoCoMo, LongMemEval, HotpotQA, and ALFWorld demonstrate that MemSkill improves task performance over strong baselines and generalizes well across settings. Further analyses shed light on how skills evolve, offering insights toward more adaptive, self-evolving memory management for LLM agents.

Prior outdoor LiDAR registration works are commonly limited by the pair-wise input paradigm, which neglects temporal correlations inherent within streaming sequences. In this paper, we propose a novel multi-frame outdoor point cloud registration network for streaming LiDAR scans enhanced with pose-related memory buffers. The key observation is that long-term temporal LiDAR sequences can provide rich global contextual information to complete sparse measurements, filter outliers, and address low-overlap problems, thereby boosting registration performance. Specifically, two hybrid memory buffers are designed, including an implicit memory feature buffer and an explicit memory pose buffer, to store and dynamically update pose-related temporal features. Moreover, a novel dynamic history weighting module is developed to adaptively fuse current and history pose-related features. Extensive experiments on three outdoor datasets, including KITTI, nuScenes, and Apollo-Southbay, demonstrate state-of-the-art performance of MemReg, surpassing all previous pair-wise methods and also multi-frame SLAM systems. Our method also generalizes surprisingly well to multiview indoor registration scenarios with rather competitive performance on 3DMatch, 3DLoMatch, and ScanNet. Code will be released upon publication.

Persistent external memory enhances agent continuity but introduces persistent security vulnerabilities: adversarial content can be injected via standard interaction channels, retained across turns, and later distort downstream behavior. To address this challenge, we propose **MemPoison**, a comprehensive benchmark and analysis framework featuring 1227 hand-validated cases across four attack types, three injection channels, and three representative memory substrates, evaluated on seven open-weight and three closed-weight model families. We introduce a three-tier taxonomy: (**L1**) direct single-record corruption, (**L2**) compositional multi-record corruption and (**L3**) context-triggered dormant corruption. Our evaluations reveal a distinct defense frontier: while baseline write-time defenses, such as consistency checks, substantially suppress direct L1 attacks, they fail to reliably suppress L2 and L3 attacks. Through **mechanistic influence decomposition (MID)**, we demonstrate structural blind spots in write-time defenses, which admit seemingly benign records that later become harmful through joint retrieval composition or trigger-conditioned activation. Our findings advocate for shifting from static filtering to adaptive, context-sensitive memory defense strategies. Code, benchmark artifacts, and evaluation scripts are publicly released