NeurIPS 2026 Papers

Fine-tuning LLMs on narrow harmful datasets can induce Emergent Misalignment (EM), where models exhibit misaligned behavior far beyond the fine-tuning distribution. We argue that emergent misalignment can be better understood as a data-mediated transfer phenomenon: harmful fine-tuning examples do not induce uniform behavioral spillover, but interact with the structural properties of the dataset and the difficulty of the tasks relative to the model. Across our experiments, we find that misalignment appears more readily when fine-tuning and evaluation prompts share similar underlying functional structure, when prompts leave more room for coherent harmful completions, and when the target behavior has been more reliably learned by the model. The training pipeline itself also matters: pretraining composition shapes later misalignment. We further study Subliminal Learning (SL), where misalignment is transmitted by fine-tuning on seemingly benign data generated by a harmful teacher. Moving beyond the standard SFT setting, we for the first time compare this transfer under off-policy and on-policy distillation as well, allowing us to separate the roles of the teacher guidance and the training data distribution in transmitting misalignment. Together, these results argue for a data-centric view: Emergent/subliminal misalignment should not be treated as a simple consequence of isolated harmful fine-tuning examples, but as the result of interactions between fine-tuning data structure, pretraining distributions, and training channels.

Foundation models trained on large unlabeled corpora develop emergent capabilities — tasks that go beyond the training objective. We extend this paradigm to multimodal biology with Bonbon, a foundation model for molecular interactions trained on approximately 3.2 trillion tokens of paired protein-ligand sequences. From a single self-supervised objective, Bonbon exhibits four zero-shot emergent capabilities at hierarchical levels of resolution — functional (mechanism of action), residue (binding site localization, including orthosteric/allosteric discrimination), bond (covalent versus non-covalent engagement), and atom (active moiety identification at single-bond resolution). All four capabilities generalize to out-of-distribution proteins. Bonbon was trained on protein-ligand sequences alone, without any task-specific supervision and without structural coordinates. We conducted controlled scaling experiments across four model scales from 42M to 1.6B parameters; every capability evaluated shows a sharp jump at the largest scale on continuous metrics, even as pretraining loss decreases smoothly. This demonstrates that loss-based scaling laws alone are insufficient to predict when zero-shot emergent capabilities become available. The capabilities reported here emerge after training at a 1:2,000 parameter-to-token ratio to accommodate the sparsity of paired biological interaction data. The same frozen representations achieve state-of-the-art binding and affinity predictions at five orders of magnitude greater speed than structure-based methods. In prospective wet-lab validation, Bonbon achieves a 70% hit rate on de novo compounds, discovering novel chemotypes not previously documented as inhibitors for those targets. All of the tested compounds had less than 60% structural similarity to molecules in the training corpus.

Kolmogorov–Arnold Networks (KANs) expand each input coordinate through a fixed basis family, yet the optimization role of this basis lifting remains poorly understood. We identify basis lifting as a structural source of ill-conditioning in KANs and provide a mechanism-level analysis linking lifted feature geometry to convergence, accuracy, and seed sensitivity. Our analysis studies the evolution of lifted empirical Gram matrices under stochastic gradient descent (SGD) and shows that fixed basis lifting induces and preserves near-rank deficiency through eigenvalue and determinant drift. It further concentrates feature energy into a smaller number of directions, amplifying dominant eigenvalues and worsening condition numbers. These results show that basis selection is not merely an expressivity choice, but also an optimization and conditioning choice. Motivated by this mechanism, we introduce T-KAN, a learnable basis-coordinate transformation that aligns basis directions before the learned linear map. It reshapes lifted Gram geometries while preserving the expressivity and introducing negligible parameter overhead. In controlled small-to-moderate KAN regimes, T-KAN often improves conditioning and optimization with B-spline, Chebyshev, and RBF KANs.

Classifier-free guidance (CFG) is the de facto standard for conditional sampling in diffusion models, yet it often reduces sample diversity. Using tools from statistical physics, we analyze the emergence of generative distortions induced by CFG, namely the mismatch between the CFG sampling distribution and the true conditional distribution. We study this phenomenon in analytically tractable settings with exact score functions, characterizing its dependence on data dimensionality and the number of classes. For high-dimensional Gaussian mixtures, we use dynamic mean-field theory to show that distortions arise when the number of classes scales exponentially with the data dimension, whereas they vanish in the sub-exponential regime due to a dynamical phase transition. We further prove that, in the infinite-class limit, distortions remain unavoidable regardless of dimensionality because of the increasing density of classes. Finally, we show that standard CFG schedules cannot prevent variance shrinkage, and we propose a theoretically grounded guidance schedule incorporating a negative-guidance window that improves both class separability and sample diversity in real-world latent diffusion models.

Recent advances in 3D object generation have enabled the creation of high-fidelity 3D assets from 2D images. This *2D-to-3D* capability offers a promising path for alleviating 3D data scarcity, which remains a key bottleneck for scalable simulation in robotics and AR/VR applications. However, existing 3D generation methods primarily emphasize visual plausibility and geometric fidelity, with limited consideration of object-level interaction and structural understanding that are critical for downstream embodied AI tasks. To bridge this gap, we introduce **EmbodiedObject**, a unified framework for embodied 3D object understanding built upon latent representations produced by modern *2D-to-3D* generative models such as SAM3D. We first systematically analyze the part-level locality of these latent representations against existing unified 3D point encoders and show that they naturally preserve rich structural semantics suitable for object understanding. Motivated by this observation, we then design a unified DiT-style decoder that operates directly on the latent representation and supports multiple object-level tasks, including open-vocabulary 3D affordance prediction, 3D part segmentation, and 3D articulation estimation. Given a real-world 2D image containing an object of interest, **EmbodiedObject** extends *2D-to-3D* generation beyond visual reconstruction by jointly reasoning about affordance, articulation, and part semantics. This transforms generated 3D assets into actionable object representations suitable for downstream embodied AI applications. Our source-code will be open-sourced.

Modern AI is opening the door to collective decision-making in which participants express their views as free-form text rather than voting on a fixed set of candidates. A natural idea is to embed these opinions in a vector space so that the substantial literature on facility location problems and fair clustering can be brought to bear. But standard text embeddings measure semantic similarity, whereas distances in facility location problems and fair clustering require what we call preferential similarity: a participant's agreement with a piece of text should be inversely related to their distance from it. Off-the-shelf embeddings inherit a coarse preference signal through a correlation between semantic and preferential similarity, but fail to capture preferences when the correlation breaks. We formalize this as an invariance problem: text embedding models encode both a preference-relevant signal (stance and values) and semantic nuisance (style and wording), and the two are observationally correlated, so a geometry that relies on nuisance can appear preference-correct even when it is not. We show that synthetic training data designed to break this correlation provably shifts the optimal scorer away from nuisance-dominated cosine and significantly improves preference prediction across 11 online deliberation datasets.

AI-enabled Cyber-Physical Systems (CPS) are highly vulnerable to adversarial and anomalous inputs, where small perturbations can induce cascading errors and unsafe control actions. Existing approaches—such as rule-based filtering, training-time regularization, or diffusion-based reconstruction—either operate outside the model or lack mechanisms to incorporate formal security specifications into the prediction process. In this paper, we develop the first work toward embedding security properties directly into AI-enabled CPS, enabling predictive models to enforce system-level constraints during inference rather than relying on external defenses. We introduce a logic-conditioned bi-stage diffusion framework that integrates Signal Temporal Logic (STL) specifications into forecasting. STL serves as a first-class conditioning signal that guides both an input repair stage and an output refinement stage, allowing the model to jointly mitigate adversarial perturbations and enforce desired temporal behaviors to satisfy security-critical properties. We evaluate our approach on a real-world multivariate CPS forecasting task under both physical sensor and cyber attacks. Our method improves robustness and specification compliance, degrades more gracefully as attack strength increases, and generalizes better to unseen attacks. Ablation studies further show that embedding logical security properties yields gains unattainable by reconstruction-based methods alone, highlighting a new direction for integrating formal methods with generative models in secure CPS.

Tabular foundation models achieve strong accuracy on choice prediction tasks, but their predictions often violate the economic logic those tasks require: raising a price can increase predicted demand, implied willingness-to-pay estimates are frequently negative or implausible, and unavailable alternatives receive nonzero probability. We propose a two-stage adapter that takes a foundation model's predicted choice probabilities as a precomputed feature and embeds them inside a multinomial logit's utility. In Stage 1, we fit the multinomial logit's structural coefficients by maximum likelihood with sign constraints; in Stage 2, we freeze those coefficients and fit a small neural correction operating on the foundation model's predictions. We prove that this composition exactly preserves the multinomial logit's marginal rate of substitution, so analytically computable value-of-time becomes a mathematical guarantee rather than an empirical accident. Across three datasets and two foundation models, the adapter gains 6.4 percentage points (pp) of test accuracy on average over the multinomial logit and up to 12.8 pp, maintains 100\% cost monotonicity, and produces values of time within the published transportation-economics range on the transportation datasets. Performance degrades gracefully under foundation-model context restriction, retaining at least 6 pp of accuracy gain even at 10\% of the original foundation-model context.

Embedded devices from wildlife monitoring stations to clinical wearables require local AI inference due to latency, communication, or privacy constraints. Optimizing models for heterogeneous microcontrollers (MCUs) requires simultaneously satisfying hard physical constraints on memory, power, and temperature while preserving accuracy, a multidimensional optimization that is today performed manually by experts. We ask whether an LLM agent can autonomously navigate this complex, multi-turn pipeline guided by real hardware feedback, and introduce a hardware-in-the-loop agent arena in which the agent iteratively refines both model and firmware---compiling, flashing, and measuring on real hardware---to enable iterative optimization. Frontier models, including Claude Opus 4.7 and Gemini 3.1 Pro, fail entirely without hardware feedback (0\% deployment success), whereas our closed-loop formulation achieves the first successful deployment within three iterations and can surpass human expert results within seven. This agentic co-optimization achieves 250× compression for vision models with <3.3\% accuracy loss and 400× for audio with <6\% Feature Error Rate loss, enabling battery-free operation on a commercial MCU via solar harvesting. We demonstrate practical impact in two real-world systems: an elk-detection camera trap (96.7\% accuracy) and a phonetic-transcription wearable (8.44\% FER) for child development research.

Large Language Models (LLMs) have demonstrated remarkable capabilities in Information Extraction (IE) tasks; however, their performance in zero-shot Named Entity Recognition (NER) remains suboptimal. We argue that this limitation stems not from intrinsic defects within the models, but rather from the failure of existing instruction paradigms to effectively elicit their latent capabilities. Drawing inspiration from cognitive science, we propose an instruction optimization framework termed Instruction Semantic Elaboration (ISE), which fully elicits models' latent NER capabilities by mimicking the semantic elaboration process—specifically by contrasting related concepts, decomposing constituent elements, and illustrating with concrete examples. We evaluate our framework under a zero-shot setting across four domain-specific datasets. Results demonstrate that ISE effectively and stably unlocks the intrinsic NER capabilities of LLMs, achieving a 10.99\% improvement in F1 score over initial instructions and outperforming strong baselines by 6.89\%. Furthermore, the framework generalizes well across model architectures and parameter scales, while remaining robust to the quality of initial instructions. This study offers a novel perspective on activating NER capabilities in LLMs, facilitating their efficient and stable deployment in real-world scenarios.

Hour-long videos, streaming feeds, and movie-length content are pushing vision-language models (VLMs) past their breaking point—the standard recipe of encoding each frame into visual tokens yields sequences that grow linearly with video length, causing prohibitive memory consumption and inference latency. Existing approaches, such as token reduction and keyframe selection, attempt to shorten the token sequence but leave the $O(n)$ scaling intact, failing to resolve the fundamental bottleneck. We propose \model, \emph{a strikingly simple, training-free frame packing framework that compresses arbitrarily long videos into a fixed-size visual representation at constant memory cost.} \model requires no fine-tuning, no architectural modification, and no auxiliary models—it plugs directly into off-the-shelf VLMs at inference time, yet consistently surpasses more elaborate baselines. Drawing inspiration from FramePack in generation literature, we propose a new content-aware compression strategy that operates under a fixed token budget for visual understanding. Specifically, \model proceeds in two steps: (1) scoring each frame by its semantic similarity to the text prompt and ranking frames in a diversity-driven manner, and (2) packing the selected frames at varying resolutions into a fixed-length context window. This design concentrates the memory budget on frames that are both instruction-relevant and visually diverse. While in theory \model can pack arbitrarily many frames into the fixed budget, we identify an empirical instantiation that generalizes robustly across different settings. Extensive experiments on LongVideoBench, MLVU, and Video-MME show that \model consistently outperforms full-resolution baselines as well as prior token reduction and keyframe selection methods, across diverse model families, parameter scales, and frame budgets. Notably, on Qwen3.5-35B-A3B with a 128-frame budget, \model yields gains of \textbf{+12.9\%, +10.6\%, and +9.7\%} over the official model on the three benchmarks, respectively.

Vision-Language-Action (VLA) models have emerged as a powerful paradigm for generalist robotic control. However, their high computational cost and limited control frequency hinder real-time robotic manipulation, especially when large vision-language backbones and iterative action heads are executed at every control step. Existing VLA acceleration methods often optimize individual components or rely on fixed acceleration rules, treating different control steps with largely fixed computation and overlooking the non-uniform reasoning demands of sequential embodied control. Inspired by human motor control, where cognitive and feedback resources concentrate on goal-sensitive stages, we argue that VLA models should learn when to invest full computation and when to reuse prior computation. To this end, we propose ElegantVLA, a plug-in phase-adaptive inference framework that accelerates VLA models through intra-model dynamic compute scheduling. ElegantVLA introduces a lightweight scheduler that observes temporal representation similarity, robot-motion cues, and episode progress to jointly allocate computation across the vision encoder, LLM, and action head. For perception-language reasoning, the scheduler selects a five-level Vision--LLM compute mode, from full recomputation to multi-step temporal reuse, based on visual-language representation stability. For action generation, it selects a three-level denoising mode, reusing intermediate denoising states during stable motion while preserving full refinement for goal-sensitive stages. By coordinating these decisions, ElegantVLA provides a general acceleration framework for modern VLA pipelines with explicit action-generation modules, without modifying or retraining the base model. Extensive experiments on GR00T, CogACT, and real-world tasks show that ElegantVLA preserves or improves task success while substantially accelerating inference. On GR00T, it achieves up to 2.55$\times$ average speedup. On CogACT, it delivers a 3.77$\times$ average speedup. In GR00T-based real-world experiments across six tasks, it reduces computation by 2.18$\times$ and increases control frequency from 13.8 Hz to 26.3 Hz.

Learned representations (or embeddings) are the foundation of modern machine learning systems, yet they are typically trained as fixed-size embeddings, without accounting for varying downstream resource or task constraints. Matryoshka Representation Learning (MRL) addresses this limitation by learning nested representations, but requires costly end-to-end retraining and does not support post hoc expansion, while Contrastive Sparse Representations (CSR) reduce training overhead via lightweight adaptors but operate at a 4x larger dimensionality. Additionally, all prior works restrict their framework towards compression only. In this work, we introduce \textbf{H}yperbolic \textbf{E}lastic \textbf{R}epresentation \textbf{L}earning (HERL), a framework for learning dimension-adaptive embeddings (compression and expansion) through a lightweight adaptor without retraining the base encoder. Our key idea is to exploit the exponential capacity of hyperbolic geometry by learning a non-linear mapping from the encoder space to hyperbolic space. We train a simple MLP to downsample the representations before projecting them onto a Lorentzian and learn adaptive representations that respect the Lorentzian geometry. Our approach is model-agnostic and effective in both supervised and unsupervised settings. Through extensive experiments across image and text modalities, we show that HERL preserves downstream performance across a wide range of embedding sizes, exhibits smooth interpolation between dimensions, and achieves these benefits at a fraction of the computational cost.

Touch is fundamental to dexterous manipulation, yet most egocentric human data increasingly used for robot learning lacks tactile information. Directly collecting large-scale tactile data is challenging due to sensor limitations, while human video data is abundant, contact-rich, and easily scalable. This motivates a natural question: can tactile signals be inferred purely from vision? To address this, we introduce **EgoTac**, a generalizable model that predicts rich **tac**tile information directly from **ego**centric human videos. EgoTac is trained on a unified corpus of over 5.7M image-tactile pairs, covering both continuous force measurements and binary contacts. By learning from this diverse dataset, EgoTac captures nuanced touch dynamics across varied interactions. Experiments demonstrate strong performance: in-domain prediction achieves an average force error below 0.06N. On out-of-domain contact prediction benchmarks, EgoTac consistently outperforms the state-of-the-art contact estimator. It also captures the rise and fall patterns of real tactile data and enables zero-shot predictions on unconstrained real-world videos. Scaling analyses further reveal that both data diversity and volume improve performance steadily. Overall, EgoTac provides a scalable pathway to extract tactile priors from egocentric human videos, enabling broadly applicable tactile-aware robot learning.

We introduce EgoSurgHands, an egocentric 3D hand-pose dataset and benchmark for open-surgery training tasks captured with Meta Project Aria glasses on standard surgical training pads. To our knowledge, EgoSurgHands is the first public benchmark for surgical training that is simultaneously egocentric, sensor-grounded, and human-validated. EgoSurgHands contains 41,909 training hand-instance rows across 55 procedure recordings and 9,476 inter-annotator-agreement (IAA)-validated rows across 13 held-out procedure recordings from 4 wearers, 13 distinct tasks, 3 glove colors, and 3 stages of training. Each frame includes Aria intrinsics, SLAM extrinsics, MPS (Machine Perception Services) landmarks, per-joint confidence, timestamps, and eye gaze. Validation 3D keypoints are initialized from Aria MPS and independently corrected by four expert annotators under a multi-rater IAA workflow, yielding evaluation ground truth independent of any tested model's training signal. Using EgoSurgHands, we benchmark six state-of-the-art hand reconstructors under zero-shot transfer: HaMeR, WiLoR, WildHands, JointTransformer, SimpleHand, and MeshGraphormer. Ego-pretrained models consistently outperform non-egocentric baselines: WildHands improves over the non-ego HaMeR reference by 4.51 mm PA-MPJPE, while JointTransformer achieves the best zero-shot result at 14.95 mm PA-MPJPE. These results show that generalist egocentric pretraining transfers to surgical egocentric tasks, but also reveal a substantial remaining surgical-domain gap. As a lightweight diagnostic adaptation experiment, a 58K-parameter two-headed (pose + translation) residual corrector further reduces error on frozen backbones, reaching 10.40 mm PA-MPJPE and 75.8 mm absolute MPJPE on WildHands ($-$49.2\% and $-$85.8\% vs zero-shot). We release the dataset, benchmark code, checkpoints, and multi-rater annotation framework at [URL withheld for double-blind review].

Continuous episodic memory is a core capability for autonomous agents operating in dynamic, real-world environments, yet current streaming video benchmarks provide limited tools for diagnosing what models remember and for how long. We introduce Egostream, a diagnostic benchmark for streaming episodic memory evaluation in egocentric vision. \egostream organizes 2,250 curated questions along seven cognitive dimensions: detail, spatial, temporal, event, social, causal, and prospective memory. We introduce the Answer Validity Window (AVW), which specifies the temporal span an answer remains valid as the observed scene evolves. This allows us to expand the questions into 8,528 recall-conditioned evaluations, enabling controlled testing from instant to ultra-long-term recall while separating genuine model forgetting from natural world-state changes. We rigorously establish baseline performance through a unified streaming MLLM framework that compares several state-of-the-art memory-management mechanisms, covering sliding windows, attention sinks, KV-cache pruning, merging, and offloading. Experiments within a unified Qwen3-VL backbone reveal that comparable aggregate accuracies mask starkly different memory profiles. For instance, token pruning preserves fine-grained details and temporal structure significantly better than token merging, while quantized offloading rescues ultra-long-term recall. Ultimately, all mechanisms operate well below real-time (>1s per frame), and top performing methods ceil at about 45\% accuracy, exposing critical gaps in current architectures. Egostream provides the diagnostic testbed needed to close these gaps.

The complex interactive dependencies between human behavior and 3D scenes make accurately predicting human intent a major challenge for human motion prediction (HMP). While recent studies use human gaze to infer intent, gaze signals suffer from ambiguity, drift, and a reliance on eye-tracking hardware. Research in cognitive science indicates that egocentric views inherently encode potential interactions, while head motion serves as a crucial precursor in motion planning. Motivated by this, we propose EgoHMP, a novel framework that uses egocentric views and head motions as robust carriers of interaction intent. By translating this intent into an envisioning of future interactions, EgoHMP achieves precise HMP in 3D scenes. EgoHMP first employs a vision-motion alignment encoder to align the visual and motion feature spaces. Based on these aligned features, an egocentric intent predictor with a modality-aware modulation mechanism adaptively suppresses invalid interaction cues to predict an accurate interaction probability map. Furthermore, to enhance the stability and realism of the generated motions, we propose a contact-aware motion decoder. Specifically, inspired by the human cognitive process of planning trajectories prior to action execution, we first predict a global trajectory to serve as a self-prompt, effectively mitigating the optimization conflict in diffusion models. Subsequently, we introduce a multi-stage denoiser, which relieves motion artifacts by incorporating human contact modeling through a multi-stage optimization strategy. Finally, a GCN-based motion decoder is employed to synthesize physically plausible and semantically consistent motions. Extensive experiments demonstrate that EgoHMP achieves state-of-the-art performance.

As AI agents increasingly operate in open, real-world environments, they require a deep synergy of multimodal perception, tool invocation with multi-hop reasoning, and dynamic interaction with users. However, existing benchmarks fail to jointly evaluate these capabilities due to challenges in designing strictly coupled multi-capability tasks, simulating natural and task-constrained user feedback, and ensuring objective evaluation of dynamic interaction. To bridge this gap, we introduce EgoBench, the first interactive multimodal benchmark for tool-using agents. EgoBench comprises 1,590 egocentric-video-grounded tasks covering five daily scenarios, along with a user-agent-tool interactive environment for evaluation. We implement a three-stage synergistic pipeline through which each task is designed to enforce the joint application of visual perception and tool-augmented multi-hop reasoning. We additionally develop a multi-agent simulated user to evaluate agents' interaction capabilities, which generates high-fidelity, task-aligned responses to agents. Furthermore, we establish a deterministic joint validation framework that guarantees objective assessment through process-based and result-based equivalence. Benchmarking eight video-MLLM agents on EgoBench reveals a severe performance ceiling: the best-performing model achieves only a 34.95% average Joint Success Rate across the three interaction modes. Finally, we conduct a multi-dimensional error analysis to disentangle failure modes, exposing capability bottlenecks for advancing future AI agents.

Children acquire language with remarkable speed and robustness from limited linguistic input in ways that surpass today's best large language models. Recent research suggests this efficiency stems from learning through multimodal sensory experiences where semantic alignment between what children see and hear grounds language in the external world. However, vision-language models (VLMs) trained on curated web data fail to generalize to the sparse, weakly-aligned egocentric streams produced by wearable devices, embodied agents, and infant head-cams—and no fixed evaluation pipeline exists for measuring progress on this regime. We train VLMs on datasets with varying degrees of semantic alignment between visual and linguistic inputs, including naturalistic infant egocentric videos, and evaluate them with a comprehensive suite spanning multimodal language grounding and unimodal vision and language tasks. At the core of this suite is _Machine-DevBench_, a corpus-grounded benchmark of lexical and grammatical competence, automatically generated from the model's training vocabulary across logarithmic frequency bins to eliminate the train/eval mismatch and low statistical power of prior developmental benchmarks. Our results show that current VLM paradigms hinge on the tight semantic alignment of curated data and fail to exploit the weakly-aligned signal that dominates naturalistic egocentric input—the very regime in which humans thrive. To motivate progress, we introduce the EgoBabyVLM Challenge to drive the development of models capable of grounded language learning from the kind of naturalistic data that human infants experience.