NeurIPS 2026 Papers

Model merging has emerged as a promising paradigm for composing large language models directly in weight space, enabling training-free integration of specialized models. However, existing methods rely on parameter-space averaging that systematically induces generation dysregulation---a spectrum of failure modes ranging from structural repetition and termination failure to circular reasoning, and at the extreme, full semantic collapse into incoherent symbols. Even when source models exhibit near-zero dysregulation, existing methods introduce it at $>$97\% (14B/32B scales), collapsing reasoning benchmarks by up to 70 points. We propose Sparse Complementary Fusion with Reverse KL (\textbf{SCF-RKL}), a data-free merging framework that selects complementary parameters via reverse KL divergence on parameter-group proxy distributions. This structure-preserving, sparsity-inducing design maintains proxy-space geometry---theoretically motivated via entropy and subspace bounds---and empirically suppresses generation dysregulation while integrating new capabilities. Extensive experiments on 24 benchmarks across 7B--32B models spanning reasoning, instruction following, safety, and vision demonstrate that SCF-RKL achieves the best overall performance across scales while maintaining near-zero dysregulation ($<$1\%) and strong generalization.

Multimodal alignment under scarce paired supervision aims to align independently pretrained unimodal encoders using limited image--text pairs, offering a practical alternative when large-scale joint pretraining is infeasible. Such scenarios call for lightweight objectives that go beyond local correspondences and capture distributional mismatch between heterogeneous embeddings. Existing methods often derive supervision from limited pairs through contrastive objectives or geometry-preserving regularization, encouraging instance-level correspondence or structural consistency but providing limited direct supervision on global cross-modal mismatch. This motivates a distribution-level objective that complements paired supervision by comparing image and text embedding distributions. In this paper, we propose $\textbf{Spec}$tral Characteristic Distribution $\textbf{Align}$ment ($\textbf{SpecAlign}$), a lightweight spectral alignment framework that reduces cross-modal mismatch by comparing image and text characteristic functions over learnable spectral queries. Since characteristic functions uniquely determine probability distributions, SpecAlign provides a principled signal for capturing multi-scale discrepancies beyond pointwise alignment. To obtain informative and stable queries, SpecAlign employs a Structured Direction--Radius Sampler (SDRS) that decouples direction and radius modeling. Extensive evaluations on transfer, retrieval, robustness, and representation analyses demonstrate consistent gains, validating spectral distribution-level supervision for data-efficient multimodal alignment.

In multimodal classification, late-fusion approaches classify concatenated modality-specific features extracted by unimodal neural networks. When modality imbalance is pronounced, various regularization techniques have been proposed to balance the learning process and overcome the inferior performance of late-fusion networks. In contrast, this work demonstrates that multimodal data can be effectively classified without any explicit modality fusion, using deep ensembles of unimodal networks. We systematically compare deep ensembles to late-fusion networks at equal parameter count and show that ensembles consistently outperform state-of-the-art late-fusion methods designed to address modality imbalance. This advantage also holds over intermediate-fusion techniques we evaluated and over hybrid methods that combine unimodal and multimodal predictions. We propose and empirically validate a method for selecting the number of models per modality in an ensemble, avoiding computationally expensive exhaustive search. Under extreme modality imbalance and small ensemble sizes, the heuristic indicates that ensembles of unimodal models trained solely on the stronger modality are preferable; as the ensemble scales up, incorporating models from the weaker modality becomes beneficial. Both predictions align with our empirical findings. To systematically explore the challenges of optimizing multimodal models, we propose a synthetic multimodal framework that allows control over both the number of modalities and their predictive strength; our findings are consistent across synthetic and real-world datasets. Finally, by fitting scaling laws to bimodal datasets, we estimate the asymptotic performance of ensembles.

We introduce a new algorithm to sample from a probability density $\pi \propto e^{-V}$ with $V(x): \mathbb{R}^d \rightarrow \mathbb{R}$ a strongly multi-modal potential. Such distributions are challenging to sample from using standard gradient based methods, as exploration typically relies on inefficient diffusions. Inspired by connections between Schrodinger operators, Langevin diffusions and Morse theory, we develop a particle based method that exploits fast relaxation to transition pathways of the so-called Witten PDE on 1-forms (vector fields). Once pathways joining modes are sufficiently well explored, the method relaxes to inexpensive Langevin dynamics for sampling within approximately convex basins. Numerical experiments demonstrate superior performance over cheap gradient based sampling methods and competitive performance at lower cost compared to a state of the art method, parallel tempering.

Sparse Autoencoders (SAEs) are increasingly used to interpret foundation models, but their role as an actionable intervention space remains less understood, especially in vision. We study whether sparse visual features can be used not only for post-hoc analysis, but also to steer frozen vision-language models. We introduce \textbf{Visual Sparse Steering (\ours)}, a label-free method that trains a top-$k$ SAE on unlabeled activations from a frozen CLIP image encoder and, at test time, constructs an interpretable steering vector by amplifying the input's active sparse features and decoding the induced change. We show that this procedure admits a closed-form decomposition as \emph{centroid-deviation steering}: each input is moved along its deviation from the SAE-learned centroid. The residual term is controlled exactly by the SAE's per-sample reconstruction error, measured by FVU, yielding an FVU-based residual bound and motivating a reliability gate that falls back to zero-shot CLIP when SAE reconstruction is unreliable. With target-domain SAEs trained on unlabeled CLIP image-encoder activations, \ours improves zero-shot accuracy across nine image-classification datasets, achieving gains up to $+4.12\%$ with less than $0.1\%$ additional inference compute. Finally, a controlled upper-bound study, \oursplus, shows that selective amplification of sparse features can yield gains up to $+21.44\%$, exposing a \emph{reconstruction-vs-task saliency gap}: features salient for reconstruction need not align with features useful for downstream prediction.

As LLMs increasingly serve in advisory and deliberative roles, users rely on them for non-verifiable reasoning in domains lacking objective ground truths. However, traditional evaluations of LLM reasoning focus almost exclusively on fact-based domains, such as mathematics and science, leaving uncertainty over whether and to what degree models can handle ambiguous, subjective, or value-laden problems over time. To address this concern, we propose moral reasoning as a paradigmatic subdomain of non-verifiable reasoning. We define moral robustness as a model’s capacity to exhibit sound moral reasoning across time and contexts, and we introduce a scalable, adversarial, multi-turn evaluation framework to empirically measure this capability. We simulate 48,000 user-agent moral deliberations across four frontier LLMs, varying premise relevance, premise order, conversation duration, and the user's stated moral view. We find that models successfully ignore morally-irrelevant distractors, but shift their reasoning by up to 6.5\%, on average, towards the user's stated preferred moral view, and varying their reasoning depending on factors such as order (altering moral judgments by order in 13-22\% of the cases) and duration (altering moral judgments between single-turn and multi-turn in 10-24\% of the cases). Our analysis indicates that models tailor not just their final verdicts but their underlying justifications to align with a user’s moral viewpoint — a failure mode we characterize as moral deliberative sycophancy.

Despite remarkable progress in image-text understanding, vision-language models (VLMs) still struggle with compositional reasoning. In particular, they often fail to distinguish relational direction and attribute-object binding, leading to similar representations for semantically different image-text pairs. This problem mainly stems from the reliance on global image-text alignment, which captures coarse correspondence but overlooks fine-grained compositional structures. Toward this end, we propose an evidence-aware framework, termed Directional Relation Bucketing with Binding Localization (DELTA), to improve compositional understanding in VLMs. The key idea of our DELTA is to improve compositionality of VLMs from two complementary perspectives: direction modeling and binding localization. More specifically, we first leverage a learnable gate to model the cumulative contextual distance between anchor terms, thereby assigning relation concepts to direction-aware discrete buckets. To enrich textual representations with fine-grained relational semantics, we calibrate global semantic representations by incorporating intermediate-layer hidden states. Furthermore, DELTA leverages textual cues to ground visual evidence for object attributes, pulling image patches closer to their matched attribute descriptions while pushing them away from incorrect augmented ones. Extensive experiments on four compositional datasets demonstrate the effectiveness of the proposed DELTA. Our implementation is available at https://anonymous.4open.science/r/DELTA-2D60.

As collaborative machine learning scales to thousands of edge devices, two failure modes emerge: centralized federated learning creates server bottlenecks and single points of failure, while flat peer-to-peer systems force every node to increase its communication degree as the network grows. We propose Tiered Gossip Learning (TGL), a two-layer push–gossip–pull protocol that decouples data-holding leaf nodes from a decentralized relay backbone. In each round, leaves push local models to a subset of relays, relays gossip among themselves to mix models globally, and leaves pull updates from another random relay subset. This asymmetric design keeps per-leaf communication fixed as the network scales, offloading the mixing burden to a thin layer of higher-capacity relays without relying on any central coordinator for aggregation. Across CIFAR-10, FEMNIST, and AG News, TGL matches or exceeds baseline accuracy with up to 80\% fewer model exchanges. We provide convergence guarantees under standard smoothness, bounded variance, and heterogeneity assumptions, with explicit stage-wise consensus bounds characterizing how relay-layer connectivity governs the global mixing rate.

Recent studies have shown that leveraging the reasoning and analytical capabilities of large language models (LLMs) for long video understanding has become a promising approach. However, these methods are constrained by a structural limitation: representing video as a flat frame sequence makes it difficult to model the temporal structure and logical dependencies of events, which in turn hampers long-range reasoning and risks overlooking important information. To address this limitation, we propose LVGraph, a framework that models video content using a coarse-to-fine hierarchical graph. LVGraph constructs a semantic hierarchy where high-level graphs characterize the global event structures within the video, while low-level graphs capture fine-grained, query-relevant information. Instead of linear scanning, reasoning is performed via a query-aware traversal of this graph, adaptively identifying and retrieving the most salient keyframes by navigating the nodes and edges most relevant to the query. Finally, we leverage a Vision Language Model (VLM) to augment the keyframe information, and subsequently answer the question by reasoning over the graph and the enriched keyframe captions. Extensive experiments on long video understanding benchmarks confirm the effectiveness of our method. LVGraph significantly outperforms existing LLM-based state-of-the-art methods on EgoSchema, NExT-QA, and long-duration segments (averaging 44 minutes) of the Video-MME benchmarks.

Recovering 3D pose and shape of birds from single images is bottlenecked by the long tail of avian diversity: out of $\sim 10{,}000$ species, only a few dozen have 3D-labeled training data, and in-the-wild test sets routinely contain species never seen at training. Generalizing across clades is therefore a load-bearing problem. The current state of the art, AniMer+, mitigates it with a family-level contrastive loss that conveys only a binary same-family indicator per pair and discards the continuous structure of the species tree. We propose a **taxonomic Ornstein--Uhlenbeck prior** on the AVES shape code: a multivariate Gaussian whose covariance decays exponentially with Linnaean rank distance, with a learnable selection strength and noise scale. The prior generalizes family contrastive into a continuous, generative form, exposing the graded cross- and within-family similarity that a binary indicator cannot, and brings continuous comparative phylogenetics in as a biology-grounded inductive bias for the long tail. Our method consistently improves over AniMer+ on every reported avian benchmark cell, with the largest gains on the unseen-species CowBird benchmark. A permutation ablation isolates taxonomic structure from generic Gaussian smoothing, and a rank-granularity ablation identifies cross-order distinctions—the structure beyond a family-vs-not indicator—as the dominant source of the unseen-species gain.

As foundation models scale toward fusing more heterogeneous visual streams, understanding how diverse encoders interact under joint training becomes a prerequisite for principled design. Yet large vision-language models (LVLMs) currently lack the tools to do so, and parameter-efficient encoder configurations remain hard to identify before training. To re-examine encoder roles under joint training, on the 16-benchmark Cambrian-1 suite we retrain and evaluate all 31 non-empty subsets of five common vision encoders under a unified pipeline ($\approx$20k GPU-hours total), and report three findings. First, retraining each subset from scratch reveals encoder rankings that differ from those obtained by masking encoders on a fixed checkpoint, including which encoder ranks first overall. Second, we decompose each encoder's contribution into two axes, \emph{Capacity}, the score an encoder reaches on its own, and \emph{Necessity}, the drop when it is removed from the full pool. The two axes are not interchangeable. Pairing the two highest-Capacity encoders is suboptimal, while pairing a high-Capacity anchor with an adaptive complement matches the full five-encoder model. Adding further encoders beyond this pair yields only marginal gains. Third, at fixed parameter count, per-encoder pre-projector effective rank explains the residual score variation. The strongest pairs combine an anchor whose rank survives joint training with a complement whose rank \emph{expands} under it, suggesting that higher-rank, less-collapsed projector inputs correspond to a more favorable optimization regime at the encoder–projector interface. Together, the Capacity–Necessity decomposition and the pre-projector rank analysis, along with comprehensive evaluation through retraining, expose a methodological gap in multi-encoder LVLM design, and offer concrete primitives for closing it.

Scaling robot data is crucial for building generalist Vision-Language-Action (VLA) models, yet robot trajectories are fundamentally harder to scale than web-scale image-text data because they require embodied collection and sparsely cover the physical world. This makes representation quality a central bottleneck: under a fixed robot-data budget, VLA pre-training must convert limited trajectories into transferable visual-action knowledge, rather than merely fit actions. We propose **VLAct**, a VLA-oriented VLM backbone trained with a representation-centric pre-training recipe. VLAct preserves the broad VLM prior, avoids over-specializing the backbone to a single action head, and encourages shared action semantics across embodiments through VLM-prior preservation, multi-head continuous action co-supervision, and a partially unified cross-embodiment action layout, while leaving downstream users free to attach task-specific action heads during fine-tuning. Across multi-embodiment simulation benchmarks, real-world robot experiments, and unseen-embodiment transfer, VLAct consistently improves downstream performance under fixed fine-tuning protocols. On LIBERO-Plus and RoboTwin 2.0, VLAct surpasses large-scale industrial VLA systems such as ABot-M0 and LingBot-VLA, achieving 82.6\% and 92.5\% success, respectively. Most notably, on RoboCasa-GR1, a humanoid embodiment never seen during pre-training, **VLAct with only 20\% of downstream trajectories already outperforms the full-data GR00T-N1.6 baseline**. These results are obtained using fully open-source data and only a **16-GPU** training setup, showing that representation-centric pre-training is an important independent axis of VLA progress beyond data scaling. All models and training pipelines will be open-sourced.

Online map estimation is a crucial component of autonomous driving systems that reduces the reliance on costly high-definition maps. State-of-the-art (SOTA) methods commonly predict map elements as ordered sequences of points that form polylines and polygons. The evaluation of these methods relies predominantly on mean average precision (mAP) based on thresholded Chamfer distance (CD). This framework lacks sensitivity to point ordering and provides limited granularity in assessing geometric quality, making it difficult to distinguish which methods truly excel over others. In this work, we address these limitations on two fronts. For the single-instance similarity measure, we introduce sequence optimal sub-pattern assignment (SOSPA), an order-aware metric that enables fine-grained evaluation of individual geometries while satisfying all metric axioms. For the multi-instance evaluation framework, we propose polyline localisation and detection (PLD), a soft metric that jointly captures detection quality and geometric accuracy, replacing the hard thresholding of mAP with a principled soft assignment. Through evaluations on nuScenes, we demonstrate that PLD effectively ranks SOTA online mapping methods (MapTRv2, StreamMapNet, MapTracker) while providing a decomposed error analysis. This analysis identifies detection capability as the dominant bottleneck in current methods, revealing a performance trend that mAP fails to capture. Code for evaluation using our metrics will be released.

Image fingerprinting assigns each distributed copy a user-specific watermark for source tracing. Averaging collusion is a central threat: several recipients can average their differently marked copies to suppress each individual mark. Discrete bit-string watermarks are especially vulnerable because averaging drives bit evidence toward ambiguous decisions. We introduce **OrthoMark**, which maps user keys to near-orthogonal high-dimensional unit vectors and identifies colluders by cosine similarity to the extracted watermark direction. Under ideal averaging of $K$ vector keys, the normalized average preserves a cosine signal of order $1/\sqrt{K}$ for each colluder, while unrelated keys remain concentrated near zero. The spherical geometry of random directions gives a shifted-Beta null distribution for cosine similarities, enabling analytic per-key false-positive-rate control after validation on unwatermarked images. OrthoMark uses a neural watermark encoder and centered extractor trained with JND masking and a progressive curriculum for photometric, geometric, and collusion robustness. Experiments on MS-COCO, OpenImages, and AI-generated images show that OrthoMark maintains robust single-key detection under photometric and geometric distortions, and achieves the best strict all-colluder detection among evaluated methods, while preserving high visual quality.

Modern vision-language models struggle with basic compositional reasoning, failing to bind attributes to objects or relations to their referents. Existing benchmarks either rely on noisy real images that conflate confounding visual variables with the reasoning failure, or use simplistic synthetic scenes lacking the realism modern VLMs are tuned for. We introduce Auto-Comp, a fully automated, concept-driven pipeline that bridges this gap by generating photorealistic compositional benchmarks at scale. Its core innovation is a parallel A/B construction: for each concept, the pipeline emits a Minimal sample (template caption, isolated objects on a white background) and a Contextual sample (LLM-rewritten caption, objects embedded in a realistic scene), isolating core binding ability from visio-linguistic complexity. We instantiate four task families spanning the two canonical axes of compositional binding: Color and Shape-Color (attribute binding), and Position and Relative Size (relational binding). We evaluate over 25 VLMs spanning CLIP, SigLIP, hard-negative-trained, and frontier generative models. The findings are consistent across architectures and scales: every model exhibits a large Swap-vs-Confusion gap, with low-entropy distractors (e.g., repeated objects or colors) exposing failures beyond the known bag-of-words limitations. We further uncover a task-dependent trade-off: visio-linguistic context aids relational reasoning but hinders attribute binding through visual clutter. We publicly release the pipeline and benchmarks.

Vision-language-action (VLA) models have achieved strong performance in embodied manipulation, but still lack a clear mechanism to balance behavioral stability with task-semantic sensitivity. We identify two complementary failure modes. Under task-preserving changes, where task semantics remain unchanged but scene appearance varies (e.g., style, illumination, clutter, or paraphrasing), policies often exhibit unnecessary action drift. Conversely, under semantic-breaking changes, where key task semantics such as the target object or constraint are altered, policies frequently fail to produce sufficiently distinct behaviors and instead follow the original trajectory. To address this gap, we propose BAS-VLA, a task-semantic action calibration framework built on top of a frozen base VLA. BAS-VLA adopts a breaking-centered calibration core as the default path, and introduces a selective evidence-gated preserving auxiliary that activates only when nuisance variation is detected while task semantics remain consistent. On the OpenPI-pi0.5 / LIBERO-Object Milk-Swap benchmark, BAS-VLA maintains high success on clean (98.0%) and semantics-preserving conditions (97.5%), while reducing clean-criterion success to 0.0% under deliberate target-object swaps, demonstrating strong stale-task suppression and task-semantic separation. On validated style-preserving shifts, it improves success from 42% to 70% without degrading clean performance. These results highlight that reliable VLA behavior requires moving beyond appearance robustness toward explicit task-semantic action calibration.

Existing depth-compression methods for large language models (LLMs) operate on layers individually or restrict grouping to adjacent or contiguous positions, implicitly assuming functional redundancy is local. We challenge this assumption by computing full pairwise Centered Kernel Alignment (CKA) similarity matrices across thirteen LLMs spanning six architecture families and 1.6B--14B parameters, revealing that non-adjacent layers exhibit 3.6--15.9$\times$ more high-similarity pairs than adjacent ones. We propose Fusion via Community Mining (FCM), which builds the full $L \times L$ similarity graph, applies spectral community detection to recover clusters of functionally equivalent layers, retains the highest-Block-Influence layer per cluster, and applies LoRA recovery. At 2$\times$ compression, FCM lowers mean perplexity below ShortGPT on twelve of thirteen models, with statistically significant improvements on ten under a Welch's $t$-test with Holm--Bonferroni correction (up to 50\% on Qwen3-4B, $p_\text{corr}<0.001$); two rows are ties and Mistral-7B is the lone loss ($+$14.3\%). On three anchor models, FCM also outperforms three recent adjacent-pair methods (FlattenGPT, SWM, DeltaLLM) by 7.8--35.3\% under matched calibration and recovery. The perplexity advantage transfers downstream: on a five-benchmark zero-shot reasoning suite (ARC-Easy, ARC-Challenge, HellaSwag, WinoGrande, PIQA) over four anchor models, FCM matches or exceeds ShortGPT on 15 of 20 (model, benchmark) cells, and the 2$\times$-compressed Qwen2.5-7B realises 46\% fewer parameters, 42\% lower peak VRAM, and 84\% higher decode throughput on a single A100. Ablations confirm spectral grouping outperforms Fisher contiguous segmentation by 41\% and uniform partitioning by 51\%, and a compression sweep reveals a 2$\times$ crossover beyond which FCM's advantage grows monotonically and persists at the 14B scale. Code: \url{https://anonymous.4open.science/r/fcm-layer-fusion-D820}.

Language models (LMs) obtain much of their capability from the diversity of their training data. This is possible because neural architectures like transformers are remarkably flexible and can handle a variety of domains such as natural language, code, and mathematics. This raises the question: are transformers optimal for any specific domain? In this position paper, we present empirical evidence that the inductive biases of transformer-based LLMs should be re-examined, since, while broadly useful, they are often suboptimal on subsets of their training data. **Methods.** To support this position, we use a method that learns new activation functions within a transformer as a tool to explore the space of inductive biases. We train the resulting modified architectures on various data subsets. A large-scale evaluation shows that vanilla transformers are relatively easy to improve, but mostly on a per-domain basis. For example, an architecture optimized for a natural language dataset like FineWeb performs well on another such dataset like TinyStories, whereas architectures optimized for mathematics perform poorly on natural language. We also identify the mechanisms responsible for these improvements. **Implications.** These results call for re-evaluating core design choices in language models. We do not advocate a return to handcrafted, domain-specific models. Instead, we emphasize that different domains can benefit from different learning mechanisms. This will require new designs that preserve cross-domain interactions, analogous to how reasoning in the brain arises from the interaction of multiple specialized networks. We argue that this perspective may be key to improving reliability and data efficiency.

Many safety and alignment failures of large language models (LLMs) occur due to out-of-distribution (OOD) situations: unusual prompt or response patterns that are unforeseen by model developers. We systematically study whether LLM monitoring pipelines can detect these **OOD alignment failures** by introducing a benchmark called Misalignment Out of Distribution Benchmark (**MOOD**). It is difficult to find failures that are truly OOD for off-the-shelf models trained on vast safety datasets. We sidestep this by including a restricted training set in MOOD that we use to train our own monitors, as well as seven test sets with diverse alignment failures that are outside the training distribution. Using MOOD, we find that guard models (safety classifiers) often fail to generalize OOD. To fix this, we propose combining guard models with OOD detectors. We test four types of OOD detectors and find that a combination of a guard model and Mahalanobis distance-based OOD detector can improve recall from 39% to 47%. We also establish positive scaling trends across model scales for monitors that combine a guard model and OOD detector; we find that incorporating OOD detection into monitoring achieves a higher recall gain than using a guard model with 20 times more parameters. Our work suggests that OOD detection should be a crucial component of LLM monitoring and provides a foundation for further work on this important problem.