NeurIPS 2026 Papers

Large language models (LLMs) have achieved remarkable success across diverse tasks, yet their ever-growing model sizes impose significant computational and memory costs that hinder practical deployment. Pruning addresses this by introducing sparsity into weight matrices, but existing layer-wise pruning methods operate independently on each linear layer, ignoring cross-layer dependencies and yielding globally suboptimal masks. End-to-end pruning methods address this limitation but incur prohibitive computational overhead. In this work, we propose FiedlerPrune, a pruning framework that achieves a better balance between the two paradigms by incorporating cross-layer dependencies. Guided by graph theory, FiedlerPrune constructs weighted multipartite graphs over consecutive linear layers within each Transformer block, and derives an inter-layer importance score from each weight's contribution to the Fiedler Value of the corresponding graph. Combined with any intra-layer metric, FiedlerPrune provides a unified and versatile pruning criterion reflecting both local quantitative importance and cross-layer connectivity-preserving importance . Extensive experiments demonstrate that FiedlerPrune consistently improves over layer-wise baselines, and matches or outperforms end-to-end methods at orders-of-magnitude lower pruning cost. We further demonstrate FiedlerPrune's practical utility through inference speedup, compatibility with quantization, and post-pruning fine-tuning.

Federated Multiple Instance Learning (MIFL) has emerged as a promising paradigm for privacy-preserving weakly supervised learning, particularly in medical image analysis where instance-level annotations are expensive or unavailable. Existing federated MIL methods mainly rely on parameter aggregation to learn a global model across clients. However, in MIL, the key challenge is not only client-level data heterogeneity, but also the ambiguity of instance-level evidence. Since supervision is only provided at the bag level, each client must independently infer discriminative instances through local attention mechanisms. Under heterogeneous data distributions, different clients may focus on inconsistent or even misleading instances, resulting in cross-client critical evidence drift. To address this problem, we propose FedSEM, a Federated Shared Evidence Memory framework that introduces an evidence-level communication pathway in addition to standard model aggregation. Specifically, each client identifies high-confidence evidence instances using local attention scores and compresses them into compact prototypes with importance scores. The server then refines these prototypes through redundancy removal, diversity selection, and score normalization to construct a shared evidence memory. Importantly, FedSEM does not require transmitting raw patches or dense instance embeddings; it only exchanges a small number of evidence prototypes and periodically broadcasts the refined memory, thereby introducing limited additional communication overhead compared with standard federated training. The shared memory serves as cross-client evidence anchors to guide local attention learning and instance-level representation optimization. By explicitly aligning critical evidence patterns across clients, FedSEM mitigates evidence drift while maintaining communication efficiency and privacy preservation. Extensive experiments under heterogeneous federated settings demonstrate that FedSEM consistently improves MIL performance and generalization over existing federated learning baselines.

In multiclass softmax cross-entropy, the full generalized Gauss-Newton (GGN) curvature couples all output logits through the softmax covariance, making second-order updates harder to scale as the number of classes grows. We show that the standard multiclass GGN can be decomposed exactly into a rank-one true-vs-rest term and a positive semidefinite within-competitor covariance term. Fast Gauss-Newton (FGN) retains the first term and drops the second, yielding a positive semidefinite under-approximation of the multiclass GGN that is exact for binary classification. The derivation uses an exact true-vs-rest scalar-margin representation of softmax cross-entropy: the loss and gradient are unchanged, and the approximation enters only at the curvature level. Exploiting the FGN curvature structure, the damped update can be written as an equivalent whitened row-space system with one row per mini-batch example. We solve this system matrix-free by conjugate gradient using Jacobian-vector and vector-Jacobian products of the scalar margin map. Targeted mechanism experiments and a fixed-feature head evaluation support the predictions from the decomposition: FGN stays closest to the full softmax GGN when competitor mass is concentrated or damping is large, and deviates as the dropped within-competitor covariance grows.

Cross-view object correspondence (CVOC) aims to establish associations of the same target between query and search videos captured from different viewpoints. Existing methods often rely on visual cues for CVOC. Despite recent progress, these methods struggle in complicated scenarios with substantial viewpoint changes and appearance variations, due to lacking sufficient target information. Addressing this, we introduce TeSCo, a novel framework that exploits rich Textual Semantics of an object, in addition to its visual cues, for cross-view object Correspondence. Our key insight is, textual description of a target, by capturing diverse attributes such as category, texture, and color, provides viewpoint- and appearance-invariant semantics, which are complementary to visual cues and can hence enhance correspondence robustness in complicated scenes. Inspired by this, TeSCo first generates a textual expression of the target from the given query view using a vision-language model, and then extracts textual feature to enhance visual features for localization in the search view. To realize this, we introduce a simple yet effective text-conditioned cross-modal fusion (TCF) module that incorporates the textual feature into visual features with text-guided modulation, producing more robust multimodal target representation for object correspondence. Since not all textual cues are equally compatible with the target in current search-view frame, we propose an iterative textual feature refinement (ITFR) module that progressively adjusts textual feature using search-view information before the TCF module, enabling search-view-aware textual semantics to be fused into visual features and leading to better performance. In extensive experiments on Ego-Exo4D and HANDAL-X, TeSCo demonstrates state-of-the-art results and largely outperforms other models, validating its efficacy. Our code and model as well as results will be released.

Children acquire language with remarkable speed and robustness from limited linguistic input in ways that surpass today's best large language models. Recent research suggests this efficiency stems from learning through multimodal sensory experiences where semantic alignment between what children see and hear grounds language in the external world. However, vision-language models (VLMs) trained on curated web data fail to generalize to the sparse, weakly-aligned egocentric streams produced by wearable devices, embodied agents, and infant head-cams—and no fixed evaluation pipeline exists for measuring progress on this regime. We train VLMs on datasets with varying degrees of semantic alignment between visual and linguistic inputs, including naturalistic infant egocentric videos, and evaluate them with a comprehensive suite spanning multimodal language grounding and unimodal vision and language tasks. At the core of this suite is _Machine-DevBench_, a corpus-grounded benchmark of lexical and grammatical competence, automatically generated from the model's training vocabulary across logarithmic frequency bins to eliminate the train/eval mismatch and low statistical power of prior developmental benchmarks. Our results show that current VLM paradigms hinge on the tight semantic alignment of curated data and fail to exploit the weakly-aligned signal that dominates naturalistic egocentric input—the very regime in which humans thrive. To motivate progress, we introduce the EgoBabyVLM Challenge to drive the development of models capable of grounded language learning from the kind of naturalistic data that human infants experience.

Flow cytometry is the primary assay for single-cell immune profiling, yet cross-study modelling remains under-explored. The challenge is two-fold: Studies measure different marker panels, inducing feature-space inconsistency, while batch effects shift signals even on shared markers, making transfer harder than in vision or language where input spaces are stable. We conduct a large-scale empirical study, pretraining an encoder on 89,324 samples from 112 public studies spanning 525 distinct marker panels. We evaluate it frozen --- isolating representation quality from task-specific adaptation, on five clinical tasks under a taxonomy of panel-overlap conditions, including three panel-zero-shot tasks whose marker panels were unseen during pretraining, and one negative control with no expected discriminative signal. We further characterise transfer across scaling axes and probe robustness to marker removal at test time. Despite heterogeneity that disrupts direct transfer, the frozen encoder matches or exceeds in-distribution specialist encoders on 4 of 5 tasks, including all three novel-panel tasks, and behaves as expected on the negative control. Scaling analyses reveal saturation in both encoder capacity and random corpus subsampling, and representations degrade gracefully under marker reduction. These results suggest a pretrained encoder can serve as a general-purpose feature extractor for clinical cytometry data, without task-specific retraining.

Mixture-of-Experts (MoE) architectures have become essential for building capable large language models, with recent work demonstrating the benefits of fine-grained expert designs. However, training MoE models from scratch is computationally expensive, and existing upcycling methods that convert dense models to MoE face limitations: they either initialize experts as identical copies (limiting routing diversity) or use random perturbations (risking knowledge loss). We propose \textsc{DivMoE}, a framework that addresses these challenges through two innovations. First, we introduce \emph{domain-specialized expert initialization}, deriving fine-grained experts from models fine-tuned on distinct domains (mathematics, code, science, commonsense), providing meaningful diversity while preserving pre-trained knowledge. Second, we propose \emph{diversity-constrained routing}, which enforces that each token selects at most one expert per domain group, structurally preventing routing collapse and enabling cross-domain knowledge composition. Experiments on two base models demonstrate that \textsc{DivMoE} outperforms all upcycling baselines (47.0% vs.\ 44.4% average accuracy) and achieves competitive performance with MoE models trained from scratch, matching Moonlight-MoE at 64.5% average while being constructed via efficient upcycling.

Cross-entropy (CE) loss is central to deep learning, but existing theory often relies on simplifications—such as squared loss or convex models—that miss key aspects of CE optimization. In this work, we study multi-class CE dynamics using a two-layer linear network with orthogonal inputs, the simplest non-convex setting where the CE implicit bias remains unresolved. This coincides with the unconstrained features model used to study neural collapse (NC). Our analysis is based on a key observation: Hadamard initialization diagonalizes the softmax operator. This allows us to extend the spectral initialization framework that Saxe et al. (2013, 2019) developed for squared loss. We prove convergence to NC under spectral CE training and give the first finite-time analysis in this setting via an explicit Lyapunov function that decreases monotonically to NC despite spurious critical points. We further identify CE-specific phenomena absent under squared loss—coupling, non-monotonic convergence, qualitative dependence on the number of classes, and exponential slowdowns. We also characterize the role of network width and show empirically that spectral dynamics qualitatively model small random initialization.

Video multimodal large language models have made strong progress on open-ended video understanding, but they still lack precise local spatiotemporal perception. When two videos share almost the same global semantics and differ only in a short time span or a small region, current models often fail to find the change and provide reliable evidence. We propose **DeltaVid**, a verifiable proxy-task framework that enhances fine-grained spatiotemporal perception with cross-video differences. The key idea is to turn cross-video spot-the-difference into a trainable perception signal, where a model identifies local changes, judges temporal boundaries, and organizes spatial evidence by comparing similar videos. To make this signal scalable to train and reliable to evaluate, we further introduce DeltaVid-10K and DeltaVid-Bench, which convert controllable local differences in real videos into evidence-labeled training and test samples. Experiments show that DeltaVid substantially improves performance on cross-video difference understanding and transfers the learned local evidence ability to general video understanding benchmarks, including MMVU, MLVU, Video-MME, VideoHolmes, VideoMMMU, LVBench, TempCompass, and LongVideoBench. These results show that cross-video differences are not only an effective way to diagnose fine-grained perception failures, but also a scalable proxy supervision that moves Video MLLMs from coarse semantic understanding toward fine-grained spatiotemporal evidence reasoning.

Speculative decoding accelerates LLM inference by having a small drafter propose tokens that a larger target model verifies in parallel. Current SOTA diffusion-based parallel drafters such as dFlash predict the full $B$-token block in one forward pass, allowing deeper drafters and higher accepted length. However, across the field, from autoregressive tree drafters to parallel block drafters, training losses use fixed per-position weights that do not adapt to the drafter's current bottleneck position. We derive per-position training weights directly from the expected accepted block length, so that each position's weight reflects its actual contribution to acceptance. The resulting loss, D-PACE (Dynamic Position-Aware Cross-Entropy), automatically identifies where the block's acceptance bottleneck lies and shifts training signal there as the drafter improves. D-PACE consistently improves both wall-clock speedup and accepted length across diverse benchmarks and positions as a drop-in replacement with negligible training overhead and requires no changes to the drafter architecture or inference pipeline, making it broadly applicable to general model architectures.

Modern machine learning progresses through empirical work, benchmarking new methods to evaluate relative performance. However, the statistical variability inherent to evaluation -- exacerbated by the stochastic nature of many algorithms -- often makes performance estimation unreliable due to the limited test samples available, leading to a validation crisis in which genuine advances are difficult to discern. In this work, we show that cross-validation improves markedly confidence when evaluating and comparing learning algorithm performances. We introduce the concept of \emph{sample gain}, which quantifies the virtual data augmentation achieved by using multiple cross-validation splits to reduce benchmarking variance. Experiments on both synthetic and real-world datasets (histopathologic scans and NLP fine-tuning) demonstrate that multiple splits can substantially improve the reliability and stability of performance estimates, with diminishing returns often setting in later than expected. We also introduce a procedure to dynamically early-stop cross-validation by estimating from the first few folds if subsequent folds will bring large sample gains. Our findings highlight the value of pushing cross-validation on available samples to achieve robust and reliable benchmarking.

Graph matching aims to establish node correspondences while preserving both unary appearance compatibility and pairwise structural consistency. Recent deep graph matching methods have achieved strong empirical performance, but many of them either compress edge information into node embeddings before matching or rely on surrogate optimization signals for quadratic objectives. These design choices can weaken structural discrimination and may lead to optimization directions that are not aligned with the relaxed quadratic assignment problem (QAP). We propose Cross-order Consensus Graph Matching (CCGM), a framework that explicitly couples first-order node assignment and second-order edge assignment. CCGM constructs line graphs to convert edge matching into a node matching problem, learns node and edge affinities in parallel, and refines them through a Cross-order Consensus Solver motivated by the exact gradient of the factorized relaxed QAP. We further introduce Cross-order Alignment Regularization to encourage agreement between node-induced edge assignments and edge-induced node support during training. Experiments on standard visual graph matching benchmarks show that CCGM consistently improves matching accuracy over competitive baselines.

AI systems operating near the limits of their capabilities can maximize utility while minimizing the risk of error by abstaining or deferring questions they are not confident about to a human or a stronger model. Recently, large language models (LLMs) have been post-trained to estimate confidence in their own answers. To achieve this, methods reinforce confidence scores based on the correctness signal to each question-answer pair \emph{individually}, with rewards that encourage assigning confidence 1 to correct answers and 0 to incorrect answers. However, estimating confidence scores is useful in that it enables selecting only the most confident answers, to maximize the share of solved questions while accommodating the user's risk tolerance. Assigning confidence 1 to correct answers and 0 to incorrect ones is theoretically optimal, but unattainable in practice and detrimental for selective prediction with a non-zero risk tolerance: confidences pushed to 0 and 1 carry no information about how answers compare to one another. We therefore introduce \ours (Cross-question Abstention Reward Estimation), which computes rewards over a set of question-answer pairs: At each of several risk tolerances, the confidences across the set induce an abstention threshold, and each confidence is rewarded for lying on the correct side of it. Since the threshold itself comes from the set, every verbalized confidence is reinforced relative to other questions' confidence, not against its own binary correctness label. On math, \ours increases the coverage at 5\% risk (C@5) across MATH-500, GSM8K, and Big-Math-Digits from 0.8\% to 42.4\%, compared to existing pointwise correctness-matching rewards. When trained on multi-hop Wikipedia QA from HotpotQA, \ours improves C@5 in out-of-distribution general-knowledge, reasoning, and math benchmarks (including CommonsenseQA, GPQA, SimpleQA, and TriviaQA) from 9.9\% to 23.6\%.

Neural Posterior Estimation (NPE) trains an amortized conditional density estimator to approximate posterior distributions in simulation-based inference. Each training pair is a single draw from the joint distribution of latent and observation, so posterior variance at an observation must be inferred from variation across pairs rather than within them. However, with a sufficiently flexible variational distribution, the NPE training objective is unbounded, and standard optimizers yield arbitrarily small variance. Regularizers like early stopping, dropout, and weight decay constrain network capacity without targeting the correct posterior variance. We propose Cross-Fitting NPE (CF-NPE), which separates mean estimation from higher-moment estimation via sample splitting. Concretely, CF-NPE estimates a conditional-mean ensemble by $K$-fold cross-fitting, then fits a conditional density to the resulting out-of-fold residuals. The conditional variance of an out-of-fold residual is bounded below by the true posterior variance, giving CF-NPE a well-defined population-level target that standard NPE lacks. Across eight synthetic benchmarks and a cosmological application, CF-NPE achieves higher held-out log-likelihood than standard NPE, with gains that grow with the latent dimension.

Large language models from different families use different hidden dimensions, tokenizers, and training procedures, making behavioral directions difficult to compare or transfer across models. We introduce an anchor-projection framework that maps hidden representations from each model into a shared anchor coordinate space (ACS). Behavioral directions extracted from source models are projected into ACS and averaged into a canonical direction. For a new model, the canonical direction is reconstructed into its native hidden space using only anchor activations, without fine-tuning or target-specific direction extraction. We evaluate five instruction-tuned model families and ten behavioral axes. We find that same-axis directions align tightly across the Llama-Qwen-Mistral-Phi cluster on the ACS. This shared structure transfers to downstream tasks: held-out targets achieve \(0.83\) ten-way detection accuracy and \(0.95\) mean binary AUROC, while canonical steering induces refusal-rate shifts of up to \($\Delta$ = +0.46\) under distribution shift. Sensitivity analyses show that two source models and small anchor pools already suffice to approximate transferable directions. Overall, ACS provides a novel perspective on cross-family interpretability, revealing that representation-level transfer remains robust across model families. The code for this paper is available at \url{https://anonymous.4open.science/r/ACS-annoymous/}.

Predicting genetic perturbation responses across cell lines is challenging not only for models, but also for the metrics used to evaluate them. We introduce a strict cross-cell-line CRISPRi stress test on Replogle Perturb-seq, training on K562 and RPE1 (or additionally HepG2) and evaluating on held-out Jurkat. We pair this setting with two necessary controls: target-control-copy, which predicts no perturbation effect, and shifted-control baselines, which translate target-line controls by a source-derived mean shift. Across classical baselines, recent deep models, CellFlow, and context-conditioned flow-matching variants with five frozen single-cell foundation-model encoders plus a PCA/null context, we find that aggregate distributional metrics can be strongly driven by target-line cell-state variability rather than perturbation-specific response. Target-control-copy is competitive on aggregate distributional metrics, while shifted-control approaches CellFlow on Energy and substantially outperforms it on perturbation-specific Pearson. Used as fixed per-cell context for a shared downstream flow head, none of five frozen scFM encoders matches a PCA/null encoder in this strict cross-line setting. Finally, anchored mean-preserving residual-flow probes reveal a metric-specific trade-off: full-space residual flows improve $W_2$ but degrade per-gene moment matching, whereas low-rank constrained flows preserve moments but lose most $W_2$ gain. No tested method simultaneously dominates perturbation-specific response and biologically relevant distributional fidelity. These results argue for perturbation-aware cell-level evaluation in cross-cell-line virtual-cell benchmarks.

Selecting a pretrained encoder for a downstream task without retraining or labels is a standard preprocessing step in transfer learning. The dominant label-free score, RankMe, was validated within architecture families and is widely applied across them. We show that its cross-architecture use is unsound: rankings are systematically driven by the penultimate-layer dimension rather than representational quality, and on heterogeneous model pools the score performs worse than uniform random selection. We give a random-matrix explanation establishing this failure as inherent to any rotation-invariant score, and propose Subspace-Corrected Ranking (SCR), a family of label-free corrections that breaks rotation invariance via alignment with a fixed reference encoder. Across three architecturally distinct pools and 23 aggregated task units, the proposed SSR-dagger reduces mean regret from +0.165 (RankMe) to +0.071 (95% CI [+0.045, +0.099], p < 1e-4). Paired with a within-family score in a two-stage pipeline, it matches this quality at one-third of the forward-pass cost. Five labelled architecture evaluations suffice to match the best fully label-free strategy; K=35 labelled evaluations reduce regret to +0.018 --- a 5.7x improvement at the same forward-pass cost.

Three-dimensional Gaussian Splatting (3DGS) has recently become a promising representation for dense SLAM, due to its explicit scene parameterization, efficient differentiable rendering, and high-quality visual reconstruction. Extending 3DGS to semantic SLAM enables the construction of maps that are not only geometrically accurate but also semantically interpretable. However, existing semantic Gaussian SLAM methods usually optimize semantic, depth, and normal renderings through separate supervision objectives. Such a design underexplores the fact that these outputs are generated from the same Gaussian field and should therefore provide a concordant explanation of the underlying scene structure. This limitation is especially problematic around object boundaries and geometric discontinuities, where inaccurate Gaussian growth can cause cross-boundary expansion, boundary blurring, and semantic leakage. To address this problem, we propose Concord-SLAM, a boundary-native semantic Gaussian SLAM framework that establishes concordance among semantic, depth, and normal renderings while promoting boundary-aware Gaussian growth. Our framework contains two key components. First, Cross-Render Concordance (CRC) jointly examines semantic, depth, and normal views rendered from the same Gaussian map, and converts their structural discrepancies in boundary responses, regional continuity, and local geometry into optimization signals. In this way, disagreement among different renderings is used as an intrinsic cue for improving the semantic-geometric coherence of the Gaussian field. Second, Boundary-Native Splatting (BNS) actively inserts new Gaussians near semantic boundaries, depth discontinuities, and salient geometric variations, encouraging primitives to align with physical and semantic boundaries during map construction rather than correcting boundary mixing only through later optimization. Experiments on Replica and ScanNet demonstrate that Concord-SLAM achieves superior performance in camera tracking, scene reconstruction, and semantic mapping.

Cross-modal knowledge distillation transfers knowledge from a teacher modality to a student modality. Existing feature-level alignment methods typically assume that teacher and student features reside in structurally alignable representation spaces. However, this assumption does not hold when cross-modal features are structurally heterogeneous and lack clear unit-level correspondence, such as 2D spatial visual grids and 1D temporal audio sequences, thereby limiting the applicability of feature-level alignment. To address this challenge, we propose a cross-modal distillation framework that enables effective knowledge transfer across structurally heterogeneous feature spaces via a vector-quantized codebook. Specifically, teacher features are abstracted into a set of vector-form codes regardless of their original feature structure, and the selected codes serve as concept-level anchors for student learning. Code selection is guided by both task relevance and student compatibility, allowing the student to receive transferable teacher knowledge without requiring direct unit-level feature alignment. Experimental results across diverse cross-modal distillation scenarios demonstrate the effectiveness of the proposed framework on classification and semantic segmentation tasks.