NeurIPS 2026 论文列表

Chunked prefill has become a widely adopted serving strategy for long-context large language models, but efficient attention computation in this regime remains challenging. Existing sparse attention methods are primarily designed for one-shot prefill and do not translate efficiently to chunked prefill: block-sparse kernels lose efficiency when the query length is limited by the chunk size, while fine-grained pattern search becomes costly when repeated over the accumulated KV cache at every chunk. QUOKA, a recent method that directly targets chunked prefill, avoids sparse-kernel overhead but relies on query-subsampled, token-level KV selection, which can miss query-specific KV entries and introduce explicit KV-copy overhead. To address these limitations, we propose CompactAttention, a chunked-prefill attention mechanism based on Block-Union KV Selection. CompactAttention treats 2D block-sparse masks as KV-selection signals rather than direct sparse-kernel execution plans, and converts them into GQA-aware per-group KV block tables through Q-block union and intra-group union. This construction produces the minimal block tables that preserve all KV blocks selected by the input masks under paged execution constraints, enabling selected KV blocks to be accessed in place without explicit KV compaction. On LLaMA-3.1-8B-Instruct, CompactAttention maintains accuracy close to dense attention on the RULER benchmark while delivering up to 2.72$\times$ attention speedup at 128K context length under chunked prefill.

Self-improvement is a critical capability for large language models and other intelligent systems, enabling them to refine their behavior and internal consistency without external supervision. Despite its importance, prior approaches largely rely on empirical heuristics and lack formal guarantees. In this paper, we propose a principled framework for self-improvement based on the concept of coherence, which requires that a model's outputs remain consistent under task-preserving transformations of the input. We formalize this concept using projection-based mechanisms that update a baseline model to be coherent while remaining as close as possible to its original behavior. We provide rigorous theoretical guarantees that these mechanisms achieve monotonic improvement, measured by a reduction in expected Bregman divergence. Our analysis is comprehensive, covering both \emph{direct} and \emph{two-step} projection methods, and robustly extends these guarantees to non-realizable settings, empirical (finite-sample) distributions, and relaxed coherence constraints.

Large-scale approximate nearest neighbor search commonly relies on partitions for indexing: database vectors are partitioned into clusters, and for each query a probing function selects the clusters to be scanned. The query probing function and the database partition are rarely treated as separate entities: most techniques assign queries with the same assignment function as the database vectors, which is suboptimal especially when database and query distributions differ. This paper introduces CwA (Cluster with Auctions), which addresses this limitation by jointly learning a balanced database partition and a neural probing function. CwA optimizes search performance directly for the query distribution. It minimizes its objective by alternating two steps: (i) gradient descent on the neural network of the probing function, and (ii) a large-scale combinatorial optimization of the cluster assignment for the database vectors. We solve the latter with a parallelizable auction algorithm that balances the partition by design. To further scale CwA, we extend the method to a Cartesian product of clusters that increases the partition’s granularity. When database and query distributions differ, CwA achieves up to 4.7$\times$ throughput over the state of the art at equal recall. In the in-distribution (ID) setting, even a simple linear probing function trained with CwA outperforms competing deep neural methods.

Cauchy activations and Cauchy-adaptive layers can give large gains on scientific learning tasks, but the gains are selective and not uniquely Cauchy-specific. We organize this selectivity with an empirical loss-architecture alignment taxonomy: an architecture can help when the objective exposes structures it can use, and can lose under a mismatched objective. In score learning, Cauchy score networks improve over SiLU under a Fokker-Planck PDE loss, but the same advantage disappears under denoising score matching and a Gaussian activation is comparable to Cauchy under the PDE loss. In loss-activation factorials, robust residual losses explain more of the heavy-tailed and spectral gains than the activation alone; an Allen-Cahn trap initially escaped by CauchyAct+Cauchy loss is also escaped, sometimes more strongly, by CauchyAct+L1, CauchyAct+Huber, and ReLU+Cauchy loss. We also analyze ResidualCAN, a CauchyAct main path plus a softmax-weighted Cauchy Adaptive Node residual. It achieves 1520x lower stationary Allen-Cahn error than a SiLU MLP at epsilon = 1.0 and remains strongest or second strongest against KAN-style, tuned SIREN/Fourier, and adaptive PINN baselines across an Allen-Cahn epsilon-suite. Ablations show that most of this gain comes from a dense local-basis residual rather than from Cauchy activation alone. A Poisson/Burgers boundary check prevents a broad PDE dominance claim: tuned SIREN/Fourier and KAN-style baselines can dominate away from transition-layer structure. The resulting claim is deliberately bounded: Cauchy components are one useful instance of robust/local-basis co-design in low-dimensional scientific tasks, not a universal activation or a uniquely rational mechanism.

Production training of large models alternates between model edits and hybrid-parallel strategy planning. Fixed-configuration planners solve each revision from scratch, while unvalidated reuse across edits risks returning a different plan than full re-search would. We introduce a dependency-keyed per-layer primitive interface for re-planning: closed-form layer primitives expose communication, memory, and time to the solver, and the same templates expose the input dependencies needed for reuse-equivalent re-planning. CONCORD attaches dependency sets and value signatures to primitive evaluations, enabling dependency-keyed re-planning to reuse only signature-valid primitive-evaluation work while preserving the same logical candidate set and feasibility predicate as cache-disabled full search. On production MoE strategy-quality runs—including a 4096-device MoE-438B deployment and a 32k-token long-context point—CONCORD's single-call plan achieves 1.14–1.24× training step-time speedup over an expert-tuned Megatron-style baseline (1.70× on the long-context point). Across a 9-fixture sweep at 256–4096 cards, CONCORD's warm re-planning achieves median 2.74× planner wall-clock speedup over cache-disabled full search across 54 in-dispatch cells. Full-search equivalence holds throughout: candidate and feasibility counts match in every cell, the top-1 plan matches in 54/54 cells, and stale-hit mismatches (cached value disagreeing with cold recomputation) are 0/162 over paired trials.

As sequence lengths continue to grow, distributed attention based on sequence parallel (e.g., Ring-Attention) has become a mainstream solution for large language models (LLMs). Nevertheless, inherent communication dependencies limit computation–communication overlap, reducing GPU utilization. Moreover, in cross-hardware (NUMA or node) architectures, due to imbalanced communication bandwidth, frequent communication synchronization will limit overall performance. Based on these insights, we propose CHARM+, Cross-Hardware Attention with Re-Merge Multistream Mechanism. We reorganize sequential communications into several levels and achieve intra-level parallelism and computation overlap. Furthermore, we decouple cross-hardware communication from intra-hardware communication, eliminating synchronization overhead caused by bandwidth imbalance. Forward and backward algorithms were evaluated on multiple hardware architectures and consistently outperformed state-of-the-art methods. Compared to Ring-Attention, our forward and backward algorithms achieve average speedups of $1.3\times$ and $1.7\times$ on a dual-NUMA PCIe architecture, and $2.8\times $ and $3.9\times$ on a dual-node NVLink architecture, respectively.

Layout generation for real-world facilities is a challenging problem, requiring reasoning over irregular site boundaries, heterogeneous orientations, access-aware placements, and motion-planning feasibility. Yet, most existing layout benchmarks in the generative AI space target simpler placements over rectangular domains and rely on distributional metrics such as FID and IoU that reward conformity to dataset priors, thus discounting design innovation. Motivated by these gaps, we introduce $\textbf{ALPS-Bench}$, a benchmark of $1,000$ professionally annotated real-world facility layouts paired with an instance-specific scoring protocol grounded in a structured design manual. As a strong baseline for $\textbf{ALPS-Bench}$, we propose $\texttt{CANDO}$, a training-free multi-agent framework in which specialized agents iteratively refine layouts through a verification-grounded loop, concentrating reasoning on strategic spatial decisions. We demonstrate that $\texttt{CANDO}$ surpasses state-of-the-art baselines on the widely adopted $\textbf{PubLayNet}$ and $\textbf{RICO}$ benchmarks by a significant margin, establishing cooperative agentic design as a broadly effective recipe for constraint-aware layout synthesis. Code and benchmark will be released upon acceptance.

The design of modern neural architectures has converged through incremental empirical choices, yet the mechanisms governing their training dynamics remain only partially understood. We identify and analyze a negative weight drift induced by the interaction between standard losses and positively biased activation functions. We prove that under MSE or cross-entropy loss, the gradient with respect to positive pre-activations is non-negative in expectation at initialization, driving downstream weights toward negative values during early training. The drift is intrinsic to optimization rather than data, and persists across architectures (MLP, ResNet, ViT, GPT-nano, MP-SENe) and asymmetric activation functions (ReLU, GELU, SiLU). Coupled with ReLU, weight drift produces activation sparsity reaching up to 90\% in GPT-nano. We characterize the sparsity–accuracy tradeoff across 79 configurations and identify a sharp accuracy cliff above $\sim$70\% activation sparsity. While ReLU$^2$ achieves a good sparsity–accuracy tradeoff, it pathologically amplifies identified activation spikes in intermediate transformer layers. Clipping resolves this while preserving the representational benefits of squaring: clipped ReLU$^2$ outperforms its unclipped version, and GELU$^2$ achieves the lowest validation loss on GPT-nano.

Diffusion policies have achieved strong performance in reinforcement learning (RL) due to the exceptional expressivity in capturing complex action distributions. However, existing approaches focus primarily on positive samples and lack explicit modeling of negative samples, thereby failing to fully exploit this representation capacity and ultimately limiting performance. To mitigate this problem, we propose BOBAC (BOotstrapped Bipartite Actor-Critic), a novel method for diffusion policy optimization via pairwise preference learning. Bridging the gap between distribution modeling and preference learning, we reformulate diffusion policy optimization within a preference-aware framework via the Bradley-Terry (BT) modeling, which enables the effective exploitation of negative samples through relative ranking. Since the original BT loss assigns the same weight to all pairs regardless of their varying importances, we further design a soft-gate mechanism to adaptively weight each pair according to its Q-gap. Experimental results on 6 MuJoCo tasks and 4 H1Bench tasks demonstrate that, compared with representative model-free and generative-policy RL baselines, BOBAC achieves state-of-the-art performance with consistent improvements in all continuous-control environments.

Recently several methods have shown promise for purely in silico “de novo” design of antibodies which bind to drug targets. The methods which have shown success in vitro propose candidates via either a generative diffusion model or hallucination-based sequence optimization, and then filter these candidates with a structure predictor model. Hallucination methods rely on compute-intensive backpropagation through a structure predictor model for each candidate, and the generative methods require expensive structure re-prediction and filtering of many (often mostly non-passing) candidates, limiting their application to test-time generation of large libraries. Given the highly variable per-drug-target hit rates of current methods, screening large libraries is a well established route to improved antibody candidate discovery. Thus, we propose an alternative approach, Bonobo, which instead formulates this problem as black-box optimization of a per-drug-target generative model, amortizing candidate generation into training. This is accomplished by using a structure predictor as a reward signal to directly train a GFlowNet generative model. We show that this approach can match or exceed the in silico metrics of state-of-the-art approaches, while allowing for dramatically more efficient generation of diverse and arbitrarily large numbers of antibody binder candidates.

Vector quantization is a fundamental primitive for scalable machine learning systems, enabling memory-efficient storage, fast retrieval, and compressed inference. Recent rotation-based quantizers such as EDEN, RabitQ, and TurboQuant have introduced strong guarantees and empirical performance, but the surrounding comparisons have been difficult to interpret because they rely on different distortion criteria, probability regimes, and implementation assumptions. As our first contribution, we provide a unified theoretical comparison of these methods and show that their relative advantages are criterion-dependent rather than absolute: TurboQuant is favorable for MSE distortion, EDEN is effective for expected inner-product distortion, and RabitQ provides strong high-probability control. This comparison clarifies the design principles behind each advantage and shows that no existing method uniformly dominates across the relevant measures. As our second contribution, we introduce *Block-Sphere Quantization* (**BlockQuant**), a new rotation-based block quantization algorithm designed around the spherical geometry of randomly rotated vectors. Unlike coordinate-wise quantizers, **BlockQuant** quantizes blocks on the sphere, preserving the geometry of rotated embeddings more faithfully. We prove that this block-spherical design improves expected distortion performance, including both reconstruction MSE and expected inner-product distortion. Experiments on real embedding data support these theoretical improvements.

Plug-and-Play (PnP) methods have become standard tools for solving imaging inverse problems by replacing the intractable maximum a posteriori (MAP) denoiser with the MMSE one. While this mismatch has been widely treated as unavoidable, recent works have sought to close this gap by targeting the MAP with diffusion-model scores. We show this is problematic in practice: learned scores do not match the true ones, so MAP-targeting iterations converge to cartoon-like images rather than realistic ones, and better results are obtained by stopping short of convergence. We turn this observation into a design principle and introduce $\texttt{ProxiMAP}$, an iterative MAP approximation whose noise schedule keeps the iterate's residual noise matched to the denoiser's training noise. This keeps the denoiser in-distribution where its score is reliable, and yields implicit early stopping that avoids the failure mode above. $\texttt{ProxiMAP}$ is a modular drop-in replacement for MMSE denoisers in standard PnP algorithms and consistently sharpens reconstructions across deblurring, inpainting, super-resolution, and phase retrieval. Building on the same principle, we propose a hybrid variant that applies $\texttt{ProxiMAP}$ only in the late iterations of PnP, where the denoiser is most reliable—matching or exceeding the full-replacement variant at a fraction of the cost.

We prove the identifiability of deep generative models (DGMs) with Piecewise-Affine (PWA) decoders and Gaussian Mixture Model (GMM) priors, in a purely unsupervised setting. We introduce three algebraic contrast principles for symmetry breaking: *domain contrast*, which trivializes the mixture symmetry group; *mechanism contrast*, which ensures every decoder branch is witnessed by a unique boundary; and *interaction contrast*, which forbids parameter conspiracies between latent components and decoder branches. Together they exploit the interplay between the discrete combinatorics of the PWA map and the continuous symmetry structure of the latent GMM. Continuity is replaced by algebraic symmetry conditions; injectivity is decoupled from structural identification and required only for pointwise inversion. Our results form a hierarchy: from *law identifiability* (LID; latent distribution up to a global affine map) through *map identifiability* (MID; decoder up to the same map) to *posterior* and *pointwise identifiability*. The ICA-form ambiguity emerges under diagonal covariance conditions; classical ICA is a further specialization under independence. Assumptions are only on the data-generating process, not on learning methods, except for the interaction contrast.

We study fixed-confidence best arm identification in generalized linear bandits under a hybrid feedback model: at each round, the learner may query either (i) absolute reward feedback from a single arm or (ii) relative (dueling) feedback from an arm pair, both governed by generalized linear models. We introduce a likelihood-ratio–based confidence sequence that unifies heterogeneous generalized linear observations and yields an explicit ellipsoidal confidence set under a self-concordance assumption. Building on this confidence set, we propose a hybrid Track-and-Stop algorithm that adaptively allocates queries by tracking a minimax-optimal design over a joint action space of arms and pairs. We establish $\delta$-correctness and provide high-probability upper bounds on the stopping time. We further extend the framework to a cost-aware setting that accounts for heterogeneous acquisition costs across feedback modalities. Empirical experiments demonstrate that the proposed algorithms significantly improve sample efficiency over baseline methods.

Reinforcement learning (RL) is now standard for post-training large language models. The same reward optimization that elicits useful capabilities, however, can also reinforce latent backdoors planted earlier in the pipeline: attack success rates under 0.2% after SFT rise to 40--98% on training-distribution inputs after RL, and generalize to 13--30\% on held-out evaluation, with no modification to the RL data, reward function, or training loop. We study this phenomenon in the context of agent models, where SFT poisoning teaches the victim model (Qwen3-8B and a small frontier model) to call an attacker-controlled oracle. Because the oracle can be set up to always return correct answers, calling it earns higher reward than the model's own attempts, and RL reinforces the behavior. We further show that the oracle's RL-time responses can instill persistent biases, such as brand preferences, that survive into deployment even when no tool calls happen. Such patterns appear difficult to detect with current tools: traditional guard models do not flag this mechanism, and LLM-based auditors remain unreliable, achieving only 4.5\% precision even after iterative prompt refinement. Our findings point to the value of post-RL safety evaluation, in particular scrutinizing tool-call patterns such as unnecessary external invocations.

Biosynthetic gene clusters (BGCs) are co-located genes that encode the biosynthesis of secondary metabolites, a major source of clinically used antibiotics and anticancer agents. Recent advances in deep learning, particularly self-supervised foundation models, have spurred growing interest in BGC sequence modeling, but evaluation infrastructure has lagged behind. Current practice often relies on coarse classification accuracy over small, experimentally curated datasets, making it difficult to discriminate model capabilities or assess downstream utility. We introduce BGC-Bench, an evaluation suite designed to resolve this limitation by systematically incorporating biochemical knowledge in the benchmark design. In terms of generalization, we test whether representations transfer to genetically and chemically novel samples; for functionality, we examine application-relevant tasks, including retrieval, prediction, and active learning. BGC-Bench reveals that no single model class dominates: pretrained foundation models perform strongly under out-of-domain generalization but remain sensitive to training conditions; compositional baselines are competitive and lead in cross-modal retrieval; and active-learning gains vary substantially across tasks, strategies, and model families. Beyond these BGC-specific findings, BGC-Bench highlights a broader principle: domain-informed benchmark design over existing datasets enables better evaluation in data-scarce scientific domains.

Causal experimental design often assumes direct control over treatment assignment, but many experiments can only assign encouragements that affect treatment receipt indirectly. We formulate this as adaptive instrumental variable (IV) design under noncompliance: the experimenter chooses unit--instrument queries, observes stochastic treatment realizations and outcomes, and seeks the treatment-level conditional average treatment effect (CATE). This creates an acquisition mismatch: uncertainty in prospective feedback does not necessarily correspond to uncertainty in the target causal estimand, so outcome-predictive or marginal variance-based criteria may select queries weakly informative for CATE. We introduce instrumental variable information gain (IVIG), a Bayesian acquisition principle that scores queries by the expected information their joint treatment-realization and outcome feedback provides about CATE values over a target population. Under the working Bayesian posterior, IVIG is Bayes-risk aligned under logarithmic loss; we also characterize the residual information discarded by outcome-only acquisition and connect greedy IVIG-style acquisition to target posterior-variance reduction under a fixed-covariance analysis. To approximate IVIG, we use an empirical-Bayes Gaussian Process IV posterior approximation with first-stage treatment-realization modeling and treatment-realization-specific fantasy updates. Experiments on synthetic and two semi-synthetic IV benchmarks show improved CATE sample efficiency over adaptive-design baselines; a real-label diagnostic shows improved recovery of full-data CATE estimates from observed IV data.

Feed-forward models for 3D reconstruction have achieved strong performance using deep cross-view attention to exchange information across images. However, these approaches often depend on heavy decoder stacks and lack a structured mechanism for geometry refinement, resulting in poor multi-view consistency. We address this by drawing inspiration from classical bundle adjustment (BA), which can be viewed as an iterative information propagation process between poses and local geometry. Inspired by BA, we propose BA-T, an iterative Transformer that implements BA-style structured updates as a repeatable layer in implicit token space. Instead of relying on deep attention stacks, BA-T refines predictions based on latent residual by a single lightweight layer. Experiments demonstrate that BA-T progressively improves pose and reconstruction accuracy across iterations, achieves stronger cross-view consistency than conventional decoders, and matches or surpasses substantially larger models while using only 16\% of their decoder parameters. BA-T provides a compact, efficient, and structural alternative to depth-heavy attention, enabling accurate 3D reconstruction within a lightweight architecture. The code will be made publicly.

Manifold embedding for data visualisation is dominated by a small set of hand-designed state-of-the-art (SOTA) algorithms. Their design targets are fixed at development time, resulting in inflexible algorithmic behaviour that cannot be readily steered toward user-specified structural priorities. Hand-designing new embedding algorithms to address user priority requires highly specialised expertise and is time consuming. To address autonomous algorithm design tailored to user preference, we introduce an agentic algorithm generation pipeline AutoManifold. It composes new manifold embedding algorithms from a constrained vocabulary of affinity, cost, and optimisation primitives, supported by multi-agent large-language-model (LLM) orchestration. AutoManifold conditions every stage of its design on user-specified structural preservation preferences, and iteratively refines the algorithm configuration through an LLM-guided, metric-grounded iterative loop. We compare the generated algorithms against three strongest and most frequently used SOTA (t-SNE, UMAP, and PaCMAP) on real-world datasets spanning different difficulty regimes under identical evaluation infrastructure. The results demonstrate that LLM-driven, priority-conditioned algorithm synthesis can move beyond hyperparameter tuning, producing genuinely new algorithms that outperform traditional hand-designed methods, in their ability to steer towards user-specified properties without compromising much inherent structure in the original data.