NeurIPS 2026 论文列表

Diffusion Transformers (DiTs) enable high-quality video generation but suffer from substantial inference latency, primarily attributable to the computationally expensive full spatio-temporal attention. While sparse attention methods offer potential solutions, existing approaches face an inherent flexibility--efficiency dilemma: predefined masks lack the flexibility to capture diverse attention patterns, while runtime-determined masks introduce overheads and sacrifice hardware efficiency. We identify the lack of a unified structural characterization of DiT attention as a key limitation of existing methods, and establish that video DiT attention exhibits periodic diagonal stripe structures along both temporal and spatial dimensions. To formally encode these structured patterns within a single efficient kernel, we present PSA, a parameterized stripe attention that formalizes the observed stripe regularity, unifying diverse attention patterns for efficient mask generation. This unified representation enables a single hardware-efficient CUDA kernel to process all sparse patterns, achieving FlashAttention-3-level Model FLOPs Utilization. To determine optimal sparsity configurations, we propose a training-free offline search algorithm that automatically maximizes sparsity under a specified error tolerance for each attention head. Experiments on HunyuanVideo and Wan 2.1 demonstrate that PSA achieves 1.57x and 1.37x end-to-end speedups over FlashAttention-3 baselines, with acceptable visual quality degradation.

Peer review is sometimes seen as noisy and difficult to predict. How much of the final accept/reject decision is recoverable from the paper itself? We investigate this question with PaperLens, text and vision models trained to predict binary acceptance from anonymized paper artifacts. For a clean evaluation, we construct balanced, shortcut-controlled OpenReview and source-derived arXiv datasets that remove deanonymization artifacts and majority-class shortcuts. Supervised binary decision training is surprisingly strong. PaperLens outperforms frontier prompting and review-trained baselines on decision accuracy and ranking, while its acceptance probability remains well calibrated after validation-set scaling and correlates better to ratings than baselines. Vision consistently improves over markdown text, showing that layout, figures, and visual presentation carry reviewer-relevant signal. Scaling from 3B to 14B parameters yields only modest gains, with clean supervision and faithful paper representations mattering more than model size. Together, our results show that decision prediction provides a powerful signal for paper assessment and can even improve the alignment of AI reviewing agents.

Multimodal large language models (MLLMs) still struggle with spatial understanding under the dominant perspective-image paradigm, which inherits the narrow field of view of human-like perception. For navigation, robotic search, and 3D scene understanding, 360$^\circ$ panoramic sensing offers a form of supersensing by capturing the entire surrounding environment at once. However, existing MLLM pipelines typically decompose panoramas into multiple perspective views, leaving the spherical structure of equirectangular projection (ERP) largely implicit. In this paper, we study pano-native understanding, which requires an MLLM to reason over an ERP panorama as a continuous, observer-centered space. To this end, we first define the key abilities for pano-native understanding, including semantic anchoring, spherical localization, reference-frame transformation, and depth-aware 3D spatial reasoning. We then build a large-scale metadata construction pipeline that converts mixed-source ERP panoramas into geometry-aware, language-grounded, and depth-aware supervision, and instantiate these signals as capability-aligned instruction tuning data. On the model side, we introduce PanoWorld with Spherical Spatial Cross-Attention, which injects spherical geometry into the visual stream. We further construct PanoSpace-Bench, a diagnostic benchmark for evaluating ERP-native spatial reasoning. Experiments show that PanoWorld substantially outperforms both proprietary and open-source baselines on PanoSpace-Bench, H$^\ast$Bench, and R2R-CE Val-Unseen benchmarks. These results demonstrate that robust panoramic reasoning requires dedicated pano-native supervision and geometry-aware model adaptation. All source code and proposed data will be publicly released.

Foundation models (FMs) have shown great promise in medical imaging, but most FMs are trained on unimodal data within isolated domains, such as brain MRI alone. Human aging and disease arise through coordinated biological processes across organs, therefore motivating multimodal FMs that learn whole-body representations. A key challenge, however, is that real-world multimodal biomedical data are often missing not at random, which can reduce power, limit generalizability, and introduce bias. We propose Pan-FM, a pan-organ foundation model pre-trained on imaging from seven organs (Brain, Heart, Adipose, Liver, Kidney, Spleen, and Pancreas) under realistic missing-organ scenarios. Pan-FM uses a unified backbone that handles organ missingness during both training and inference, and is pre-trained with masking-based self-distillation. We find that naive multimodal pre-training leads to dominant-organ shortcut learning bias, with the model over-relying on dominant organs such as adipose and heart. To address this, we introduce Saliency-Guided Masking (SGM), which uses the model attention distribution to adaptively mask dominant organs during pre-training, thus encouraging more balanced cross-organ, whole-body learning. Notably, SGM introduces negligible computational overhead and can be seamlessly integrated into existing self-supervised learning frameworks to improve multi-organ representation learning. On the UK Biobank, Pan-FM achieves stronger prediction across 13 disease categories and 14 single disease entities than single-organ and multi-organ baselines, with improved robustness under missing-organ settings. Pan-FM serves as a scalable solution to realistic modality-missingness in multimodal learning in system neuroscience and as a step toward more generalizable whole-body FMs.

Current safety alignment of foundation models largely follows a \emph{one-size-fits-all} paradigm, applying the same refusal policy across users and contexts. As a result, models may refuse requests that are unsafe for general users but legitimate for authorized professionals, limiting helpfulness in specialized professional settings. Existing approaches either require costly realignment or rely on inference-time steering that suffers from imprecise control and added latency. To this end, we propose \textsc{Palette}, a modular, controllable, and efficient framework that selectively relaxes refusal behavior on authorized target domains while preserving standard safety elsewhere. Our method identifies a refusal direction via multi-objective search and internalizes it into the model through lightweight adaptation. \textsc{Palette} further supports modular composition: it learns domain-specific safety controls independently and composes them through parameter merging, enabling on-demand multi-domain authorization without retraining. Experiments across four safety benchmarks, multiple model variants, and both LLMs and VLMs show that \textsc{Palette} delivers precise safety control without sacrificing general utility, offering a practical path toward foundation models that adapt to diverse professional needs.

Diffusion large language models (dLLMs) generate text by iteratively denoising masked token sequences. Although dLLMs can predict all masked positions in parallel within each step, the large number of denoising iterations still makes inference expensive. This cost can be reduced spatially by unmasking multiple tokens per step, or temporally by collapsing multiple denoising steps into one verification call. We propose Parallel Speculative Decoding (PSD), a training-free framework that jointly improves inference along both axes. Using the confidence scores from a single forward pass, PSD selects positions to unmask via a configurable, adaptive unmasking policy and constructs multi-depth speculative drafts without extra model calls. A final batched verification pass then applies hierarchical acceptance, keeping the deepest draft that remains consistent with the updated predictions. Experiments on three dLLMs across reasoning and code generation tasks show that PSD achieves favorable trade-offs between inference efficiency and generation quality, reaching up to $5.5\times$ tokens per forward pass with accuracy comparable to greedy decoding.

Text watermarking for large language models (LLMs) enables model owners to verify the origin and protect the intellectual property of AI-generated text. While text watermarking for closed-source LLMs is relatively mature, open-sourcing the LLM introduces unique challenges: existing methods inject the watermark during the model owner's decoding process, but for open-source LLMs, end users control decoding and can arbitrarily remove it at inference time. The watermark must therefore be embedded into the released model weights, so that standard decoding natively produces watermarked text. Recent methods attempt this by distilling from a decoding-based watermarked LLM, but two challenges remain: (i) learnability: the LLM may not perfectly internalize the predefined watermark pattern, producing a generation-detection inconsistency between what the LLM generates and what the detector expects; and (ii) robustness: open-sourcing the model exposes the embedded watermark to weight-level user modifications such as fine-tuning and model merging. A practical open-source LLM watermark must therefore align the LLM's learned watermark with the detection pattern, and remain robust against weight-level user modifications. Guided by these requirements, we propose PRO, a precise and robust open-source LLM watermarking method. PRO jointly trains a watermark policy model with the LLM so that the LLM's learned watermark aligns with the detection pattern, and further incorporates a perturbation-aware regularizer that improves robustness against weight-level modifications. Evaluated on several mainstream open-source LLMs, PRO improves watermark detectability to $0.92$ TPR at $1\%$ FPR while reducing perplexity by $20.5\%$ over the strongest prior method, and preserves detectability under aggressive weight-level modifications, e.g., AUC $0.87$ vs. $0.79$ under $0.5$-ratio model merging. The code is publicly available at https://anonymous.4open.science/r/PRO-44B7.

Parameter-Efficient Fine-Tuning (PEFT) has become a dominant paradigm for adapting Multimodal Large Language Models (MLLMs) to specific tasks, resulting in a proliferation of task-specific expert models. Merging these experts into one universal model offers a promising path to creating versatile systems without costly retraining. However, existing methods for full fine-tuning merging often falter in parameter-efficient model merging, as they manipulate weights directly while ignoring the intrinsic geometric structure of PEFT modules. When diverse tasks exhibit conflicting feature directions, this inevitably leads to destructive interference. To address this issue, we propose PRISM, a data-free framework that reframes the merging of PEFT modules as a spectral signal reconstruction problem. PRISM operates in two stages: (1) Spectral Pruning, which decouples task updates into singular components and retains only high-energy directions to attenuate task-irrelevant noise; and (2) Energy-Prioritized Orthogonal Reconstruction, which prioritizes dominant components and projects overlapping vectors onto orthogonal directions to eliminate inter-task interference. We curate a benchmark comprising diverse multimodal tasks to evaluate our method. Extensive experiments demonstrate that PRISM significantly outperforms state-of-the-art baselines. Our code and weights will be released.

Real-world image dehazing (RID) aims to remove haze-induced degradation from real scenes. This task remains challenging due to non-uniform haze distribution, spatially varying color shifts, and the scarcity of paired real hazy-clean data. In PRISM, we propose Proximal Scattering Atmosphere Reconstruction (PSAR), a physically structured framework that jointly reconstructs the clear scene and scattering variables under the atmospheric scattering model, making the restoration process more interpretable in complex real-world conditions. To bridge the synthetic-to-real gap, we design an online non-uniform haze synthesis pipeline and a Selective Self-Distillation Adaptation (SSDA) scheme for unpaired real-world scenarios, which enables the model to selectively learn from high-quality perceptual targets while leveraging its intrinsic scattering understanding to audit residual haze and guide self-refinement. Experiments on real-world benchmarks demonstrate that PRISM achieves competitive performance on RID tasks.

Vision-Language-Action (VLA) models promise a single language-conditioned policy adaptable to many manipulation behaviors, yet deployment additionally requires skill retention and long-horizon composition: a policy must retain previously acquired skills and reuse them inside composite tasks whose behavior changes across phases. The standard recipe absorbs every new task into one shared update, which provides neither a parameter-level locus for reusable skills nor a mechanism for invoking each skill along the trajectory. We observe that this missing structure is latent rather than absent: per-skill low-rank updates of a pretrained VLA occupy a robustly distinguishable, low-dimensional subspace, and a single composite-task LoRA concentrates roughly half of its Frobenius energy in this subspace, against a near-zero matched random-rotation control. Building on this geometry, we introduce PRISM, which represents reusable skills as parameter-level primitives: each skill family is stored as a dedicated low-rank action-expert adapter in a primitive bank, primitives are composed at inference by a per-chunk VLM-conditioned router that reads the prefix representation to track task progress, and the router is supervised by per-chunk labels from a deterministic, action-derived oracle. During composite training, only the vision encoder, the language LoRA, and the router are updated; every primitive is held frozen, so the per-skill action-expert parameters are preserved by construction. On RoboCasa365, PRISM preserves atomic-skill success through composite training and reaches 20% on long-horizon composite tasks, while a standard $\pi_{0.5}$-LoRA two-stage fine-tune instead loses roughly 25 atomic points and reaches only 11% composite. We additionally validate PRISM on an AgileX PIPER real-robot setup.

Multimodal large language models (MLLMs) leverage object-agnostic tokenization by default, which represents images as flat grids of patch tokens and uniformly discretizes semantically rich foreground regions and information-sparse background areas. This introduces substantial redundancy and constitutes a major source of inference cost. Existing object-level tokenizers could either discard intra-region saliency with predefined region grouping and coarse pooling or limit to a single semantic scale by employing slot attention to the final encoder feature map only. In this paper, we propose a two-stage framework named POST that decouples unsupervised object-centric grouping from language-grounded multimodal reasoning to achieve progressively semantic-aligned object-level visual tokenizers. Specifically, we develop Progressive Slot Attention (PSA) in Stage I to introduce slot attention across multiple layers of a frozen ViT encoder and propagates slot states through a cross-layer refinement chain for learning hierarchical slot--patch correspondences without segmentation supervision. Subsequently, we transfer the learned PSA to the LLaVA pipeline in Stage II for object-level visual tokenization. We design slot-weighted merging to aggregate ViT patch tokens into object-grounded visual tokens and remove low-support slots with adaptive slot pruning. Remarkably, POST eliminates the need for external segmentation models, enables progressive object-centric refinement, and adapts visual token budget to image complexity in visual tokenization. Experimental results show that PSA achieves strong unsupervised object-centric segmentation on PASCAL VOC and COCO. Furthermore, POST is comparable or superior to LLaVA-1.5 on most multimodal benchmarks using only 8.3% visual token budget and improves referring expression comprehension on RefCOCO/+/g. It also outperforms recent patch-level and object-level token compression baselines at comparable budgets.

Precision medicine relies on integrating diverse patient data, including genetic variants, clinical phenotypes, and medical imaging, to inform personalized prevention, diagnosis, and treatment. However, learning general patient-level representations from these heterogeneous modalities remains challenging due to their high dimensionality and pervasive missingness. We propose PM1, a multimodal foundation model that learns meaningful patient-level representations across genetic, clinical, and imaging data from 337,129 individuals in the UK Biobank, with a focus on retinal imaging and ophthalmic traits. To address modality dimensionality, PM1 adopts an intermediate fusion architecture with modality-specific encoders and a shared Transformer backbone, where a novel pathway-based genotype encoder enables the incorporation of over 600,000 genetic variants by reducing dimensionality while preserving trait-relevant variation. In addition, a contrastive loss invariant to modality missingness allows PM1 to learn robust patient-level representations despite only 6\% of UK Biobank participants having complete modality coverage. By partitioning modalities into interpretable tokens, PM1 enables interpretability methods to investigate known and potentially novel associations across genetic, clinical, and imaging features, and their interactions. Evaluated across downstream tasks, including phenotype prediction, conditioned image generation, and attribution analyses, PM1 demonstrates the value of multimodal patient representations, improving predictive performance over unimodal baselines and competing methods while enabling interpretable analyses in settings with heterogeneous and incomplete patient data.

While While Vision Foundation Models (VFMs) are increasingly used in several computer vision tasks, their internal representations remain substantially opaque. This mostly stems from polysemanticity, i.e., individual neurons encode mixtures of unrelated visual patterns. In stark contrast, human reasoning is inherently monosemantic, i.e., it naturally relies on distinct, isolated features to process information. Therefore, monosemanticity needs to be implemented within the model's activation space to obtain representations that are human-understandable concepts. Existing methods mostly operate on convolutional networks, so when applied to VFMs their "crop-and-resize" approach distorts the model's representations by disrupting global self-attention mechanisms and discarding spatial geometry. Furthermore, prior work requires downstream classification labels and is based on KL-divergence, which requires to propagate the gradients back from the classifier head. Ultimately incurring in excessive concept extraction time, and making it hardly applicable to extract task-agnostic concepts. To overcome these issues, we propose PLACE, a fully unsupervised concept extraction framework that operates directly on the native patch-token activations of a VFM. By introducing a geometric Gram-matrix alignment loss, PLACE mechanistically controls downstream KL divergence, thus enforcing faithfulness -- i.e., the notion that extracted concepts, although obtained in an unsupervised fashion, are relevant to downstream classification tasks. Evaluations on image classification tasks (ImageNet and COCO) on DINOv2, ViT-MAE and ViT-B/16 show that PLACE (i) using Gram-alignment it extracts concepts about $20\times$ faster compared to KL-supervised extraction and $11\times$ faster than Sparse Autoencoders (SAEs). Moreover, PLACE yields highly sparse concepts that (ii) achieve up to 40 percentage points sparser compared to the state of the art (i.e., max 0.933 Gini sparsity score), and (iii) are up to 92.3% similar to those obtained under classifier supervision (i.e., 0.923 cosine similarity).

Multi-modal cloud removal exploits complementary optical and synthetic aperture radar (SAR) images to restore cloud-degraded remote sensing images. Most existing methods rely on black-box, data-driven architectures with limited interpretability of cross-modal interactions. Although some attempts introduce physical priors into networks, their oversimplified formulations fail to faithfully characterize the underlying physical imaging process and the networks remain loosely coupled with optimization variables. To address these issues, we propose a physics-informed deep unfolding network for cloud removal via optical-SAR image fusion, termed as PIU-CR. We first formulate cloud removal as a physics-informed optimization problem. Specifically, we fundamentally ground the optimization in the physical imaging mechanism of cloud degradation by embedding the atmospheric scattering model into data fidelity term. Then, the regularization terms establish an explicit cross-modal interaction mechanism, overcoming the single-modal information scarcity and the obscure, uninterpretable interactions in black-box models. They are dedicated to SAR-guided structure and spectral regularization, jointly introducing cross-modal structural feature alignment and coordinated spectral consistency constraints. The resulting optimization problem is then unfolded into a multi-stage neural network with dedicated structure and spectral branches. Each stage corresponds to explict optimization iteration step, enabling interpretable and effective reconstruction process. Experiments demonstrate that the proposed method outperforms state-of-the-art methods.

Modern automatic-rigging predictors (UniRig, Make-It-Animatable, RigNet) place a skeleton on a 3D mesh in seconds, but the resulting skinning weights animate poorly. Linear-blend skinning (LBS) is non-linear in pose space, so noisy weights amplify into visible candy-wrapper artifacts---the surface twisting and contracting at joint bends---and volume collapse. ARAP-style remedies penalize these symptoms one pose at a time and cannot reach the cross-pose curvature that produces them. We propose \emph{Pose-Interpolation Smoothness} (PIS), a self-supervised, training-free, test-time refiner: minimize the per-vertex gap between the deformation at the SO(3) midpoint of two random poses and the linear average of the two endpoint deformations. The midpoint is obtained by orthogonally projecting the arithmetic mean of the two rotations back onto the rotation group, sidestepping the matrix-logarithm gradient that breaks down near identity and antipodal poses. Because PIS only re-tunes the weights LBS already consumes, it slots into any LBS-based pipeline---generative-3D outputs, automatic-rigging predictions, and the LBS pathways of game engines and AR/VR renderers---without touching shaders. On Objaverse-LVIS Balanced-102 (98 valid models, 51 LVIS categories), PIS reduces fragility (per-vertex deformation variance under random poses), ARAP energy, Jacobian anisotropy, and area distortion by $35.5$\%, $39.3$\%, $19.8$\%, and $43.2$\% on average; $93.9$\% of models improve on all four simultaneously, Wilcoxon $p<10^{-10}$. On the worst-case slice---the top decile of UniRig-initialized fragility, where animation visibly tears---PIS reduces unstable vertices by up to $10.2\times$. Effective bone count and weight entropy are unchanged, and the optimization runs in $\sim$30~s/model on a single H200 GPU.

Optical microscopy lets us watch living bacteria in real time without stains or labels, and it remains the standard instrument for identifying species across clinical, environmental and industrial microbiology. Yet real samples rarely hold a single species. They arrive as mixtures and sometimes carry an organism no model has ever seen. What happens when a model trained on familiar mixtures meets one it has never encountered? No benchmark has asked this of phase-contrast images, so we built one. **PHOEBI** brings together 120,000 phase-contrast images of 40 mixtures of six bacterial species that we cultured and imaged ourselves, and it holds whole mixtures out of training so that every test is a combination the model has never seen. The answer is sobering. Classifiers that are nearly perfect on the mixtures they know collapse on new ones, and no choice of backbone or pretraining rescues them. The features turn out to be sound and the failure lies in how the evidence within each image is pooled. A culture looks the same everywhere, and decoders built on this simple fact hold steady where large networks fail even though one of them is never trained by gradient descent. The same features go further still. They notice when a species outside the catalogue appears, and the simplest way of giving it a class of its own proves the most effective while leaving the known species almost untouched. PHOEBI turns the mixed culture, the everyday case at the bench, into a measurable target for open-world recognition.

Camera localization methods are mostly multi-stage pipelines relying on dedicated and complex maps that are costly to construct, store, and maintain over time. Alternatively, well-structured maps that organize scenes with compact yet expressive primitives are often readily available, providing rich structural and semantic cues that humans can interpret. However, their potential for direct camera localization remains largely unexplored due to the significant modality gap with visual observations. This motivates a new concept for localization, which estimates camera poses by *grounding query observations to map primitives*. In this paper, we instantiate the concept in the context of planar primitives with **PGGT**, the **P**rimitive-**G**rounded **G**eometry **T**ransformer network trained to predict multiple localization targets from a single query image and a set of planar map primitives, enabling feed-forward 6-DoF camera localization that generalizes to unseen environments. Based on this flexible and scalable architecture, our method pushes the state-of-the-art localization performance on both detailed planar surface maps and minimal architectural layouts. The code and models will be made publicly available.

Photonic crystal fibres (PCFs) underpin critical applications in telecommunications, sensing, and high-power laser delivery, yet their design remains a manual, expert-intensive process that couples geometric intuition with deep waveguide physics. Vision-language models (VLMs) offer a promising path toward automating this pipeline, but can they truly reason about photonic physics from microscopy images, or do they merely recognise visual patterns? We introduce PCFBench, the first comprehensive benchmark for evaluating VLMs on the complete photonic inverse-design workflow. PCFBench spans 26 tasks organised into six categories of increasing cognitive demand (Geometry Perception, Physics Understanding, Text Generation, Multi-modal Reasoning, Inverse Design, and Code Generation), built from over 220K FDTD simulation samples across 18 structurally diverse fibre families. We evaluate both individual VLMs on isolated tasks and multi-agent systems (MAS) on the full sequential pipeline from perception to design. Our results reveal a striking perception-design gap: current models can identify fibre geometry with moderate success, yet consistently fail to translate that visual understanding into physics-aware design actions. Moreover, cascading agents through the pipeline exposes significant error propagation, where upstream perception mistakes compound into downstream design failures. These findings suggest that neither model scaling nor naive agent chaining suffices; closing the loop demands explicit physical grounding. PCFBench provides the community with a rigorous, multi-dimensional testbed to measure progress toward AI-assisted photonic engineering.

Infrared-visible image fusion (IVIF) aims to combine visible and thermal infrared observations for human perception and downstream vision tasks. Visible and thermal infrared observations are physically complementary but unreliable in different ways. Existing deep IVIF methods mainly improve feature interaction and reconstruction, but often lack an explicit local rule for deciding when visible appearance should dominate and when thermal responses should contribute. To address this issue, we reformulate IVIF as local Bayesian posterior rendering, where fusion is controlled by reliability posteriors rather than direct modality mixing. In this paper, we propose Proposal-Context-Detail Fusion (PCDFusion), a visible-anchored framework that implements this formulation through three rendering stages. PCDFusion preserves potentially useful thermal responses, updates exposure-degraded visible regions, and renders reliable band-limited infrared structures into the final luminance. Extensive experiments on public IVIF benchmarks show that PCDFusion improves fusion quality, target recovery, and downstream detection and segmentation performance under challenging illumination. To support reproducibility, the code will be publicly released upon acceptance.