NeurIPS 2026 Papers

Post-training quantization (PTQ) is a key technique for efficiently deploying large language models, as it compresses weights into low-bit representations without retraining. Existing PTQ solvers, including GPTQ, LDLQ, and block coordinate descent, commit each weight block to a single codeword before its neighbors are considered, leaving near-optimal alternatives unexplored. In this work, we cast row-wise PTQ as posterior inference over discrete codebooks, in which existing solvers correspond to a zero-temperature, hard-decision limit retaining only the top codeword per block. We propose \emph{Message-Passing Quantization} (MPQ), which preserves soft posterior information across blocks during inference and commits to hard codewords only at the output stage. MPQ first uses vector approximate message passing to compute approximate blockwise posteriors, then refines a hard-decision initialization by exact block coordinate descent. Across 2-bit lattice and 3- and 4-bit scalar quantization on dense language models from Llama, Qwen, and Mistral families, MPQ improves perplexity and zero-shot accuracy over strong hard-decision baselines, and composes favorably with preprocessing techniques such as activation-aware scaling and incoherence processing.

Document parsing converts visually rich documents into machine-readable structured representations, forming a crucial foundation for information systems. Although many benchmarks have been proposed for document parsing, they remain inadequate for realistic scenarios. Existing benchmarks either focus on specific tasks or assess only single-page, text-centric settings, making them insufficient for practical multi-page parsing. Moreover, they lack fine-grained evaluation of semantic continuity, hierarchical structure recovery, and visual content preservation. To address these gaps, we propose MPDocBench-Parse, a benchmark for multi-page document parsing in real-world applications. It contains 433 manually annotated documents with 3,246 pages, covering 15 document types in English and Chinese, with diverse layout styles, and supports document-level end-to-end evaluation. We further design a comprehensive protocol for content fidelity and logical structure, covering text, table, and formula recognition, truncated text and table merging, figure extraction, reading order, and heading hierarchy recovery. Experiments show that, while existing models perform well on basic text extraction, they still suffer clear limitations in semantic continuity integration, visual content parsing, and hierarchical structure recovery. MPDocBench-Parse provides a unified foundation for advancing document parsing toward more realistic scenarios.

As language-model agents evolve from passive chatbots into proactive assistants that handle personal data, evaluating their adherence to social norms becomes increasingly critical, often through the lens of Contextual Integrity (CI). However, existing CI benchmarks are largely text-centric and primarily emphasize negative refusal scenarios, overlooking multimodal privacy risks and the fundamental trade-off between privacy and utility. In this paper, we introduce MPCI-Bench, the first Multimodal Pairwise Contextual Integrity benchmark for evaluating privacy behavior in agentic settings. MPCI-Bench consists of paired positive and negative instances derived from the same visual source and instantiated across three tiers: normative Seed judgments, context-rich Story reasoning, and executable agent action Traces. Data quality is ensured through a Tri-Principle Iterative Refinement pipeline. Evaluations of state-of-the-art multimodal models reveal systematic failures to balance privacy and utility and a pronounced modality leakage gap, where sensitive visual information is leaked more frequently than textual information.

Understanding and predicting wildlife movement is critical for ecology and conservation. While trajectory forecasting has advanced for human and vehicle movement, wildlife trajectories present distinct challenges: they are unconstrained in space, highly stochastic, and influenced by environmental conditions. We introduce MOVEBENCH, the first large-scale benchmark for probabilistic wildlife movement forecasting, containing 2.6M GPS locations from 800+ individuals across 110 species in 127 countries, paired with 1.6B environmental raster tiles capturing 160 covariates known or hypothesized to influence movement. We propose a probabilistic evaluation protocol for movement trajectory forecasts, addressing limitations of point-prediction metrics for inherently stochastic phenomena. Through comprehensive empirical evaluation of four method families across multiple temporal and spatial scales, we reveal that: (1) existing predictive methods generalize better to future timepoints than to unseen individuals, (2) deep learning approaches do not consistently outperform simpler baselines, and (3) environmental covariate selection significantly impacts performance. MOVEBENCH enables standardized evaluation of movement forecasting methods and provides a foundation for methodological advances on this ecologically important task.

Large language model (LLM) agents have shown strong potential in tackling complex, multi-step tasks, yet scaling them to long-horizon settings remains fundamentally constrained by two problems: context window saturation and uniform allocation of inference-time compute across steps of vastly different cognitive demands. We introduce **MOSAIC** (**M**ulti-Scale **O**rchestrated **S**cale-**A**daptive **I**nference **C**ontrol), a hierarchical agent framework that decomposes long-horizon decision making into three temporal scales: strategic, tactical, and operational, each governed by distinct context scopes and reasoning budgets. Central to MOSAIC is a *Scale Scheduler*, a lightweight policy trained via reinforcement learning that dynamically determines (i) which abstraction level should be invoked at each time step, and (ii) how many reasoning tokens to allocate, based on the estimated decision complexity. We further propose *inter-scale context distillation*, a bidirectional information compression mechanism that maintains coherent state representations across scales without redundant token consumption. Theoretically, we show that under a fixed total inference budget, MOSAIC's adaptive allocation policy achieves a strictly lower regret bound than any fixed-rate policy in a hierarchical semi-MDP formulation. Empirically, MOSAIC achieves state-of-the-art or competitive results on four diverse long-horizon benchmarks: SWE-Bench Verified, BrowseComp, WebArena, and GAIA (Level 2\&3), while reducing total inference tokens by 38--57\% relative to flat ReAct baselines. Ablation studies confirm that each component of multi-scale decomposition, adaptive scheduling, and context distillation contributes meaningfully to both performance and efficiency.

Causal representation learning (CRL) seeks to recover latent variables with identifiability guarantees, typically up to permutation and component-wise reparameterization under appropriate assumptions. However, identifiability does not imply interpretability: latent semantics are typically assigned post hoc by alignment with known ground-truth factors. This limitation is particularly acute in scientific time series, where underlying mechanisms are unknown and discovering interpretable structure is a primary goal. In contrast, scientific observations (such as residue-pair distances, climate indices, or process sensors) are inherently semantic, as they correspond to named physical quantities. This raises a key question: can the interpretability of observations be transferred to the identifiable latent space? We propose MOSAIC (Module discovery via Sparse Additive Identifiable Causal learning), a sparse temporal VAE that integrates temporal CRL identifiability with support recovery over observed variables. MOSAIC identifies latent variables via regime-conditioned temporal variation, and recovers for each latent a sparse set of associated observations through an additive decoder, yielding module-level interpretability. We show that ANOVA main-effect supports are identifiable under general smooth mixing functions, and provide finite-sample recovery guarantees for a tractable sparse-additive variant. Empirically, MOSAIC recovers domain-consistent variable groups across RNA molecular dynamics, solar wind, ENSO climate, the Tennessee Eastman process, and a synthetic tokamak benchmark, enabling interpretable discovery of latent mechanisms in scientific time series.

Concept Bottleneck Models (CBMs) provide interpretability but typically incur a penalty in accuracy. Recent CBMs rely on LLM-supplied concept vocabularies that are misaligned with the underlying image evidence and obscure dataset-specific discriminative signals. Augmenting these representations with task-specific linear probes recovers little of the lost accuracy. Methods that instead learn concepts directly from data also fall short of the supervised CLIP linear-probe ceiling. We introduce MOSAIC (Mixtures Of Sparse Anchored Interpretable Concepts), a CBM whose concepts are discovered directly from image representations via text-anchored optimal transport, a balanced-marginal pass that softly assigns each image to a small set of learned concepts while keeping them tethered to class-text embeddings, and then named post-hoc by a VLM. Across 5 datasets, MOSAIC matches the supervised CLIP linear-probe ceiling on most dataset-backbone settings and beats raw zero-shot CLIP by +5.2 to +24.4 top-1 and class-level retrieval R@1 by +5.7 to +27.0. Beyond classification, the discrete concept basis enables a sparse per-image decomposition that exposes which features drive each prediction and aggressive bottleneck compression to as little as 2.5% of dense CLIP.

Coding agents often pass per-prompt safety review yet ship exploitable code when their tasks are decomposed into routine engineering tickets. The challenge is structural: existing safety alignment evaluates overt requests in isolation, leaving models blind to malicious end-states that emerge from sequenced compliance with innocuous-looking requests. We introduce MOSAIC-Bench (Malicious Objectives Sequenced As Innocuous Compliance), a benchmark of 199 three-stage attack chains paired with deterministic exploit oracles on deployed software substrates (10 web-application substrates, 31 CWE classes, 5 programming languages) that treats both exploit ground truth and downstream reviewer protocol as first-class evaluation axes. On this benchmark, nine production coding agents from Anthropic, OpenAI, Google, Moonshot, Zhipu, and Minimax compose innocuous tickets at 53–86% end-to-end ASR with only two refusals across all staged runs. In a matched direct-prompt experiment over four frontier Claude/Codex agents, vulnerable-output rates fall to 0–20.4%: Claude primarily refuses, while Codex primarily hardens rather than emitting the vulnerable implementation — ticket staging silences both defense modes simultaneously. Downstream, code reviewer agents approve 25.8% of these confirmed-vulnerable cumulative diffs as routine PRs, and a full-context implementation protocol closes only ∼50% of the staged/direct gap, ruling out context fragmentation as the sole explanation. As a deployable but non-adaptive mitigation, reframing the reviewer as an adversarial pentester reduces evasion across the evaluated reviewer subset; pentester-framed evasion ranges from 3.0% to 17.6%, and an open-weight Gemma-4-E4B-it reviewer under this framing detects 88.4% of attacks on the dataset with a 4.6% false-positive rate measured on 608 real-world GitHub PRs. We publicly release our dataset at https://huggingface.co/datasets/MosaicBenchmark/mosaic-bench, and a verifiable and adaptable evaluation framework at https://github.com/mosaic-benchmark/mosaic-benchmark.

Out-of-distribution (OOD) detection is essential for deploying machine learning models in high-stakes scientific applications, yet methods that flag OOD inputs without retraining the model have been validated almost exclusively on image classification. We introduce MOOD (Materials Out-of-Distribution Detection), a benchmark for evaluating whether these post-hoc OOD detectors transfer to graph neural networks (GNNs) on materials property prediction. MOOD evaluates 15 post-hoc detectors and 2 supervised diagnostic probes across 4 GNN architectures, 5 MatBench tasks, and 15 physically motivated compositional and structural shifts, yielding over 200 evaluation settings and 4,000 AUROC measurements. Our evaluation reveals three key findings. First, current post-hoc detectors perform near chance; all 15 detectors achieve an AUROC around 0.5. Second, detector failures decompose into two regimes: method bottlenecks and encoder bottlenecks. Method bottlenecks arise when supervised probes can separate in-distribution (ID) from OOD samples but unsupervised detectors fail, meaning that the embeddings contain OOD signal that current detectors do not extract. Encoder bottlenecks arise when even supervised probes fail to separate ID from OOD samples, indicating that the relevant shift is not accessible in the frozen GNN representation. Third, when OOD prediction error exceeds ID error, the AUROC of distance-based and density-based detectors correlates with the size of that gap. Collectively, these findings indicate that reliable OOD detection in scientific domains requires representation learning strategies that explicitly preserve OOD relevant signal, particularly for tasks where current GNN architectures encode no extractable signal at all.

Training large text-to-image models requires high-quality, curated datasets with diverse content and detailed captions. Yet the cost and complexity of collecting, filtering, deduplicating, and re-captioning such corpora at scale hinder open and reproducible research in the field. We introduce MONET, an open *Apache 2.0* dataset of $103.8$M image–text pairs collected from 2.9B raw pairs across heterogeneous open sources through successive stages of safety filtering, domain-based filtering, exact and near-duplicate removal, and re-captioning with multiple vision-language models covering short to long-form descriptions, and further augmented with synthetically generated samples. Each image is shipped with pre-computed embeddings and annotations to accelerate downstream use. To validate the effectiveness of MONET, we train a 4B-parameter latent diffusion model *exclusively* on it and reach competitive GenEval and DPG scores, demonstrating that our dataset lowers the barrier to large-scale, reproducible text-to-image research.

Multi-modal image fusion is crucial for comprehensive scene representation, yet real-world degradations compromise its efficacy, necessitating degradation-robust fusion paradigms. Existing degradation-robust multi-modal image fusion methods are often hindered by cascaded sub-optimal solution or black-box opacity. To address these challenges, we propose a theory-inspired mutual-promoting deep unfolding framework. It reformulates degradation-robust image fusion as a joint optimization problem, utilizing the degradation model and modality generation mechanism to explicitly model both single-task priors and cross-task dependencies. By decomposing the complex multi-task optimization problem into task-related iterative subproblems, the framework establishes a bidirectional reciprocal information flow between restoration and fusion. This process is further unfolded into a multi-stage deep neural network, where each component explicitly corresponds to a specific mathematical operation. In our framework, the fused cross-modal prior actively regularizes the ill-posed restoration, while the purified modality features continuously refine the fusion output. It ensures a transparent architecture that combines the merits of both model-based and data-driven methodologies. Extensive experiments demonstrate that our paradigm achieves state-of-the-art performance while providing superior interpretability.

Modeling event sequences requires understanding when future events will occur and how event types causally influence one another over time. Existing temporal point process (TPP) models typically assume static or first-order dependence structures, which limits their ability to capture the dynamic and multi-order causal mechanisms commonly observed in real-world systems. We propose MOCHA, a Multi-Order Causal Hierarchical Architecture for multivariate TPP that jointly models time-varying causal structure and multi-hop influence propagation in continuous time. MOCHA learns latent dynamic weighted directed acyclic graphs (DAG) over event types, where acyclicity and sparsity constraints promote structurally valid causal graphs. Based on the learned graph, the model decomposes event dynamics into multi-order causal paths, and incorporate both direct influence and indirect propagation into the intensity function. The entire framework is end-to-end differentiable and optimized jointly for event prediction and causal structure discovery. Experiments on seven real-world datasets from different domains show that MOCHA consistently achieves superior negative log-likelihood, while also recovering dynamic causal patterns that align with domain knowledge. These results demonstrate that MOCHA provides an effective framework for dynamic causal structure learning in TPP.

Multi-objective Bayesian optimization (MOBO) traditionally aims to approximate the entire Pareto front, but real-world deployment is often limited to a small, representative subset of solutions due to resource constraints. Current two-stage methods first approximate the full front before selecting solutions, wasting evaluation budget. We formalize this problem as Multi-objective Bayesian Optimization with Cardinality-aware Pareto Selection (MOBO-CAPS), where the goal is to identify a set of exactly $M$ solutions with high hypervolume. We propose two practical cardinality-aware acquisition strategies: REHVI, a replacement-aware approximation to expected hypervolume improvement, and CHO-UCB, which constructs an optimistic $M$-point set by optimizing a collective UCB-based hypervolume objective. We prove a sublinear simple regret bound for the batched version of CHO-UCB, and empirically show that cardinality-aware acquisition design improves over standard MOBO followed by post-hoc subset selection across nine benchmarks.

Time-series analysis is increasingly studied in multimodal settings, yet most benchmarks pair temporal signals with text or with visualizations derived from the same signal. This leaves open whether multimodal large language models (MLLMs) can integrate time series with real-world visual evidence. We introduce **MMTA** (**m**ulti**m**odal **t**emporal **a**nalysis), a benchmark of tri-modal tasks with diverse cross-modal complementarity patterns among time series, text, and vision. MMTA spans nine domains — robotics, autonomous driving, human action, manufacturing, AR assistance, healthcare, recommendation, finance, and climate — under *video+TS* and *image+TS* layouts, with a 3,524-sample test split covering classification and forecasting. The benchmark standardizes heterogeneous real-world sources into a shared schema with domain-specific prompts, provenance documentation, and success-weighted metrics that penalize both wrong answers and failed generations. Evaluating eight open-source and proprietary MLLMs reveals a shared bottleneck: representing time series as text creates very long token sequences and weakens fine-grained alignment with visual evidence, especially in long-video settings. We then introduce **TimeOmni-v**, a multimodal LLM extension with a dedicated time-series encoder, temporal positional alignment, modality-aware token layout, and a forecasting head. TimeOmni-v improves long-video classification and time-series inference efficiency, showing that MMTA not only exposes representation and grounding failures but also motivates targeted model design.

Reusable skills have become a core substrate for improving agent capabilities, yet most existing skill packages encode reusable behavior primarily as textual prompts, executable code, or learned routines. For visual agents, however, procedural knowledge is inherently multimodal: reuse depends not only on what operation to perform, but also on recognizing the relevant state, interpreting visual evidence of progress or failure, and deciding what to do next. We formalize this requirement as \emph{multimodal procedural knowledge} and address three practical challenges: (I) \textbf{what} a multimodal skill package should contain; (II) \textbf{where} such packages can be derived from public interaction experience; and (III) \textbf{how} agents can consult multimodal evidence at inference time without excessive image context or over-anchoring to reference screenshots. We introduce \emph{MMSkills}, a framework for representing, generating, and using reusable multimodal procedures for runtime visual decision making. Each MMSkill is a compact, state-conditioned package that couples a textual procedure with runtime state cards and multi-view keyframes. To construct these packages, we develop an agentic trajectory-to-skill Generator that transforms public non-evaluation trajectories into reusable multimodal skills through workflow grouping, procedure induction, visual grounding, and meta-skill-guided auditing. To use them, we introduce a branch-loaded multimodal skill agent: selected state cards and keyframes are inspected in a temporary branch, aligned with the live environment, and distilled into structured guidance for the main agent. Experiments across GUI and game-based visual-agent benchmarks show that MMSkills consistently improve both frontier and smaller multimodal agents, suggesting that external multimodal procedural knowledge complements model-internal priors. The code is accessible in \href{https://anonymous.4open.science/r/MMSkills}{https://anonymous.4open.science/r/MMSkills}.

Driven by increasingly complex real-world applications, Retrieval-Augmented Generation (RAG) has evolved from textual pipelines to multimodal settings, comprising two complementary dimensions: knowledge organization and retrieval enhancement. However, existing methods still face challenges in high construction cost, context window limitations, and the lack of unified training across offline and online knowledge sources. To address these challenges, we propose MMGraph-Agent, a unified multimodal RAG framework based on cache-inspired multimodal knowledge hypergraphs. MMGraph-Agent enables extremely lightweight hypergraph construction with zero API overhead and dynamic memory-based retrieval that alleviates long-context issues. Experiments across offline, online, and joint retrieval settings demonstrate consistent improvements in performance, efficiency, and architectural unification. Our software and data are publicly available.

Recent advances in Vision Language Models (VLMs) have driven significant progress in visual reasoning. However, open-source multimodal models still lag behind proprietary systems, largely due to the lack of high-quality reasoning data. Existing datasets offer limited coverage of challenging domains such as STEM diagrams and visual puzzles, and lack consistent, long-form Chain-of-Thought (CoT) annotations essential for eliciting strong reasoning capabilities. To bridge this gap, we introduce MMFineReason, a large-scale multimodal reasoning dataset comprising 1.8M samples and 5.1B solution tokens, featuring high-quality reasoning annotations distilled from Qwen3-VL-235B-A22B-Thinking. The dataset is established via a systematic three-stage pipeline: (1) large-scale data collection and standardization, (2) CoT rationale generation, and (3) comprehensive selection based on reasoning quality and difficulty awareness. The resulting dataset spans STEM, visual puzzles, games, and complex diagrams, with each sample annotated with detailed, visually grounded reasoning traces. We fine-tune Qwen3-VL-Instruct on MMFineReason to develop MMFineReason-2B/4B/8B versions. Our models establish new state-of-the-art (SOTA) results for their size class. Notably, MMFineReason-4B surpasses Qwen3-VL-8B-Thinking, and MMFineReason-8B even outperforms Qwen3-VL-30B-A3B-Thinking while approaching Qwen3-VL-32B-Thinking, demonstrating remarkable parameter efficiency. Crucially, we uncover a "less is more" phenomenon via our difficulty-aware filtering strategy: a subset of just 7\% (123K samples) achieves performance comparable to the full dataset. Notably, we reveal a synergistic effect where reasoning-oriented data simultaneously boosts general capabilities.

As multimodal agents increasingly rely on visual perception to navigate complex digital workflows, the vulnerability of their reasoning-action cycles to cognitive manipulation has emerged as a critical security priority. This systemic reliance on visual-semantic grounding introduces a fundamental susceptibility to executive hijacking. Unlike traditional pixel-level adversarial noise, this threat operates at the cognitive-semantic level by synthesizing high-fidelity, deceptive UI elements—such as forged system alerts or mandatory compliance overlays—that seamlessly subvert an agent's safety alignment. We propose a targeted evaluation methodology that injects context-aware payloads into specific vulnerability windows within an agent's reasoning trace. By specifically changing the attack content within the observation space while strictly maintaining a static attack strategy, we demonstrate the consistent subversion of executive intent across diverse task trajectories. Building upon this, we introduce MMA-SafetyBench, the first comprehensive safety evaluation suite for multimodal agents. Comprising 1,083 adversarial trajectories, the benchmark evaluates these vulnerabilities across five ubiquitous environments: web automation, desktop/mobile OS control, visual document understanding, and long-form video reasoning. Our evaluation of nine state-of-the-art foundational models reveals a pervasive risk of executive hijacking, where agents disproportionately prioritize deceptive observations over benign textual constraints. With the Semantic Compromise Rate (SCR) reaching an alarming 98.09%, our findings expose a critical blind spot in current safety alignments and underscore the urgent need for cross-modal cognitive defenses in autonomous multimodal architectures.

Vision-Language Models increasingly make socially consequential judgments from image-text inputs, yet current evaluations often stop at verdict-level accuracy: whether the final label is human-aligned, not whether it follows from the correct evidence. We introduce MM-SCALE (Multimodal Moral SCALE), a benchmark for evidence-grounded moral judgment built around a design choice: each image is paired with multiple action scenarios, so models must compare how different actions interact with the same visual context rather than score images in isolation. MM-SCALE contains 8,444 image contexts and 21,977 action scenarios, each annotated with a 5-point moral acceptability rating and a modality-grounding label indicating whether the human judgment relied on text, image, or both. Three tasks evaluate whether models (i) assign calibrated scalar moral scores, (ii) preserve human orderings within a shared visual context, and (iii) ground rationales in the evidence source humans found critical to judgment. Across models, aggregate metrics overstate alignment: NDCG@5 remains high across models, while within-image pairwise accuracy remains only 0.53--0.62 even on scenario pairs whose human mean ratings differ by at least one point. CoT has inconsistent effects on calibration and does not reliably improve within-image ordering. Evidence-grounding evaluations further show that models often describe visual context without making it critical for the verdict. These results reveal an evidence-grounding failure in current VLMs and position MM-SCALE as an evaluation testbed for moral judgment under shared image contexts. This dataset includes potentially harmful and sensitive visual content. Images are intended solely for evaluation.