NeurIPS 2026 Papers

Prompt learning is a popular parameter-efficient method for adapting foundation models, but learned prompts are typically task-specific and fail to generalize to new classes, domains, or compositions of tasks. In this paper, we introduce a $\textbf{Diffusion Meta-Prompt (DMP)}$ model , a framework that models the distribution of learned prompts using diffusion models. DMP is trained only on a repository of previously learned prompts to synthesize new prompts conditioned on natural language task descriptions, without access to the task data. To improve the sampling stability, we introduce a test-time steering strategy for DMP, which uses the best training-selected prompt in the repository as a latent anchor during diffusion sampling, without retraining the DMP or accessing test classes. DMP improves generalization across classification, retrieval and text-to-image generation tasks, supports concept composition and negative prompting without explicit training. It reduces storage and inference costs by over 90% compared to prompt retrieval methods. For composite classification, DMP achieves upto $\textbf{2.0}$% average gain over prior meta-learning methods across 55 pairs of datasets with gains as high as $\textbf{8.5}$% on specific pairs such as Eurosat and Flowers. DMP also enhances cross-task generalization with $\sim$$\textbf{2-9}$% improvement for hierarchical classification task. We further provide a theoretical guarantee bounding the expected task loss of prompts sampled from a DMP.

While Large Vision-Language Models (LVLMs) excel at bounding-box grounding, they struggle with precise spatial tasks such as polygon grounding. We attribute this bottleneck to two properties of the vertex-level autoregressive (AR) paradigm used by current LVLMs: (1) errors in early-emitted vertices propagate uncorrected through the rest of the sequence, and (2) the model commits to local vertex placement before observing the full contour, leading to suboptimal allocation of a fixed vertex budget. We propose Diffusion Fine-Tuning (DFT), which removes the vertex-level sequential dependency by placing all 2N coordinate tokens under a single discrete-diffusion denoiser, while introducing a much shorter digit-level factorisation: each coordinate is decomposed into hundreds, tens, and units digits, and the reverse process is trained to predict them coarse-to-fine. We train this with a Hierarchical Curriculum Learning strategy that progressively refines loss supervision from macro-contour to per-pixel detail. Under matched fine-tuning protocols, DFT matches strong AR LVLMs on 2D bounding-box grounding and improves over them on 16-point polygon grounding; the same network and training recipe extend to 9-DoF monocular 3D bounding-box grounding under a 9-parameter coordinate parameterisation. A block-wise top-k decoder closes most of the latency gap to AR and recovers most of the quality lost when scaling beyond 16 vertices. We do not claim to eliminate sequential dependence in general: vertex-level O(N) AR decoding is replaced by K joint denoising steps (K=12 in this work), with the three-stage digit hierarchy entering as a training-time loss factorisation rather than an inference-time chain.

State estimation is a critical challenge in modeling dynamic systems. Traditional Kalman filters and data-driven methods have shown good results in complex, nonlinear systems. However, most existing methods optimize the posterior without considering prior modeling, which leaves room for improvement in state estimation accuracy, and predict high-dimensional Kalman parameters end-to-end, which lead to unstable training and poor convergence. To address this, we propose Diff-Kalman, which leverages state differences to predict state increments and scaling matrices for correcting both the prediction and update phases of the Kalman filter. Diff-Kalman combines traditional dynamic models with neural networks to adaptively correct system behavior. In the prediction phase, Diff-Kalman utilizes a Transformer-based architecture with self-attention to extract localized patterns from historical states, followed by cross-attention to predict future state increments. These increments are then fused with model-based priors from the dynamic model. In the update phase, a GRU-based Scaling Predictor (GSP) dynamically adjusts the Kalman covariance and gain matrices based on differential residuals, which improves the accuracy and robustness of the estimation. Experimental results on linear MOT trajectories and nonlinear Lorenz dynamics show that Diff-Kalman improves estimation accuracy over representative baselines.

Mechanistic interpretability seeks to explain neural network behavior by decomposing model computations into interpretable features and circuits. While transcoder-based circuit tracing has recently enabled detailed causal analyses of large language models, multimodal diffusion transformers for image generation remain comparatively opaque. We still lack tools for understanding how semantic information propagates across denoising steps and how text and image representations interact within double-stream MM-DiT architectures. Existing methods provide only partial insight: attention maps expose a limited view of token interactions, while sparse autoencoders can discover interpretable features but do not directly reveal how these features are transformed and composed through nonlinear MLP layers. In this work, we extend transcoder-based circuit tracing to multimodal diffusion transformers. We train timestep-conditioned transcoders that faithfully approximate the input–output behavior of MLP sublayers in FLUX.1[schnell]. By replacing MLPs with transcoders and linearizing the remaining computation, we obtain exact feature-to-feature attribution and recover compact, interpretable circuits. Empirically, our transcoders match or slightly outperform sparse autoencoders on the sparsity–faithfulness tradeoff. The resulting circuits reveal mechanisms underlying attribute binding and cross-stream semantic propagation, and provide causal explanations for systematic generation errors. Moreover, circuit-guided interventions are substantially more precise and effective than standard SAE-based steering. Our results demonstrate that transcoder-based circuit analysis is feasible for state-of-the-art diffusion transformers and provides a powerful framework for understanding and controlling multimodal generative models.

Compact medical multimodal models are the most realistic path to clinical deployment, yet they often produce fluent answers that are not actually grounded in the image. We show that this failure has a clear internal signature. In five compact medical backbones and across six medical VQA benchmarks, visual tokens lose their spatial diversity shortly after entering the language backbone, their residual updates are an order of magnitude smaller than text residuals, and attention on them freezes on content free regions of the image. Zeroing the visual residual stream barely changes the output, while zeroing the text residual stream destroys it. We give a short analytical account that links the collapse of visual similarity to the residual dominance ratio, and we propose VGrip, a set of three lightweight losses that act on the three failure points without modifying the backbone architecture or the inference procedure. We also introduce the Visual Dependency Score, a simple and benchmark agnostic measure of how much a model actually uses the image. On Qwen2.5-VL-7B, VGrip lifts accuracy by 4.7 points on average across the six benchmarks and more than doubles the Visual Dependency Score over a strong LoRA baseline, with gains that are stable from two to eight billion parameters.

Long-video question answering is challenging since the answer often hinges on a few decisive moments scattered throughout the video, while memory constraints force Multimodal Large Language Models (MLLMs) to sample frames at extremely low rates, resulting in extreme sparsity that risks omitting key evidence. Though aggressively enlarging the context size is intuitive, we argue that performance remains fundamentally constrained by the bias, $\textit{i.e.}$, the misleading cues caused by spurious linguistic correlations and salient yet irrelevant visual observations. In this paper, we first introduce S-MME, a benchmark designed to systematically diagnose shortcut-biased behavior in MLLMs. To mitigate this failure, we further propose a training-free framework, Counterfactual Long-video Evidence-Aware Reasoning (CLEAR). By treating the full video-question pair as a complete view, CLEAR estimates bias effects via other views by only keeping the question or locally salient visual clips. Then, we apply hidden-state intervention to mitigate the discrepancy at inference such that the correct answer can be predicted confidently. With experimental studies across multiple benchmarks, we show that our method is generalizable and can be integrated in different MLLMs, yielding consistent gains in prediction and robustness. We hope CLEAR can offer a complementary direction in test-time scaling for long-video understanding and will publish code.

Diffusion Transformers (DiTs) achieve state-of-the-art image generation quality but incur substantial memory and computational costs at inference. While aggressive Post-Training Quantization (PTQ) to 4-bit precision offers significant efficiency gains, it typically results in severe quality degradation. Existing approaches, including smoothing-based methods, mixed-precision schemes, rotation techniques, and low-rank residual methods, partially mitigate this issue but still leave a noticeable gap to FP16/BF16 performance. In this work, we introduce *DiRotQ*, a W4A4 PTQ framework that mitigates this degradation through rotation-aware activation quantization. DiRotQ identifies a low-rank subspace capturing dominant activation variance via Principal Component Analysis (PCA), preserving coefficients in this subspace at higher precision while quantizing the remaining components to 4-bit. Activations are rotated into the PCA basis at inference time using calibration-derived orthogonal transformations, while the inverse rotation is fused into the layer weights offline. Combined with GPTQ-based weight quantization, DiRotQ achieves an FID $(\downarrow)$ of $15.9$ and PSNR $(\uparrow)$ of $19.1$ dB on PixArt-$\Sigma$ over the MJHQ-30K dataset, outperforming the prior state-of-the-art SVDQuant (FID $18.9$, PSNR $17.6$) under the same INT W4A4 setting. Beyond standard metrics, we introduce a *VLM-as-a-Judge* evaluation protocol for diffusion model quantization, the first such evaluation in this setting, providing a more holistic assessment of perceptual quality and prompt alignment under aggressive compression. On the systems side, we implement a *Triton-based custom kernel* to enable efficient end-to-end inference, reducing memory usage of the 12B FLUX.1-dev model by $2.1\times$ and delivering $2.3\times$ speedup over the BF16 baseline, on a 24GB RTX 4090 GPU. Our anonymous codebase is available [here](https://anonymous.4open.science/r/DiRotQ-5BF6/).

Towards more general and human-like intelligence, large language models should seamlessly integrate both multilingual and multimodal capabilities; however, extending an existing multimodal model to many languages typically requires expensive multilingual multimodal data construction and repeated end-to-end retraining. We study a training-free alternative: injecting multilingual capability into an existing multimodal model by composing residual updates in the shared language model backbone. The key challenge is that multilingual and multimodal updates are heterogeneous, reflecting different functional roles in the shared model. To address this, we propose Direction- and Magnitude-aware Multilingual Multimodal merging (DiM$^3$), which selectively composes the two updates at each parameter dimension while preserving the original vision encoder and multimodal projector. Experiments on multilingual benchmarks in both text-only and vision-language settings, covering 57 languages across LLaVA- and Qwen-based backbones, show that DiM$^3$ consistently outperforms existing merging baselines, substantially improves multilingual performance over the original multimodal model, and remains competitive with dedicated multilingual multimodal fine-tuning while largely retaining general multimodal ability. We further show that DiM$^3$ can be directly applied to already trained multilingual multimodal models and still yield additional gains. Further interpretability analysis shows that DiM$^3$ primarily reshapes intermediate-layer semantic representations, strengthening cross-lingual alignment under both text-only and multimodal inputs while preserving higher-layer task-sensitive structure. Our repository is on https://anonymous.4open.science/r/w-C677/.

Vision Transformers (ViTs) have been widely adopted as visual encoders in multimodal models; however, the reliance on third-party pretrained checkpoints exposes these systems to backdoor attacks. Existing test-time detection methods rely on either the predicted label or intermediate embeddings, limiting their applicability across diverse tasks and incurring substantial computational overhead. In this work, we investigate how backdoor triggers affect the multi-head attention mechanism of ViTs and identify two distinct anomalous patterns: $\textit{complete attention hijacking}$, causing the attention to exhibit abnormally high inter-head spatial overlap, and $\textit{partial attention hijacking}$, causing an abnormally large variance in the concentration of the attention distribution. Motivated by these findings, we propose $\textbf{DetectViT}$, a test-time backdoor detection method that requires neither training data, model outputs, nor any prior knowledge of the trigger. DetectViT quantifies the two hijacking phenomena via an inter-head consistency score and an inter-head entropy variance score, with thresholds estimated from a small out-of-distribution (OOD) calibration set drawn from any source. Extensive experiments on four representative attacks, spanning diverse architectures (DeiT, CLIP, LLaVA) and downstream tasks (image classification and captioning), demonstrate that DetectViT substantially outperforms all baselines with as few as 64--128 OOD calibration samples, attaining a true positive rate of $100\%$ and a false positive rate of $0.17\%$ on BadCLIP. Notably, it introduces negligible additional overhead, thanks to its reliance solely on attention weights already computed during inference. We release our code at: https://anonymous.4open.science/r/DetectViT-81AC.

Accurately matching human intentions in 3D space is an important goal of artificial intelligence. Recently, 3D Intention Grounding (3D-IG) has emerged, aiming to localize target 3D objects given a natural-language intent. Unlike conventional visual grounding with descriptive referring expressions, 3D-IG intents are abstract and non-descriptive, making object localization substantially more challenging. This requires inferring latent functional requirements from non-descriptive intents and aligning them with object-level 3D representations. However, existing methods largely rely on implicit intent–object matching, leading to logical gaps and limited interpretability, robustness, and generalization. To address these challenges, we propose Chain-of-Causal Reasoning (CoCR), a causality-inspired functional dependency reasoning framework that explicitly bridges abstract intentions and candidate objects through intermediate functional requirements. Specifically, CoCR progressively decomposes complex intents into ordered functional requirements, thereby forming an explicit intent–function–object reasoning chain that links abstract intentions to object suitability. Building on this chain, we construct a causality-inspired functional dependency graph to model requirement--attribute relationships and introduce a causal-visual alignment module that aligns function-aware representations with the geometric-semantic features of 3D point clouds, enabling bidirectional verification between structured functional reasoning and visual evidence. Extensive experiments on 3D Intention Grounding and 3D Visual Grounding demonstrate that our method enhances intent-aware object localization.

Graphic designs, such as posters, advertisements, and infographics, are an important medium for communicating information and shaping understanding. Unlike natural images, they consist of layered elements with explicit compositional order. However, existing object detection models treat these elements as an unordered set, leaving compositional order unexploited. To address this limitation, we present **D**etect **A**nything in Graphic **D**esign (DAD), a model that formulates graphic design detection as compositional deconstruction. It decodes elements in compositional order, using lower-layer elements to better detect higher-layer ones. The key feature of DAD is amodal detection, which predicts the full bounding box of each element, including regions occluded by elements placed above it. Building on this formulation, we propose **Ele**ment **R**elative **P**olicy **O**ptimization (EleRPO), which extends GRPO from sequence-level supervision to element-level optimization. EleRPO provides fine-grained training signals that capture how each detected element contributes to overall detection quality, and works synergistically with compositional order to improve detection performance. To support training and evaluation, we build a dataset of 10 million graphic designs. Experiments show that DAD outperforms all baselines and achieves human-level performance in amodal detection, supporting effective image-to-layer decomposition. EleRPO consistently improves over GRPO across nine detection benchmarks.

Vision-Language Models (VLMs) excel at open-world 2D perception but struggle with precise metric spatial reasoning, a key requirement for applications like autonomous driving, navigation, and embodied intelligence. We identify this as a structural limitation: images are 2D projections of the 3D world, inherently suffering from projective ambiguity, while VLM components favor semantic understanding and rely on 2D positional bias, leading to Spatial Collapse. To address this, we propose Despa, a Depth-grounded spatial VLMs that injects depth-based geometric information via Geometric Positional Embedding (GPE) and Depth-aware Rotational Positional Embedding (DoPE). To expose masked limitations in current benchmarks, we construct SpaDataset (1.7M QA pairs) and SpaBench (2,051 QAs), to our knowledge the largest fine-grained training corpus and benchmark suite for metric spatial reasoning. They cover three difficulty levels and ten sub-tasks across diverse settings. The fine-tuned Despa-4B model consistently outperforms general-purpose, closed-source, and specialized VLMs on SpaBench by 27.9% over Qwen3-VL-235B-A22B and 31.1% over GPT-5.2, generalizes to MSMU (66.3%), RefSpatial-U (42.4%), and the multi-frame VSI-Bench (66.0%), and consistently lifts LLaVA-1.5, Qwen2.5-VL, and Qwen3-VL backbones, achieving the performance with only a 4B model. The code will be publicly released.

Distillation enables compact Vision-Language Models (VLMs) to obtain strong reasoning capabilities, yet the prompts driving this process are typically chosen via simple heuristics or aggregated from off-the-shelf datasets. We reveal a critical inefficiency in this approach: up to 69\% of the prompts in standard chart / document reasoning datasets are effectively $\textit{zero-delta}$, meaning the teacher and student already induce the exact same answer distribution. Training on these prompts provides minimal learning signal, causing student improvement to rapidly saturate regardless of data scale. To escape the zero-delta trap, we return to first principles: distillation fundamentally minimizes distributional divergence, and thus a prompt is valuable only if it exposes a functional capability gap between the teacher and student. We quantify this gap through $\textit{answer divergence}$ ($\Delta$), demonstrating that non-zero divergence is critical for effective scaling. Building on this insight, we propose a staged synthesis pipeline that repurposes existing datasets as seeds, actively targeting student failure modes to produce better prompts. The result is $\textbf{DeltaPrompts}$, a diverse dataset of 200k synthetic, high-divergence reasoning problems. We evaluate $\text{DeltaPrompts}$ across three distinct settings: on-policy distillation with the target teacher-student pair, transfer to a novel model family without regenerating the data, and off-policy fine-tuning of a non-reasoning model. Across all scenarios, $\text{DeltaPrompts}$ drives substantial gains, yielding up to 15\% relative improvement even on top of a highly-optimized reasoning model ($\textit{e.g.,}$ Qwen3-VL-8B-Thinking)---averaged over 10 benchmarks spanning chart, document and perception-centric reasoning.

Exemplar-based image editing applies a transformation defined by a source-target image pair to a new query image. Existing methods rely on a pair-of-pairs supervision paradigm, requiring two image pairs sharing the same edit semantics to learn the target transformation. This constraint makes training data difficult to curate at scale and limits generalization across diverse edit types. We propose Delta-Adapter, a method that learns transferable editing semantics under single-pair supervision, requiring no textual guidance. Rather than directly exposing the exemplar pair to the model, we leverage a pre-trained vision encoder to extract a semantic delta that encodes the visual transformation between the two images. This semantic delta is injected into a pre-trained image editing model via a Perceiver-based adapter. Since the target image is never directly visible to the model, it can serve as the prediction target, enabling single-pair supervision without requiring additional exemplar pairs. This formulation allows us to leverage existing large-scale editing datasets for training. To further promote faithful transformation transfer, we introduce a semantic delta consistency loss that aligns the semantic change of the generated output with the ground-truth semantic delta extracted from the exemplar pair. Extensive experiments demonstrate that Delta-Adapter consistently improves both editing accuracy and content consistency over four strong baselines on seen editing tasks, while also generalizing more effectively to unseen editing tasks. Code will be made publicly available.

Depth super-resolution (DSR) aims to recover a high-resolution (HR) depth map from its low-resolution (LR) counterpart. With color image guidance, this task is typically formulated as learning the residual between HR and LR in a low-dimensional feature space. However, this additive formulation is insufficient to accurately capture the complex relationship between HR and LR, especially under spatially varying degradations. In this paper, we introduce \textbf{DegBins}, a novel DSR framework that leverages degradation-driven binning to adaptively enhance residual modeling. Specifically, DegBins reformulates the regression-based DSR as a hybrid classification-regression problem, where the residual depth is represented as a linear combination of discrete depth bins weighted by their learned probability distribution, yielding more flexible and expressive representations. Furthermore, DegBins models the degradation relationship between HR and LR in a high-dimensional feature space, enabling adaptive bin range adjustment and probability optimization conditioned on local degradation characteristics. To progressively improve reconstruction quality, DegBins adopts a multi-stage refinement scheme, where each stage performs finer-grained bin partitioning and probability updating based on the former estimation. This coarse-to-fine design facilitates more accurate depth recovery, particularly in regions with severe degradations or complex structural variations. Extensive experiments across five benchmarks demonstrate that DegBins consistently outperforms existing state-of-the-art methods in terms of accuracy, robustness, and generalization.

The optimal transport (OT) map provides a geometric transformation for aligning probability distributions and has become a useful tool in machine learning. However, existing estimators of the OT map still exhibit a gap between sharp statistical guarantees and practical parametric estimation based on stable training objectives. Theoretical estimators achieve minimax optimal convergence rates, but they are typically nonparametric and can incur demanding implementation design or inference costs. Practical estimators are parametric and scalable, but their statistical guarantees remain underexplored, and their min-max, adversarial-like training objectives can be sensitive to optimization algorithms. We propose BROT (Barycentric Regression for OT), a simple two-step method that first computes the unregularized OT plan and then fits a deep neural network (DNN) to the induced barycentric targets by least-squares regression. Under standard regularity conditions, we prove that the DNN estimator of BROT attains the minimax convergence rate, when the ground-truth OT map is Lipschitz. Numerical studies on synthetic datasets and an image dataset show that BROT provides accurate map estimates, strong target distribution matching, and competitive transport costs, compared to existing estimation methods. Experiments on two downstream tasks, single-cell perturbation prediction and unsupervised domain adaptation, further suggest that the accurate estimation of BROT can translate into stronger task performance.

Cross-modal hashing is essential for efffcient large-scale re-trieval in multimedia analysis. However, existing unsupervised methods often struggle to fully capture interactions between modalities, enforce multi-level alignment, and maintain stable optimization. To overcome these challenges, we propose Decoupled Prototype Contrastive Align-ment Hashing (DPCAH), which employs a cross-modal semantic con-trastive learning module that concatenates image and text features and encodes their interactions via a transformer to produce uniffed represen-tations. feature-level and hash-level contrastive objectives jointly align semantic information and guide discriminative hash code learning across modalities. A decoupled prototype consistency module further models cross-modal correlations independently, enhancing semantic alignment while ensuring stable and robust optimization. Experiments on three benchmark datasets demonstrate that DPCAH outperforms state-of-the-art methods in cross-modal retrieval.

Prompt learning adapts vision-language models such as CLIP by optimizing a small set of continuous context vectors. The cross-entropy objective drives the learned prompt toward base-class specialization, while generalization to unseen classes benefits from staying close to the zero-shot feature space. Because both objectives act on the same parameters, their gradients oppose each other at convergence, and single-prompt losses are confined to a fixed empirical base-novel trade-off curve across a wide range of loss designs. We propose Decoupled Mode Connectivity (DMC), which resolves this conflict by assigning each objective to a dedicated prompt. A linear mode connectivity (LMC) corridor in text-feature space enforces low classification loss across interpolated classifiers between the two endpoints. The Visual Anchor regularizer preserves CLIP's pretrained class-similarity structure during specialization; equivalently, it minimizes the KL divergence between the learned and pretrained class-similarity distributions. We introduce class-permutation invariance (CPI) as a necessary condition for regularizer transfer across the base-novel boundary, and prove via Fano's inequality that any CPI violation lower-bounds the drop in novel accuracy by a term proportional to the mutual information between the prompt and base-class labels. DMC improves base and novel accuracy on the majority of dataset-baseline combinations across 11 datasets and three prompt-learning baselines~(CoOp, KgCoOp, MMA), shifting the Pareto frontier rather than trading along it.

Direct Preference Optimization (DPO) enables offline alignment of diffusion models from preference pairs without explicit reward modeling, but scaling DPO requires large amounts of expensive human preference data. In this paper, we consider a semi-supervised approach where only a small subset of preference pairs is annotated by humans, while the remaining (much larger) pool is unlabeled and annotated by inexpensive synthetic feedback e.g., vision-language models scoring image pairs or self-training from the current model. However, such synthetic supervision is not a replacement for human judgments: it can be systematically misaligned, so naively mixing human and synthetic preferences will yield misaligned inference. We introduce DeDPO, a debiased objective that integrates a doubly robust estimator from causal inference into DPO to correct synthetic-label misalignment while preserving the simplicity of DPO. Experiments show improved label efficiency and robustness to synthetic labeler quality, allowing DeDPO to outperform standard DPO under the same human-label budget and to approach fully human-supervised performance, despite using four times fewer human-labeled data.