NeurIPS 2026 Papers

Why does the low dimensionality of representations, typically $d\approx 1000$, not prevent modern embedding-based retrieval models from scaling to billions, or even trillions, of data points? To answer this question, we study maximal-margin embeddings in the following retrieval model, classically studied in communication complexity [PS86] and more recently in embedding-based retrieval [WBNL25]. Let $A$ be a binary $N\times n$ matrix indicating whether each of $N$ queries is relevant to each of $n$ documents. We are interested in the largest margin $m>0,$ denoted by $\mathsf{m}^{\mathsf{rd}}(d, A),$ for which there exist unit norm embeddings of the queries and documents $U_1, U_2, \ldots, U_N, V_1, V_2, \ldots, V_n$ with the following property. $\langle U_j, V_i\rangle \ge m$ whenever $A_{ji} = 1$ and $\langle U_j, V_i\rangle \le -m$ when $A_{ji} = 0.$ A large margin is a key proxy for representation quality: it controls both robustness to perturbations and compositional generalization across queries. Our main theorem establishes that the best possible margin without a restriction on the dimension, $\mathsf{m}^{\mathsf{rd}}(+\infty, A),$ can be nearly achieved in dimension $d = O(\mathsf{m}^{\mathsf{rd}}(+\infty, A)^{-2}\log n)$ which improves a theorem of [BDES02]. Together with a matching lower bound in Theorem 1.5, we conclude that when $A$ is the $\binom{n}{k}\times n$ matrix containing all possible $k$-sparse rows once, dimension $d = O(k\log (n/k))$ is necessary and sufficient for the maximal possible margin $\mathsf{m}^{\mathsf{rd}}(+\infty, A) = \Theta(k^{-1/2})$ in this setting. This fully resolves the setup of [WBNL25]. We also give several constructions for large margins when $d = o(k\log (n/k)).$ Our proofs are based on connections between this problem and the literature on compressed sensing and the restricted isometry property. Finally, we empirically test the InfoNCE and sigmoid losses for producing large margin embeddings and demonstrate a clear advantage of the sigmoid loss.

Text watermarking helps identify AI-generated content, but its effect on factual reliability remains underexplored. In this paper, we study watermarking hallucination: factual errors induced or amplified by watermarking even when the required evidence is present in the context and the unwatermarked model can answer correctly. Using a controlled RAG setting, we compare paired unwatermarked and watermarked generations under the same condition. Across representative watermarking methods, including KGW, SWEET, DiPmark, GumbelSoft, Gumbel-Max, and SynthID-style watermarking, we find that watermarking hallucination is widespread: watermarked outputs can remain fluent while introducing factual errors. We attribute this failure mode to two mechanisms: (1) direct token-level bias, which can suppress fact-consistent tokens, and (2) prefix-induced drift, which accumulates through autoregressive decoding and weakens later attention to factual context. Motivated by this analysis, we propose Fact-Preserving Token Intervention (FPTI) and Fact-Preserving Attention Intervention (FPAI), two plug-in interventions that can be integrated into existing watermarking methods to improve factuality. Our experiments show that combining FPTI and FPAI mitigates around 90\% of watermark-induced hallucinations while preserving fluency and comparable decoding efficiency. Overall, this work highlights factuality as a first-class criterion in watermark evaluation, alongside detectability and robustness, and calls for careful factuality validation before deploying watermarks in fact-critical applications.

Foundation models are billion-parameter neural networks that are trained under statistical learning principles to reason inductively regarding multifarious tasks at inference. However, foundation models are increasingly deployed in settings where the statistical principles that they were trained on may not hold. During deployment, users intervene in these systems' functionality through prompts, instructions, demonstrations, analogies, retrieval, tool calls, memory, verification, search, and multi-step reasoning. These operations change the effective knowledge state available at inference time, all of which are not well described by induction alone. This paper argues that Michalski’s Inferential Theory of Learning (ITL) provides a useful conceptual basis for describing deployed foundation models as multistrategy, goal-directed inference systems. In ITL, learning is modeled as the transformation of knowledge through transmutations such as generalization, specialization, abstraction, concretion, association, discrimination, explanation, derivation, reformulation, insertion, deletion, replication, and sorting. We reinterpret these classical transmutations in the context of modern test-time computation techniques, showing that the functionality of reasoning models, intelligent prompting, retrieval-augmented generation, tool use, memory, and agentic search can be understood as transmutation programs over knowledge states. We then identify limitations of classical ITL for contemporary machine learning and propose a probabilistic extension in which transmutations are stochastic operators selected under uncertainty and cost. Finally, we connect this view to a system-level information bottleneck principle, quantifying the new knowledge that is acquired during the inference process. The resulting framework positions ITL as a bridge between statistical learning, symbolic reasoning, abductive explanations, prompting, and agentic foundation models.

Code generation and comprehension by Large Language Models (LLMs) have emerged as core drivers of industrial intelligence and decision optimization, finding widespread application in fields such as finance, automation, and aerospace. Although recent advancements have demonstrated the remarkable potential of LLMs in general code generation, existing benchmarks are mainly confined to single domains and languages. Consequently, they fail to effectively evaluate the generalization capabilities required for real-world industrial applications or to reflect the coding proficiency demanded by complex industrial scenarios. To bridge this gap, we introduce IndustryCode, the first comprehensive benchmark designed to span multiple industrial domains and programming languages. IndustryCode comprises 579 sub-problems derived from 125 primary industrial challenges, accompanied by rigorous problem descriptions and test cases. It covers a wide range of NAICS sectors, including Aerospace Product Mfg, Chemical Mfg, and Information, and incorporates diverse programming languages such as MATLAB, Python, C++, and Stata. In our evaluation, the top-performing model, Claude 4.7 Opus, achieved an overall accuracy of 71.9\% on sub-problems and 48.9\% on main problems. The benchmark dataset and automated evaluation code are now publicly available.

Clinical case search aims to retrieve disease-consistent prior cases from electronic health record databases for a natural-language patient query under a retrospective retrieval objective. The task is challenging because patient queries contain heterogeneous evidence, including symptoms, diagnoses, laboratory results, treatments, and timelines. These fields are often only partially specified, and fixed retrieval pipelines cannot adapt from prior search trajectories. Existing clinical retrieval methods typically flatten the query into a single representation, use static retrieval pipelines, or lack an explicit mechanism for cross-case adaptation. \textbf{PatientSearch-SE} addresses retrospective disease-centric clinical case search with three components: a \emph{sufficiency-aware planner} that separates dimensions suitable for direct matching from those requiring hypothesis completion, a \emph{hierarchical tool orchestration} module that maps these states to retrieval and re-ranking actions, and a \emph{trajectory-guided memory consolidation} mechanism that writes reusable disease cards and strategy templates without parameter updates. We also propose an evaluation framework that combines disease-centric retrieval outcomes with trajectory-level process probes for retrieval behavior analysis. Experiments on PMC-Patients and MIMIC-IV show higher disease-centric retrieval metrics for PatientSearch-SE than for the compared same-interface sparse, dense, and reasoning-based baselines, with larger gains on PMC-Patients and more modest gains on MIMIC-IV. Its memory-accumulation results are consistent with controlled external-memory adaptation, and its retrieved cases provide additional evidence for diagnosis-masked downstream few-shot RAG prediction under this retrospective protocol.

In-context learning has recently been linked to implicit gradient descent in linear self-attention models, suggesting that context can induce a forward-pass update. Retrieval-augmented generation (RAG) also relies on context, but retrieved documents are usually treated as static evidence rather than signals for adaptation. We study RAG as an in-context optimization process. First, we show that one linear self-attention layer can implement one gradient-descent step on a unified linearized RAG objective covering both projection-based and dot-product retrieval interfaces. This gives an exact regime where retrieval-augmented prediction and in-context optimization coincide. We use this result not as a literal model of LLM computation, but as a guide for adapting the interaction between queries and retrieved evidence. We then test the boundary of this correspondence: it remains stable under controlled linear extensions, but becomes feature-distribution dependent under nonlinear architectures. Finally, we turn this view into a lightweight method for frozen RAG LLMs. The method keeps the retriever and backbone fixed, and predicts a context-conditioned update to a generator-side evidence-use interface. Across seven QA benchmarks, two retrievers, and two frozen LLM backbones, this forward-only update improves a shared-interface baseline, transfers to held-out tasks, and approaches test-time gradient adaptation at much lower per-query cost

The Allen--Cahn equation is one of the standard phase-field models for mean-curvature flow (MCF), parameterised by the diffuse-interface width $\varepsilon$. Practical simulations span a working range of $\varepsilon$, so a single model covering this range is needed. PDE-specific neural operators are accurate but must be retrained at every $\varepsilon$; general neural operators cover $\varepsilon$ in one model but lose the phase-field structure of the target. We introduce Interface-Augmented Diffusion-Reaction (IADR), a neural operator that addresses both shortcomings: a target network with a diffusion-and-reaction structure preserves the phase-field structure of the Allen--Cahn equation, while a hypernetwork that maps $\varepsilon$ to the weights of this target network amortises the dependence on $\varepsilon$ across the working range. A single IADR matches the per-$\varepsilon$ specialists on in-distribution data, retains their cross-shape generalisation under both $\varepsilon$- and geometry-out-of-distribution shifts, and runs at the per-$\varepsilon$ specialist's per-step inference cost. We further establish a $K$-step rollout error bound that aligns with the long-horizon behaviour observed in experiments.

Large language models are increasingly capable of writing and executing code, suggesting a natural paradigm for tool-augmented agents: expressing tool use as programs rather than isolated calls. Programmatic tool calling (PTC) captures this idea through loops, batching, abstraction, and structured computation, but existing PTC approaches are largely stateless, limiting their ability to reuse intermediate results and adapt over time. This is problematic for real-world tasks, where tool environments and problem-solving processes are inherently stateful. We therefore present I-PTC, an interactive framework that executes stateful PTC: models issue code snippets that call tools, maintain reusable state, and evolve the tool interface through API corrections. To evaluate the effectiveness of I-PTC and the general PTC setting, we introduce PTC-BENCH, a benchmark of long-tail, production-style user requests whose difficulty comes from large initial environments, many dependent API interactions, extended planning, and recovery across follow-ups and transient failures. Experiments on PTC-BENCH show that I-PTC improves performance on complex multi-step tasks with large tool sets while substantially reducing token consumption over conventional tool-calling baselines and the original PTC baseline, highlighting stateful programmatic tool calling as a promising direction for scalable tool-augmented agents.

Large language models (LLMs) increasingly help people solve problems, from debugging code to repairing machinery. This process requires generating plausible hypotheses from partial descriptions, then updating them as more information arrives. Yet how LLMs perform this form of inference, and how close it is to optimal, remains unclear. We study this question in the number game, a controlled setting in which a learner infers the hypothesis supported by a few positive integers, such as $\{16, 8, 2, 64\}$: a rule like powers of 2 or an interval like numbers near 20. We measure the posterior over hypotheses using three complementary probes: posterior prediction, hypothesis evaluation, and hypothesis generation. We then compare LLM behavior with an optimal Bayesian model and human behavior, and test whether the same posterior is expressed across probes. LLMs are often well described by a two-parameter Bayesian fit, but with systematic offsets: by default they show a strong-sampling assumption that creates an implicit Occam's razor, favoring narrower hypotheses, while thinking mode shifts them toward greater prior reliance. We also find a robust evaluation--generation gap: LLMs select more correct hypotheses during hypothesis evaluation but generate simpler, more rule-like hypotheses. Finally, this Bayesian-with-bias pattern does not extrapolate. Models can behave as if they hold rule-like hypotheses over observed examples, yet generalize poorly to parts of the hypothesis domain not covered by those examples. Our results highlight a limitation of LLMs as general problem solvers, especially for scientific inference, where hypotheses must go beyond the data.

Retrieval-augmented in-context learning faces a budget problem under distribution shift: increasing top-$k$ can improve evidence coverage, but it can also add off-direction, conflicting, or overly long context that a finite reader cannot use. We introduce a theory-guided diagnostic framework based on \emph{effective evidence coverage}: retrieval is useful only while aligned evidence grows faster than reader-facing ambiguity. In a local linear-ICL model, coupled covariate and task-prior shifts create a mixed risk term that aligned retrieval can reduce up to a residual ambiguity term; a conditional transfer result extends the same budget boundary to frozen readers satisfying a prompt-level stability condition. This yields a falsifiable prediction: recall can keep rising after answer accuracy has saturated or declined. Across QA, verification, and NLI tasks, dense top-$k$ sweeps together with oracle and shuffled-context controls expose this recall--accuracy decoupling. The resulting framework separates coverage- and saturation-limited QA from interference-limited verification/NLI, while calibrated selected-context controllers are reported only as operational probes of the diagnosis.

Data augmentation is widely recognized for improving generalization in deep networks, yet its impact on the geometry of learned representations remains poorly understood. In this work, we characterize how different data augmentation strategies reshape internal representations in neural networks. Using tools from shape analysis, we embed network hidden representations into a metric space where distance is invariant to scaling, translation, rotation and reflection. We show that increasing augmentation strength leads to well-behaved trajectories in this space, and that different augmentation types steer representations in distinct directions. Moreover, we investigate how neural representation shapes are distorted along data augmentation trajectories, and show that insights from neural geometry can predict which representations provide the most improvement when ensembling models. Our results reveal shared geometric patterns across architectures and seeds, and suggest that analyzing shape-space trajectories offers a principled tool for understanding and comparing data augmentation methods.

Hallucination remains a central failure mode of large language models, but existing benchmarks operationalize it inconsistently across tasks such as summarization, question answering, retrieval-augmented generation, and agentic interaction. This fragmentation makes it unclear whether a mitigation that works in one setting actually reduces hallucinations across contexts. Current hallucination benchmarks either require human annotation and fixed references that may eventually be memorized, or rely on naturalistic observations which are often recorded in settings that are difficult to reproduce or test systematically. To enable further research on the root causes of hallucination, we introduce HalluWorld, an extensible benchmark framework grounded in an explicit reference-world formulation: a model hallucinates when it produces an observable claim that is false with respect to this reference world. Building on this view, we construct a family of synthetic and semi-synthetic benchmark environments in which the reference world is fully specified, the model's observable view is controlled, and hallucination labels can be generated automatically by construction. HalluWorld spans multiple settings that are classically representative for AI, i.e., gridworlds, chess, and realistic terminal tasks. This enables controlled variation of key factors such as world complexity, observability, temporal change, and source-conflict policy, allowing us to disentangle hallucinations into more fine-grained error categories. We evaluate frontier and open-weight language models across these settings and find consistent patterns across domains: perceptual hallucination on directly observed information is near-solved for frontier models, while multi-step state tracking and causal forward simulation are still difficult for frontier models, and are not generally solved by extended thinking. In the terminal setting specifically, models also struggle with when to abstain from answering. The uneven profile of failures across probe types and domains suggest that different hallucinations arise from qualitatively distinct failure modes rather than reflecting a single underlying capability. Our results suggest that controlled reference worlds offer a scalable and reproducible path toward measuring and reducing hallucinations in modern language models.

When a text description is extended with an additional detail, image-text similarity should drop if that detail is wrong. We show that CLIP-style dual encoders often violate this intuition: appending a plausible but incorrect object or relation to an otherwise correct description can increase the similarity score. We call such cases \emph{half-truths}. On the verified COCO evaluation set, CLIP prefers the correct shorter description only 41.0\% of the time, and performance drops to 28.2\% when the added detail is a relation. We trace this vulnerability to weak supervision on caption parts: contrastive training aligns full sentences but does not explicitly enforce that individual entities and relations are grounded. We propose CS-CLIP (Component-Supervised CLIP), which decomposes captions into entity and relation units, constructs a minimally edited foil for each unit, and fine-tunes the model to score the correct unit above its foil while preserving standard dual-encoder inference. CS-CLIP raises half-truth accuracy to 78.2\% and improves average performance on established compositional benchmarks by 5.7 points, suggesting that reducing half-truth errors aligns with broader gains in compositional understanding.

In standard acoustic anomaly detection (ASD), models are usually trained separately for each machine or domain using explicit metadata. However, in realistic deployments, machine identifiers are incomplete, unreliable, or simply unavailable. Thus, metadata-free universal ASD asks a single model to monitor mixed machine populations without machine IDs. In this paper, we argue that the main failure mode is not a weak detector family, but representation entanglement. The heterogeneous normal modes overlap in frozen pre-trained feature spaces, making anomaly scoring and approximate retrieval unreliable. We study this problem directly and propose HARMONY (Hierarchical Anchor Retrieval on Manifold for Oblivious-source acoustic aNomaly detection), an anchor-guided geometric repartitioning framework that reorganizes mixed-source features into compact Voronoi regions and reuses this structure for hierarchical retrieval. On dataset DCASE 2020 and MIMII, HARMONY improves mean AUC from 75.40% to 90.00% (+14.60% absolute improvement) over the strongest unified baseline and from 83.90% to 90.00% over a strong foundation-feature baseline. On a single CPU, its hierarchical retrieval achieves a 7.8× speedup over exhaustive search with only a 0.20% AUC drop. These results suggest that incorporating explicit geometric partitioning can improve both detection accuracy and retrieval efficiency for ASD. Code and data are available at the anonymous URL: https://anonymous.4open.science/r/HARMONY-26CC.

Supervised post-training, including supervised fine-tuning (SFT) and knowledge distillation (KD), is essential for adapting large language models (LLMs) to specialized domains. However, traditional supervised post-training methods often suffer from catastrophic forgetting, training instability, and prohibitive computational costs. While layer-wise strategies have emerged as efficient alternatives, they introduce a representation bottleneck, where constrained updates in early stages limit feature diversity and impair final generalization. To resolve these issues, we develop HALO (Homotopy-Augmented Layer Optimization), a homotopy-driven layer-wise strategy for LLM post-training. By introducing a continuous homotopy parameter and a sequence of monotonically increasing auxiliary functions, we formulate the post-training process as a continuous deformation of the optimization problem that progressively incorporates layers into the trainable set. Specifically, HALO unfreezes the model from the output back to the input in a structured sequence. We theoretically prove that this approach establishes a differentiable solution path, ensuring a stable transition from a restricted optimization state to a fully converged LLM. Extensive experiments across diverse model families and sizes demonstrate that HALO consistently enhances performance and generalization. Notably, HALO exhibits strong robustness in challenging scenarios, especially in low-resource settings, while significantly reducing convergence time and iteration counts. Our results position HALO as an effective and accessible framework for high-performing LLM specialization.

Skill-augmented agents increasingly rely on large reusable skill libraries, but retrieving relevant skills is not the same as presenting usable context. Existing methods typically return atomic skills or dependency-aware bundles whose internal roles remain implicit, leaving the agent to infer the execution entry point, support skills, visible requirements, and failure-avoidance guidance. We introduce **G**roup **o**f **Skills** (**GoSkills**), an inference-time group-structured retrieval method that changes the agent-facing retrieval object from a flat skill list to a compact, role-labeled execution context. **GoSkills** builds anchor-centered skill groups from a typed skill graph, expands support groups through a group graph, bottlenecks the selected group plan into a bounded set of atomic skill payloads, and renders a fixed execution contract with **START**, **SUPPORT**, **CHECK**, and **AVOID** fields, without changing the downstream agent, skill payloads, or execution environment. Experiments on SkillsBench and ALFWorld show that \goskills preserves visible-requirement coverage under a small skill budget, improves over flat skill-access baselines, and often improves reward and agent-only runtime relative to structural retrieval references. Code is available at https://anonymous.4open.science/r/Group-of-Skills-E861.

Graph-structured data underpins applications from citation analysis and social-network modeling to molecular design and knowledge-graph construction, and Large Language Models (LLMs) are increasingly used as prompt-driven graph synthesizers. Classical graph-generation reviews catalog deep generative models and their evaluation primitives, but predate the LLM era and provide no foundation for evaluating instruction-following graph synthesis. Recent LLM-era benchmarks evaluate models along graph-type or task-domain axes; such organizations, however, average over structural complexity and cannot localize \emph{where} in the complexity spectrum an LLM breaks down. To close this diagnostic gap, we introduce GraphInstruct, a progressive-complexity benchmark that stratifies LLM graph generation into six complexity levels and five evaluation dimensions, paired with 800 hand-authored instructions, 1,582 algorithmically synthesized reference solutions, and a 12-LLM capability evaluation across 45 (model, strategy) configurations. We find that discriminative power peaks at multi-constraint composition rather than reasoning depth, that no single prompting strategy dominates across levels or model families, and that domain-semantic constraints remain iteration-invariant under all tested methods---pointing to retrieval rather than additional compute as the next research frontier. Atop the benchmark, a verification-guided iterative framework with constraint-aware adaptive prompting consistently surpasses the prompt-engineering ceiling on tested target models, demonstrating that the benchmark's fine-grained signals drive method development. Data, code, and reproducibility artifacts are included in the supplementary materials and will be released publicly upon acceptance.

Retrieval-augmented generation (RAG) divides into two camps failing in complementary ways on broad, long-tailed questions: iterative agentic RAG re-queries a biased distribution and drifts on prominent aspects, while structural RAG commits to a hierarchy fixed upfront and cannot admit aspects it initially missed. We propose GBT-RAG, which unifies the two by transposing gradient boosting into the text domain: each round is a query decomposition tree fitted to its predecessors' residual gap. This raises two challenges mirroring the pillars of boosting---natural-language critiques are not quantifiable, and successive rounds drift rather than fit the residual. We address the first with a text gradient, a structured residual in the retriever's embedding space whose support localizes missingness to specific leaves; we address the second by growing each tree only over under-covered leaves and admitting each rewrite through a monotone-improvement gate. Across standard benchmarks, GBT-RAG consistently outperforms the strongest iterative and structural baselines, with the largest gains on the long-tailed regime that motivated the design.

Existing Retrieval-Augmented Generation (RAG) systems struggle with multi-hop reasoning due to a fundamental inability to balance relevance and novelty. Traditional dense retrievers are constrained by surface-level matching, falling into a homogenized "Similarity Trap". Conversely, emerging Graph RAGs attempt to explore novel documents via heuristic entity extraction, but they introduce prohibitive Large Language Model (LLM) overhead and easily drift into off-topic "Novelty Traps". To fundamentally resolve this contradiction, we propose Geometric Gain Graph (G$^3$RAG), the novel zero-token graph construction paradigm for RAG. G$^3$RAG completely discards the expensive heuristic extraction by LLMs, aiming instead to directly quantify the information gain between nodes through the native document feature space. To this end, we designed a concise edge weight criterion, $\cos\theta \cdot \sin\theta$, which naturally maps the trade-off between relevance and novelty into a geometric measurement of directional consistency and orthogonality in the vector space, thereby driving the token-free topological construction of the graph. Furthermore, we introduce a topological penalty to suppress the excessive connectivity of high-frequency hubs, forcing the transient diffusion process toward long-tail peripheral nodes that carry critical indirect evidence. Extensive experiments demonstrate that, while entirely eliminating graph construction costs, G$^3$RAG significantly outperforms state-of-the-art Graph RAG baselines on complex multi-hop QA benchmarks.