NeurIPS 2026 论文列表

Neural surrogate models for Partial Differential Equations (PDEs) on unstructured 3D geometries are often limited by poor generalization and the high cost of generating large-scale training datasets. Consequently, pre-training has emerged as a critical alternative to enhance the robustness and scalability of these models. In this work, we introduce a pre-training framework tailored to both steady-state and transient regimes. For steady-state problems, we propose a geometry-driven strategy that leverages intrinsic shape descriptors to learn representations of complex 3D domains. For transient problems, we introduce a physics-driven approach based on online generation of synthetic PDE data, enabling scalable pre-training without reliance on expensive datasets. This framework learns transferable features for challenging downstream tasks. Across multiple experiments, our approach achieves up to 3$\times$ faster convergence, 2$\times$ greater data efficiency, and up to 25\% higher accuracy during fine-tuning, particularly under realistic low-data regimes. This methodology provides a practical pathway toward data-efficient neural emulators for large-scale simulations.

Packed high-token 3D fields are not just long-sequence modeling: once a volume is packed into tokens, sequence order is no longer the geometry of the sample. This creates a general failure mode for attention over packed geometric data: sequence-index positional mechanisms can learn the memory layout rather than the object. We introduce Pack3D, a geometry-aware local--global attention operator that separates memory layout from attention geometry. Pack3D keeps local neighborhood attention exact, compresses only distant context into contiguous 3D block summaries, and indexes attention with axial RoPE on true 3D token-grid coordinates rather than packed sequence positions. With RoPE applied before block compression, each query interacts with a geometry-aware mixture of block tokens, so compressed global context remains spatially interpretable after packing. A synthetic repacking benchmark makes the failure mode explicit: learned positional embeddings and packed-sequence RoPE fail under layout changes, whereas axial grid-coordinate RoPE remains invariant. On Allen WTC--11 hiPSC microscopy, our main claim-grade domain, Pack3D improves matched learned-position local--global attention by 0.0978 best macro Dice and 0.1248 final macro Dice at 262k tokens and depth 8. In an Allen-15 joint comparison, Pack3D also improves over a parameter-matched TokenGrid 3D U-Net on both best and final macro Dice. Inference-time ablations show that the trained model genuinely uses its compressed summaries, and the fused forward/backward implementation is $6.49\times$ lower latency than dense SDPA at 65k tokens.

How do language models acquire such a vast array of concepts and abilities from next-token prediction alone? We introduce an interpretability framework that exposes how this single loss relates to different concepts at different points during pretraining. For each layer and checkpoint, we measure the \emph{optimization engagement} with a concept, i.e., alignment between a probe-defined concept subspace and the average gradient outer product (AGOP) of the next-token loss. The concept subspace captures where the concept currently lives in the model's representation; AGOP captures the directions in which the loss exerts the strongest pressure. Across 12 lexical, syntactic, and semantic tasks and 19 Pythia-1B and -410M checkpoints, we find that optimization engagement with a concept is temporally aligned with behavioral changes in the model related to the concept. Furthermore, optimization engagement tends to peak before we observe the largest behavioral changes in the model, suggesting that sudden changes in model performance ("grokking") may be preceded by behaviorally unobservable geometric changes. However, this "geometric precedence" pattern is not universally observed; we also see cases where optimization engagement does not peak once, but rather persists at a high value, or peaks multiple times. These results indicate that the loss contributes to concept learning in a myriad of ways that simple probe performance alone cannot reveal.

Flow matching learns a velocity field whose ordinary differential equation transports a simple reference distribution to a data distribution. Existing theory typically controls this velocity through ambient, isotropic Lipschitz regularity, which becomes extremely pessimistic when data concentrate near a low-dimensional manifold. In this regime, standard error bounds can blow up drastically near the terminal time and fail to explain why flow matching should remain stable. We study flow matching for distributions obtained by smoothing a density on a compact manifold with small Gaussian noise. Our first result is a tube-local Lipschitz regularity theorem showing that, on the relevant neighborhood of the manifold, the population flow-matching velocity has a Jacobian bound that grows only at the inverse effective noise scale. This improves substantially over the stronger singular behavior suggested by existing ambient analyses. Our second result gives a directional description of the velocity Jacobian: tangential directions exhibit mild growth, normal directions are strongly contractive, and tangential-normal coupling is lower order. This reveals that the apparent singularity is highly structured rather than uniformly harmful. These results provide geometric foundations for sharper stability guarantees and suggest a path toward intrinsic-dimension generalization bounds for flow matching on manifold-concentrated data.

Chain-of-thought (CoT) reasoning enables large language models (LLMs) to solve complex problems by generating intermediate reasoning steps. While much attention has been paid to the length and content of these reasoning chains, far less is known about their internal geometry. We study the *geometry* of CoT trajectories in the hidden state space of transformer models, formalizing each reasoning chain as a discrete curve in $\mathbb{R}^d$ and characterizing it through spectral, positional, and kinematic geometric functionals. We introduce the effective dimension $d_\rho$ as a measure of trajectory complexity and show theoretically that trajectories with flatter eigenvalue spectra correspond to harder tasks, as they explore more of the hidden dimensions. Lastly, we explore how kinematic features of the trajectory, mean position, positional dispersion, initial hidden states, mean velocity, mean speed, and speed dispersion, can be used to predict solution correctness before generation is complete, and may inform future early-stopping strategies. Experimentally, on mathematical reasoning problems from the MATH500 dataset, $d_\rho$ achieves 0.93 AUC in distinguishing easy from hard problems, while kinematic features show promise for predicting correctness from only the first 20% of generated tokens. These correctness signatures transfer across questions of varying difficulty, establishing that the *shape* of a model's internal reasoning trajectory is a principled window into both task hardness and solution quality.

Test-time scaling (TTS) improves large language model (LLM) reasoning by spending additional inference-time compute, most commonly through width-heavy repeated sampling and aggregation. However, this strategy scales cost almost linearly with the number of calls and can remain brittle when the sampled trajectories share the same failure mode. We introduce Maze, a test-time scaling framework that shifts computation away from repeated target-model calls and toward LLM-free planning over ordered few-shot exemplars. The central idea is to treat an ordered exemplar sequence as a prompt trajectory embedded in a latent geometric space. Instead of asking the LLM for many independent attempts, Maze first ranks candidate prompt trajectories using cached encoder representations and a path scorer, ${\it i.e.}$, a lightweight learnable metric, then spends one call (or a few calls) only on the most promising trajectories. This design preserves black-box deployability, since the planner is decoupled from the target LLM and can be implemented with a separate frozen encoder. Across various challenging reasoning benchmarks, single-call Maze consistently outperforms strong single-call prompt-optimization baselines, and MultiMaze recovers a large fraction of width-heavy TTS gains with substantially fewer computational cost measured by number of LLM calls and latency. We further present a theoretical perspective showing when prompt-path ranking can recover much of the benefit of best-of-$N$ style scaling: if high-utility prompt trajectories are sufficiently rankable and sufficiently covered by the explored pool, a deeper planner-side search can substitute for a substantial amount of width-heavy sampling.

Large language models solve complex problems by generating lengthy chains of explicit reasoning tokens. While effective, this makes reasoning expensive, length-sensitive, and constrained to (discrete) natural language. While latent reasoning offers a continuous alternative, determining useful structures for intermediate latent states is an open challenge. In this paper, we formulate latent reasoning as a geometric path-approximation problem within the model’s pretrained token-embedding space. We introduce Geometric Latent Reasoning (GLR), which uses a lightweight transition head to predict iterative direction updates in embedding space. Using textual chain-of-thought traces as anchors, GLR learns to approximate discrete reasoning trajectories while permitting continuous deviations from exact token embeddings. Evaluations on mathematical reasoning benchmarks using Qwen3 models reveal an emergent phenomenon: geometric latent reasoning induces substantially shorter generations without an explicit length objective. By replacing early explicit reasoning with continuous latent steps, models often reach correct answers using substantially fewer total generation steps. These findings suggest that continuous trajectories act as compact intermediate reasoning states, exposing a new tradeoff between latent computation budget, output length, and accuracy.

Existing Retrieval-Augmented Generation (RAG) systems struggle with multi-hop reasoning due to a fundamental inability to balance relevance and novelty. Traditional dense retrievers are constrained by surface-level matching, falling into a homogenized "Similarity Trap". Conversely, emerging Graph RAGs attempt to explore novel documents via heuristic entity extraction, but they introduce prohibitive Large Language Model (LLM) overhead and easily drift into off-topic "Novelty Traps". To fundamentally resolve this contradiction, we propose Geometric Gain Graph (G$^3$RAG), the novel zero-token graph construction paradigm for RAG. G$^3$RAG completely discards the expensive heuristic extraction by LLMs, aiming instead to directly quantify the information gain between nodes through the native document feature space. To this end, we designed a concise edge weight criterion, $\cos\theta \cdot \sin\theta$, which naturally maps the trade-off between relevance and novelty into a geometric measurement of directional consistency and orthogonality in the vector space, thereby driving the token-free topological construction of the graph. Furthermore, we introduce a topological penalty to suppress the excessive connectivity of high-frequency hubs, forcing the transient diffusion process toward long-tail peripheral nodes that carry critical indirect evidence. Extensive experiments demonstrate that, while entirely eliminating graph construction costs, G$^3$RAG significantly outperforms state-of-the-art Graph RAG baselines on complex multi-hop QA benchmarks.

How do transformer language models memorize factual associations? A common view casts internal weight matrices as associative memories over pairs of embeddings, requiring parameter counts that scale linearly with the number of facts. We develop a theoretical and empirical account of an alternative, \emph{geometric} form of memorization in which learned embeddings encode relational structure directly, and the MLP plays a qualitatively different role. In a controlled setting where a single-layer transformer must memorize random bijections from subjects to a shared attribute set, we prove that a logarithmic embedding dimension suffices: subject embeddings encode \emph{linear superpositions} of their associated attribute vectors, and a small MLP acts as a relation-conditioned selector that extracts the relevant attribute via ReLU gating, and not as an associative key-value mapping. We extend these results to the multi-hop setting---chains of relational queries such as ``Who is the mother of the wife of $x$?''---providing constructions with and without chain-of-thought that exhibit a provable capacity--depth tradeoff, complemented by a matching information-theoretic lower bound. Empirically, gradient descent discovers solutions with precisely the predicted structure. Once trained, the MLP transfers zero-shot to entirely new bijections when subject embeddings are appropriately re-initialized, revealing that it has learned a generic selection mechanism rather than memorized any particular set of facts.

Neural multivariate regression underpins a wide range of domains such as control, robotics, and finance, yet the geometry of its learned representations remains poorly characterized. While neural collapse has been shown to benefit generalization in classification, we find that analogous collapse in regression consistently degrades performance. To explain this contrast, we analyze models through the lens of intrinsic dimension. Across control tasks and synthetic datasets, we estimate the intrinsic dimension of last-layer features ($ID_H$) and compare it with that of the regression targets ($ID_Y$). Collapsed models exhibit $ID_H < ID_Y$, leading to over-compression and poor generalization, whereas non-collapsed models typically maintain $ID_H > ID_Y$. For the non-collapsed models, performance with respect to $ID_H$ depends on the data quantity and noise levels. From these observations, we identify two regimes—over-compressed and under-compressed—that determine when expanding or reducing feature dimensionality improves performance. Our results provide new geometric insights into neural regression and suggest practical strategies for enhancing generalization.

End-to-end speech LLMs often answer the same question less accurately from speech than from text, even when the spoken content is semantically matched to the text prompt. We study this modality gap as a layer-wise inference problem rather than a static embedding mismatch. Across four open-weight speech LLMs on SpeechMMLU and VoiceBench BBH, cross-layer Centered Kernel Alignment (CKA) with speech–text token alignment shows that speech can become geometrically text-like in middle layers while still failing to form stable late-layer answer margins. The central finding is that mid-layer alignment does not imply functional equivalence. ASR-to-LLM controls show that recognition explains part of the gap in some settings but not all of it. Matched-instance margin analysis localizes the failure to weak late-layer answer separation. Attention diagnostics further show that speech evidence remains more diffuse at the decision token. Speech-faithful temporal compaction improves BBH accuracy from 59.9% to 62.1% with tempo 1.2× and to 61.9% with silence trimming, while recovering 34.3% and 38.2% of text-correct/speech-wrong failures. These conditional repairs reveal a 2.35–2.62× asymmetry between temporal compaction and quantity-only KV merging on the same gap subset, motivating post-projection temporal abstraction rather than only input-level feature matching.

Geospatial reasoning requires solving image-grounded problems over complex object relationships and scene structures. However, developing this capability is hindered by the cost of annotating a vast and combinatorial question space. We propose GeoX, a self-play framework that acquires spatial logic through executable programs and verifiable rewards in the absence of large-scale human-curated data. Given a satellite or aerial image, our framework employs a single multimodal policy to alternate between proposing spatial problems with executable programs and solving them under three reasoning modes of abduction, deduction, and induction, using a segmentation tool and standard computational libraries. Program execution turns each program into a reward that jointly optimizes the two roles via reinforcement learning. GeoX consistently improves its base VLMs by up to 5.5 points on average, matching or exceeding conventional baselines trained on millions of curated samples. Alongside the framework, we release a benchmark for geospatial reasoning curated from self-play generations.

We present GeoTransolver, a multiscale geometry-aware physics attention transformer for Computer Aided Engineering (CAE). GeoTransolver extends the Transolver backbone with GALE (Geometry-Aware Latent Embeddings) attention, which pairs physics-aware self-attention on learned state slices with cross-attention to a shared geometry and global context computed via multi-scale ball queries (inspired by Domino) and reused in every block. Implemented and released in NVIDIA PhysicsNeMo, GeoTransolver persistently projects geometry and global parameters, into physical state spaces to anchor computations to domain structure and operating regimes. We benchmark on DrivAerML, SHIFT-SUV, and SHIFT-Wing against Domino, Transolver (PhysicsNeMo implementation), and literature-reported AB-UPT, evaluating drag/lift $R^2$ and relative $L_1$ errors on field variables. As an additional nonlinear structural mechanics application, we also report Transolver and GeoTransolver results on bumper-beam and full-vehicle Body-in-White (BIW) crash-dynamics benchmarks, evaluating relative $L_2$ trajectory error and probe-level kinematic MSE. GeoTransolver delivers improved accuracy, robustness to geometry and regime shifts, and favorable data efficiency; we include DrivAerML ablations and qualitative contour and design-trend results, advancing operator learning for high-fidelity surrogates on complex, irregular, non-linear domains.

Diffusion models achieve high sample quality but remain expensive at inference time because sampling requires many sequential neural function evaluations (NFEs). Existing acceleration methods either use fixed step-skipping schedules, adapt step sizes based on local numerical error, or require additional training. We introduce GeoSPRINT (Geometric Step Pruning for Inference in Trajectories), a training-free framework for constructing non-uniform sampling schedules from the geometry of denoising trajectories. GeoSPRINT detects geometrically redundant steps using a hyperplanarity test in latent space, implemented efficiently via QR factorization, and converts the resulting redundancy profile into a sampling schedule that allocates more steps to high-curvature regions of the trajectory. In addition, we introduce the trajectory projection score $\alpha_{\mathrm{traj}}$, a residual-variance metric that quantifies trajectory straightness and serves as a model-free diagnostic for rectified flow quality. Across CIFAR-10 ($32{\times}32$), LSUN Church ($256{\times}256$), and Stable Diffusion v1.5 ($512{\times}512$ latent), GeoSPRINT consistently improves over uniform DDIM (Denoising Diffusion Implicit Models) schedules at matched NFE budgets. On CIFAR-10, GeoSPRINT improves FID (Fréchet Inception Distance) by 0.7-1.1 over DDIM across 49-89 NFEs and surpasses DPM-Solver++ at NFE${\geq}30$ despite using a first-order DDIM solver. On LSUN Church, it reduces FID from 1.48 to 1.26 at 52 steps, and on Stable Diffusion v1.5 it achieves up to 1.93 FID improvement over DDIM. These results show that trajectory geometry provides a useful global signal for allocating inference steps and that schedule quality can substantially improve diffusion sampling efficiency without retraining.

Worldwide image geolocalization aims to predict precise GPS coordinates for images captured anywhere on Earth, which is challenging due to the large visual and geographic diversity. Recent methods mainly follow two paradigms: retrieval-based approaches that match queries against a reference database, and generation-based approaches that directly predict coordinates using Large Vision-Language Models (LVLMs). However, we observe distinct error profiles between them: retrieval excels at fine-grained instance matching, while generation offers robust semantic reasoning. This complementary heterogeneity suggests that no single paradigm is universally superior. To harness this potential, we propose GeoRouter, a dynamic routing framework that adaptively assigns each query to the optimal paradigm. GeoRouter leverages an LVLM backbone to analyze visual content and provide routing decisions. To optimize GeoRouter, we introduce a distance-aware preference objective that converts the distance gap between paradigms into a continuous supervision signal, explicitly reflecting relative performance differences. Furthermore, we construct GeoRouting, the first large-scale dataset tailored for training routing policies with independent paradigm predictions. Extensive experiments on IM2GPS3K and YFCC4K demonstrate that GeoRouter significantly outperforms state-of-the-art baselines. Our dataset, code, and checkpoints are publicly available to facilitate future research.

Panoramic 3D reconstruction enables the holistic recovery of complete 360° scene geometry, offering significant benefits for robotics and AR/VR applications; however, direct estimation is persistently challenged by inherent spherical distortions. While decomposing panoramas into perspective views effectively circumvents these distortions and leverages established geometric priors, this strategy often fails to preserve global spatial consistency across the decomposed views. To address this fundamental limitation, we introduce an enhanced feed-forward 3D reconstruction framework tailored for the view decomposition paradigm, driven by two key innovations. First, we propose a geometry-aware attention aggregator, featuring a streamlined attention architecture equipped with a novel Geo-Attention mechanism, which explicitly exploits the intrinsic global information to guide multi-view interactions, thereby suppressing geometrically inconsistent matches. Furthermore, to jointly enforce fine-grained local details and global structural coherence, we formulate a hybrid supervision strategy. Specifically, we complement the standard local point cloud loss in perspective space with our newly proposed spatially uniform global point cloud loss in panoramic space, explicitly enforcing holistic constraints within a unified coordinate system. Experiments demonstrate that our method consistently outperforms prior single-view panoramic approaches in reconstruction accuracy, geometric consistency, and visual fidelity.

Autoregressive video diffusion models have demonstrated effectiveness in interactive game generation, with Minecraft gameplay serving as a representative application. To faithfully simulate gameplay, a model must generate natural content when exploring new scenes while preserving spatial consistency when revisiting explored areas. Under limited computation budgets, it must compress and exploit historical cues within a finite context window, which exposes a trade-off: models relying solely on temporal context offer flexible exploration but suffer from poor revisit consistency, whereas adding spatial memory strengthens consistency but may degrade new scene generation quality when the model over-relies on sparse or unreliable spatial context. We present GeoMemory, a learning framework that pairs training protocols with a geometry-indexed spatial memory. Specifically, our Hybrid Training exposes the model to both exploration and revisitation regimes, guiding the model to rely on temporal memory in new scenes while effectively incorporating spatial memory upon revisits. Chained Forward Training creates larger pose variations and encourages reliance on spatial memory for maintaining consistency. For spatial memory, we integrate Point-to-Frame Retrieval with an incremental 3D cache generated by VGGT, enabling constant-time retrieval of relevant historical context regardless of sequence length. Extensive experiments demonstrate that GeoMemory achieves superior performance in both long-term spatial consistency and visual quality in new scenes with real-time interaction.

State space models (SSMs) provide an efficient route to global image restoration because they model long sequences with linear complexity. Recent restoration architectures further enlarge the effective context by reordering image tokens so that semantically similar pixels become neighbors in the 1D scan. This strategy improves texture aggregation, but it also exposes a mismatch between the geometry of images and the discretization used by SSMs: adjacent tokens in the reordered sequence can be spatially distant, while the recurrent update still treats them as consecutive samples of a continuous process. In addition, existing prompt-based modulation relies on a finite set of discrete prompts, which is poorly matched to the continuous variation of natural textures and offers no explicit compensation for the low-pass tendency of recurrent integration. We propose GeoMamba, a geometry-aware and continuously modulated SSM for image restoration. GeoMamba introduces geometry-adaptive discretization (GAD), which conditions the SSM time-step on the Euclidean distance between consecutive reordered tokens and attenuates history propagation across spatial jumps. It also introduces a continuous dynamic matrix (CDM), which regresses pixel-wise modulation parameters from feature and gradient cues to reduce prompt quantization and preserve high-frequency details. Experiments on lightweight and classical image super-resolution demonstrate consistent improvements over strong Transformer- and Mamba-based baselines, while achieving better or competitive performance in additional denoising and JPEG artifact reduction settings.

Text-to-image models can generate plausible street scenes from prompts such as “a street in New York City,” but it remains unclear whether these images resemble the requested place or show a generic city appearance. Commonly used evaluation metrics mainly measure image quality or text alignment and do not directly assess geographic fidelity. We introduce GeoFidelity-Bench, a benchmark for evaluating whether generated street-view images match a target location at the level of named street blocks. The benchmark contains 112 street blocks from 25 cities across six continents, each paired with a panel of real Mapillary images and geographic metadata. From this metadata we build prompts with the city only, with street and neighborhood names, and with these names plus raw GPS coordinates. Across six open-weight text-to-image models, street and neighborhood names improve similarity to the target location, while raw GPS coordinates add only a small and uneven gain. Control prompts that swap in the names of other blocks while keeping the city fixed reduce retrieval accuracy, with larger decreases when the neighborhood name or both names are replaced. Generated images also distinguish their target blocks from local alternatives less reliably than held-out real images. CLIP alignment between generated images and their prompts is nearly uninformative about whether the images achieve block-level geographic fidelity. GeoFidelity-Bench provides a controlled testbed for studying location-conditioned generation and for measuring progress beyond generic city-level plausibility.