paperlist.ai

AI 顶会论文

AI 顶会论文库

检索 30+ 个顶会的论文，包括 NeurIPS、ICLR、ICML、CVPR、ACL、AAAI 等。输入关键词筛选，打开任意一篇查看摘要、作者和链接。

NeurIPS 2026

Understanding the Interplay between Memorization and Learning in Large Language Models

Bishwamittra Ghosh, Soumi Das, Qinyuan Wu, Mohammad Aflah Khan, Deepak Garg, Krishna Gummadi, Evimaria Terzi

We investigate foundational questions about the interplay between memorization and learning in large language models (LLMs). When an LLM generates a string, can we disentangle the roles of rote memorization and contextual learning? Rote memorization results in an LLM regurgitating specific details of a training string, while contextual learning leads to the generation of new strings that follow generalizable patterns of the underlying language. Related questions of interest include: can an LLM avoid memorization when optimally learning a language?, and if not, can we design training schemes to reduce memorization and improve learning? To address these questions, we propose contextual memorization, a new measure that precisely disentangles memorization from learning. Using this measure, we establish that different training strings are memorized at different epochs in training, and that memorization of some training strings is unavoidable when optimally learning a language. Finally, we propose memorization-aware training, a scheme that dynamically down-samples training strings based on their memorization, and in doing so, equalizes learning across strings, reduces memorization, and improves learning. Our conclusions are supported by extensive experiments on multiple LLMs using formal languages as a controlled testbed, enabling precise language specification, exact string sampling, and elimination of data contamination; and further validated on natural languages.

NeurIPS 2026

Understanding the Effects of Hyper-Connections on Self-Attention Dynamics: A Bifurcation Analysis

Yen N Pham, Dung V Nguyen, Trinh Nguyen, Thieu Vo, Tan Nguyen

A central challenge in understanding deep sequence models is characterizing how token representations evolve across layers: whether they collapse, diverge, or converge to nontrivial structures. We study this question for Transformers with hyper-connections, whose learnable cross-layer coupling makes the stability of representation collapse analytically tractable. In contrast to standard residual connections, hyper-connections introduce additional parameters that directly control the hyperbolicity of the zero fixed point, at which the local stability is fully determined by the Jacobian eigenvalues, thereby enabling a rigorous and precise bifurcation analysis of the dynamical system underlying the model. In particular, we derive a continuous-depth ODE limit of self-attention with hyper-connections and use bifurcation theory to characterize its token dynamics. For the 1-stream case, we obtain a closed-form critical threshold at which a pitchfork bifurcation occurs, separating representation collapse from divergence or convergence to nontrivial stable states. For the 2-stream case, the richer coupling structure yields two distinct stability boundaries, corresponding to qualitatively different instabilities: one arising when a real eigenvalue crosses zero, altering the number of nearby equilibria (static type), and another when a complex-conjugate pair reaches the imaginary axis, leading to oscillatory behavior (Hopf type). These phenomena have no counterpart in the 1-stream setting. Experiments on pretrained hyper-connected language models validate our theory and show that unstable regimes yield stronger performance.

NeurIPS 2026

Understanding the Curse of Unrolling

Sheheryar Mehmood, Florian Knoll, Peter Ochs

Algorithm unrolling is ubiquitous in machine learning, particularly in hyperparameter optimization and meta-learning, where Jacobians of solution mappings are computed by differentiating through iterative algorithms. Although unrolling is known to yield asymptotically correct Jacobians under suitable conditions, recent work has shown that the derivative iterates may initially diverge from the true Jacobian, a phenomenon known as the curse of unrolling. In this work, we provide a non-asymptotic analysis that explains the origin of this behavior and identifies the algorithmic factors that govern it. We show that truncating early iterations of the derivative computation mitigates the curse while simultaneously reducing memory requirements. Finally, we validate our theoretical findings both on synthetic problems and in a meta-learning setting for few-shot classification, where we study the behavior of gradients obtained via unrolling and implicit differentiation.

NeurIPS 2026

Understanding the Convergence of Direct Training of SNNs with Surrogate Gradients

Zelin Wei, Tianxing Man, Wanli Shi, Bin Gu

Spiking Neural Networks (SNNs) offer high energy efficiency, but direct optimization is challenging because spike generation is governed by a non-differentiable hard threshold. Practical direct-training methods, including surrogate gradients (SG) and local zeroth-order (LocalZO) estimators, maintain hard spikes in the forward pass while using surrogate or perturbation-induced derivatives in the backward pass. This forward--backward mismatch raises a central theoretical question: how do local gradient discrepancies propagate through the spatiotemporal computation graph, and can the resulting biased updates still provide guaranties for the original discrete-spike objective? We develop a unified convergence analysis framework for SG and LocalZO direct SNN training. A smoothed spike objective is introduced only as an analytical bridge, allowing us to separate the core error into two components: the gradient-level mismatch between the direct-training mean direction and the smoothed-objective gradient, and the objective-level consistency gap between the smoothed and original discrete objectives. Both components are controlled by a threshold-tail functional that measures how often hard-trajectory membrane potentials lie near the firing threshold. Under suitable regularity assumptions, we obtain a non-asymptotic guarantee for the original discrete-spike objective. Experiments across representative SNN architectures and benchmarks verify convergence consistency, the existence of errors, and demonstrate that the threshold-tail statistic is an effective diagnostic for direct SNN training.

NeurIPS 2026

Understanding the Co-evolutionary Solution Exchange of the MOEA/D: Minimal Communication Gives Maximal Speed-Ups

Benjamin Doerr, Moaad Elmoutassim, Martin S. Krejca, Guy Mushiata Kibulu

Besides domination-based algorithms such as the NSGA-II, the decomposition-based MOEA/D is one of the most popular and successful optimization heuristic for multi-objective problems. It decomposes the problem into several, single-objective subproblems. By solving these in a co-evolutionary manner and archiving non-dominated solutions, it approximates or computes the Pareto front. Interestingly, so far no advantages from the co-evolutionary communication of subproblem solutions were proven. In this first mathematical runtime analysis addressing this research question, we investigate how the mutation-based MOEA/D solves the classic OMM benchmark when solutions are exchanged among subproblems in different distances $r$. We show, for the first time, that the MOEA/D profits from an exchange of solutions, and that already very small communication radii $r$ are sufficient to maximally profit from this feature of the algorithm. More precisely, for all values $r \ge 1$ we prove a tight runtime guarantee of $\Theta(n^2)$, where $n$ is the problem size. This is asymptotically faster than the known bound of $O(n^2 \log n)$ guarantee for $r=0$, which we complement with a matching bound of $\Omega(n^2 \log n)$.

NeurIPS 2026

Understanding the Challenges in Iterative Generative Optimization with LLMs

Allen Nie, Xavier Daull, Zhiyi Kuang, Abhinav Akkiraju, Anish Chaudhuri, Max Piasevoli, Ryan Rong, YuCheng Yuan, Prerit Choudhary, Shannon Xiao, Rasool Fakoor, Adith Swaminathan, Ching-An Cheng

Generative optimization uses large language models (LLMs) to iteratively improve artifacts (like code, workflows or prompts) using execution feedback. It is a promising approach to building self-improving agents, yet in practice remains brittle: despite active research, only 9% of surveyed agents used any automated optimization. We argue that this brittleness arises because, to set up a learning loop, an engineer must make "hidden" design choices: What can the optimizer edit and what learning evidence is provided per update? We investigate three factors that affect most applications: the starting artifact, batching execution traces as experiences, and determining a credit horizon with truncated traces. Through case studies in MLAgentBench, Atari, and BigBench Extra Hard, we find that decisions around these factors can determine whether generative optimization succeeds and they are not made explicit in prior works. Different starting artifacts determine which solutions are reachable in MLAgentBench, truncated traces can still improve Atari agents, and larger minibatches do not monotonically improve generalization on BBEH. We conclude that the lack of a simple, universal learning-loop setup across domains is a major hurdle for productionization and adoption, and provide practical guidance for making these design choices.

NeurIPS 2026

Understanding diffusion models requires rethinking (again) generalization

Pierre Marion, Yu-Han Wu

This position paper argues that understanding generalization in diffusion models requires fundamentally new theoretical frameworks that go beyond both classical statistical learning theory and the benign overfitting paradigm developed for supervised learning. In diffusion models, unlike in supervised learning, memorization of training data and generalization to novel samples are incompatible: a model that has fully memorized its training set generates copies rather than novel data. Several theoretical explanations for why practical diffusion models nevertheless generalize have been proposed, based on capacity limitations, implicit regularization from optimization, or architectural inductive biases, but their interactions remain unclear. We argue that the field should pivot from explaining why the diffusion models do not memorize to investigating what the models actually learn during pre-memorization phase. To highligh our stance, we conduct empirical study of diffusion models trained on CIFAR-10, and we distill the findings into concrete open questions that we believe are key to improve understanding of generalization in diffusion models.

NeurIPS 2026

Understanding axial attention in TSFMs through single location regression

Andrei Pantea, Aymeric Dieuleveut

Recent advances in Time Series Foundation Models (TSFMs) increasingly rely on axial attention mechanisms, but despite their empirical success, the theoretical properties of axial attention remain largely unexplored. To bridge this gap, we extend the Single Location Regression framework introduced in Marion et al. [2025] to 2D structured data, in order to study sparse information settings where a target depends on a small number of tokens. We formulate an analytically tractable axial attention-type predictor and rigorously analyze its statistical properties and training dynamics. We prove that under a specific asymptotic regime and with knowledge of the underlying oracle parameters, our predictor achieves Bayes optimality, whereas a class of semi-linear simplifications strictly fails to do so, underlining the role of the inner non-linearity. We prove that for a specific temperature scaling, Projected Gradient Descent converges globally to the optimal oracle parameters. Extensive numerical experiments validate our theoretical findings and highlight the critical role of inverse temperature scheduling in empirical convergence.

NeurIPS 2026

Understanding and Mitigating Under-Confidence in GNNs from the Final Layer

Jincheng Huang, Jie Xu, Ping Hu, Xiaoshuang Shi, Lei Feng, Xiaofeng Zhu

Graph Neural Networks (GNNs) have demonstrated remarkable effectiveness on graph-based tasks. However, their predictive confidence is often miscalibrated, typically exhibiting under-confidence, which harms the reliability of their decisions. Existing calibration methods for GNNs normally introduce additional calibration components, which fail to capture the intrinsic relationship between the model and the prediction confidence, resulting in limited theoretical guarantees and increased computational overhead. To address this issue, we propose a simple yet efficient graph calibration method. We establish a unified theoretical framework revealing that model confidence is jointly governed by class-centroid-level and node-level calibration at the final layer. Based on this insight, we theoretically show that reducing the weight decay of the final-layer parameters alleviates GNN under-confidence by acting on the class-centroid level, while node-level calibration acts as a finer-grained complement to class-centroid level calibration, which encourages each test node to be closer to its predicted class centroid at the final-layer representations. Extensive experiments validate the superiority of our method.

NeurIPS 2026

Understanding and Defending VLM Jailbreaks via Jailbreak-Related Representation Shift

Zhihua Wei, Qiang Li, Jian Ruan, Zhenxin Qin, Ruiyang Qin, Qingzhuo Wang, Dongrui Liu, Wen Shen

Large vision-language models (VLMs) often exhibit degraded safety alignment when visual inputs are integrated. Even when a text prompt is explicitly harmful, adding an image can substantially increase the jailbreak success rate. In this paper, we observe that VLMs can clearly distinguish benign, refusal, and jailbreak samples in their representation space, and that jailbreak responses often contain safety warnings. These observations lead to the \textit{recognize-but-fail-to-refuse} hypothesis: VLM jailbreaks do not arise from a failure to recognize harmful intent, but instead occur because adding an image induces a representation shift that steers the sample into a distinct jailbreak state where refusal is not triggered. To quantify how this image-induced representation shift contributes to jailbreak behavior, we define a jailbreak direction and characterize the jailbreak-related representation shift as the projection of the total image-induced shift onto this direction. Our analysis shows that the jailbreak-related shift reliably characterizes jailbreak behavior, providing a unified explanation for diverse jailbreak scenarios. Finally, we propose JRS-Rem, a defense method that enhances VLM safety by removing the jailbreak-related shift at inference time. Experiments show that JRS-Rem significantly improves VLM safety across multiple scenarios while preserving utility on benign tasks.

NeurIPS 2026

Understanding Sample Efficiency in Predictive Coding

Gaspard Oliviers, Elene Lominadze, Rafal Bogacz

Predictive Coding (PC) is an influential account of cortical learning. Much of recent work has focused on comparing PC to Backpropagation (BP) to find whether PC offers any advantages. Small scale experiments show that PC enables learning that is more sample efficient and effective in many contexts, though a thorough theoretical understanding of the phenomena remains elusive. To address this, we quantify the efficiency of learning in BP and PC through a metric called "target alignment", which measures how closely the change in the output of the network is aligned to the output prediction error. We then derive and empirically validate analytical expressions for target alignment in Deep Linear Networks. We show that learning in PC is more efficient than BP, which is especially pronounced in deep, narrow and pre-trained networks. We also derive exact conditions for guaranteed optimal target alignment in PC and validate our findings through experiments. We study full training trajectories of linear and non-linear models, and find the predicted benefits of PC persist in practice even when some assumptions are violated. Overall, this work provides a mechanistic understanding of the higher learning efficiency observed for PC over BP in previous works, and can guide how PC should be parametrised to learn most effectively.

NeurIPS 2026

Understanding Randomization in Greedy Model Search

Xin Chen, Jason Klusowski, Yan Shuo Tan, Chang Yu

We study feature subsampling in greedy model search, a randomization mechanism motivated by random forests and other ensemble methods. While recent theory suggests that this randomization acts solely as a variance reduction mechanism analogous to ridge regularization, these results largely rely on base learners optimized via ordinary least squares (OLS). We investigate the effects of feature subsampling on greedy forward selection, a tractable abstraction of the adaptive split search used by decision trees. Assuming an orthogonal design, we prove that ensembling with feature subsampling can reduce both bias and variance, contrasting with the pure variance reduction of convex base learners. Specifically, we show that both the training error and degrees of freedom need not be monotone in the subsampling rate, breaking the analogy with standard shrinkage methods like the lasso or ridge regression. Furthermore, we characterize the exact asymptotic behavior of the estimator, showing that it adaptively reweights OLS coefficients based on their rank, with weights that are well approximated by a logistic function. These are mechanism level results for a tractable analogue of greedy selection that depends on the response, not performance guarantees for full random forests.

NeurIPS 2026

Understanding Private Evolution as Learning-Augmented Clustering

Audra McMillan, Kunal Talwar, Felix Zhou

Private Evolution (PE) is a differentially private algorithm for synthetic data generation. While it can be viewed as a Wasserstein learning algorithm, it performs much better in practice than worst‑case Wasserstein analyses would predict. We recast PE as a generative model‑augmented Wasserstein learning. We show theoretically that when we take into account the use of a generative model that is able to capture something about the true distribution, then PE provably obtains much better performance bounds. For example, if the generator gives samples in the same low-dimensional space as the distribution, then the sample complexity depends on the intrinsic, not the ambient, dimension. We also show that standard variants of PE can fail to converge on simple well-clustered instances, and propose a new geometry-aware version of PE with provable convergence on such instances. Experimentally, we show that our new algorithm has consistent empirical gains over standard baselines.

NeurIPS 2026

Understanding Multi-View Transformers

Julien Gaubil, Michal Stary, Louis Martinez, Andreas Geiger, Maks Ovsjanikov, Ayush Tewari, Vincent Sitzmann

Multi-view transformers solve 3D vision tasks in a single feed-forward pass, yet their internal mechanisms remain poorly understood. We view their residual layers as progressively updating a latent state, and study how this state evolves during 3D reconstruction. To make the state geometrically interpretable, we train layer-wise probes that decode intermediate tokens into 3D pointmaps, yielding intermediate 3D reconstructions throughout the network. This view recasts feed-forward 3D reconstruction as progressive refinement rather than a single reconstruction step, with most layers improving reconstruction. Convergence speed varies across models and sample difficulty, with slower convergence on difficult samples. We further analyze these updates and find consistent, complementary roles for single-view and cross-view attention. Cross-view attention improves multi-view alignment, while single-view attention refines intra-view geometry and compensates for points visible in only one view. Finally, we identify cross-view attention heads that specialize in correspondence matching. We show that their correspondences are jointly refined with the reconstructed geometry, and that selected heads can be partially replaced by analytical correspondences. Our analysis provides a direct view of the reconstruction process inside multi-view transformers, and suggests a first step toward replacing emergent mechanisms with analytical operations. The code used for the analysis will be made available.

NeurIPS 2026

Understanding Layer Patching in Model Size Interpolation

Sara Kangaslahti, Jonathan Geuter, Nihal V. Nayak, Marco Fumero, Francesco Locatello, David Alvarez-Melis

Zero-shot model size interpolation aims to create new models of intermediate target sizes by combining existing models without additional training. Recent work on boomerang distillation [Kangaslahti et al., 2026] shows that a student language model distilled from a larger teacher can be expanded by iteratively patching its layers, replacing student layers with contiguous blocks of teacher layers to obtain models whose size and performance interpolate between the student and the teacher. In this work, we provide the first systematic study of student-layer selection for model size interpolation. We cast finding the optimal layer subset for each model size as an optimization problem and prove it can be viewed as a shortest-path problem in a certain acyclic graph. In experiments, we show that patching strongly shapes interpolation behavior, with effects that vary substantially across model families. We find that simple sequential strategies---patching either from the first layer to the last or from the last to the first---often achieve surprisingly strong performance in practice. We further introduce KLPatch, a greedy patching algorithm based on KL divergence, which often improves over last-to-first patching and approximately solves the optimization problem. Together, our results provide a principled understanding of how layer patching affects model size interpolation and offer practical guidance for constructing near-optimal interpolated models.

NeurIPS 2026

Understanding Generalization Requires Universal Induction

Aram Ebtekar, Marcus Hutter, Danica J. Sutherland

Classical statistical theory is insufficient to explain the successes of general-purpose AI models, because it depends on handcrafted inductive biases that it cannot justify. No Free Lunch theorems force any learner that beats chance on some environments to underperform on others, and meta-learning which environments are more likely only pushes the problem up a level. The inductive bias must therefore be grounded in something other than data. By deriving its bias from Turing universality, Solomonoff induction (SI) competes with all computable learners. However, its regret bounds include large ``constants,'' such as the size of a learner's (losslessly compressed) full codebase. We relativize SI to an information vantage point, which includes all pre-existing code and data. This reframes the inductive bias: instead of favoring some absolute notion of simplicity, we favor accessibility with respect to our vantage point. The relativized SI is sample-optimal on finite data: any learner that outperforms it necessarily contains inaccessible information about the data. While SI is incomputable and hence not a practical algorithm, it provides a formal optimum for inference in the limit of infinite compute. We argue that algorithmic information theory, which underlies SI, is necessary to explain the generalization behavior of modern (and future) AI systems.

NeurIPS 2026

Understanding Double Descent through Universal Compression

Jiaxuan Cheng, Addison Spiegel

Many deep learning models exhibit a double-descent phenomenon, where test loss exhibits a sharp peak near the interpolation threshold and then descends a second time as model size grows further, contrary to the U-shape predicted by the classical bias–variance tradeoff. The double-descent phenomenon has been found and explained in toy classical models, but is not well-understood in modern deep learning despite extensive empirical study. We interpret double descent through the novel lens of universal compression. Specifically, we upper bound the maximum-likelihood test loss as a sum of three terms: an approximation error, a minimax batch regret, and an MLE-vs-Bayes gap. We show that the minimax Bayes predictor, unlike the MLE, does not exhibit the double-descent peak. Furthermore, we isolate each term through experiments with uniform random labels, and show that double descent is primarily caused by the MLE-vs-Bayes gap in this setting; here, the MLE cross-entropy grows to be roughly five times larger at the double-descent peak, while the minimax Bayes predictor cross-entropy remains flat. Additionally, we show the possibility of a second ascent in test loss at very large model sizes, and motivate this theoretically through the minimax batch regret term of our decomposition.

NeurIPS 2026

Understanding Circulant Permutation to Extend the CMinHash Estimator

Keegan Kang, Avery Hood, Benedict H Wong

CMinHash promises a new direction for designing MinHash algorithms with reduced variance and storage space via circulant permutations. However, the original variance analysis is difficult to extend to other MinHash estimators. We present a new approach that gives explicit and easily comparable approximate variance expressions, and use it to analyze the Minner estimator. Numerical experiments show that our variance expression matches the empirical mean square error (MSE) of estimates. Our work thereby demonstrates how to apply circulant permutation to other MinHash estimators and compute their approximate variance.

NeurIPS 2026

Underlying Functional Structure of Reinforcement Learning

Ezgi Korkmaz

Advances in deep reinforcement learning have enabled the development of policies capable of solving complex, high-dimensional problems, allowing AI agents to strategize and reason purely through interaction without supervision. Despite these successes, our knowledge on the functional properties and the underlying structure of the deep neural policy manifold remains limited. In this paper, we uncover a fundamental functional property in reinforcement learning: the intrinsic alignment between the advantage function and the gradient of the loss targeting directions of incoherence. We provide a rigorous theoretical foundation for this relationship, demonstrating that this intrinsic alignment characterizes how policies learn and internalize the underlying value function and how this process governs policy decisions. By leveraging this fundamental functional property, we propose a novel algorithm that can diagnose and identify deep neural policy decision volatilities. We conduct extensive empirical analysis in high-dimensional MDPs. From algorithmic and architectural changes to natural distributional shifts and worst-case perturbations, our proposed method can identify and audit the differences by leveraging the underlying structure of the deep neural policy manifold and the intrinsic correlation. Our analysis reveals foundational properties of policies trained in high-dimensional MDPs, and our paper provides a principled step toward constructing scalable, stable, and generalizable deep reinforcement learning agents.

NeurIPS 2026

Uncovering the latent structure of interwoven population and temporal codes

Zachary Friedenberger, Yiwei Cao, Richard Naud

Population analysis methods have become standard for navigating the complexity of neural data. However, these methods often assume a rate code, neglecting information encoded in the precise timing of spikes. Critically, additional information encoded in bursts of action potentials may be missed. Here, we develop a factor analysis method that disentangles the factors associated with bursts and individual spikes. This enables burst codes to be investigated directly from the structure of the data, without requiring external covariates. We demonstrate that analyzing firing rates alone obscures the latent structure and factors underlying bursts. Applying our method to simulated and experimental data, we show that it can infer the correct latent structure and be used to test for the presence of burst coding. By merging the population and burst coding perspectives, we provide a framework for linking changes in bursting to internal variables involved in attention, perception, and learning.

继续浏览

更多 AI 顶会论文

30+ 个顶会，按会议、年份和主题浏览。

浏览全部会议

进入任一会议和年份，查看完整论文列表。

全部会议 →

搜索论文

按标题、作者或关键词，在 21 万+ 篇论文里搜索。

搜索论文 →