Efficient Training

Oscillations and synchronization are widely believed to play a fundamental role in representation and computation. However, existing machine learning approaches based on synchronization dynamics have largely been confined to specialized settings such as object discovery, with limited evidence of scalability to standard vision benchmarks or logic reasoning tasks. We propose the \emph{Winfree Oscillatory Neural Network} (\emph{WONN}), a dynamical neural architecture based on generalized Winfree dynamics. \emph{WONN} evolves representations on the torus $(S^1)^d$ through structured oscillatory interactions, combining phase-based inductive biases with flexible and hierarchical interaction mechanisms instantiated as either fixed trigonometric mappings or learnable neural networks. We evaluate \emph{WONN} on image recognition and complex reasoning tasks, including CIFAR, ImageNet, Maze-hard, and Sudoku. Across these domains, \emph{WONN} achieves competitive or superior performance with strong parameter efficiency. \textbf{In particular, \emph{WONN} is, to our knowledge, the first synchronization-based oscillatory architecture to scale competitively to ImageNet-1K.} Furthermore, on Maze-hard, \emph{WONN} achieves \textbf{80.1\% accuracy using only 1\% of the parameters of prior state-of-the-art models}. These results suggest that structured oscillatory dynamics provide a scalable and parameter-efficient alternative to conventional neural architectures.

Fine-tuning large transformers routes gradients through every token position, paying the cost in activation memory that scales linearly with sequence length and dominates the GPU budget. Existing efficient methods reduce the parameter side---low-rank adapters, prefix tuning, weight-decomposed updates---but leave the activation footprint untouched. We present the Token Hypothesis, a memory-efficient fine-tuning procedure motivated by a Neural Tangent Kernel analysis of how fine-tuning signal is distributed across token positions. We prove that every position is a low-dimensional capacity bottleneck whose width does not grow with the model, that positions with aligned hidden-state geometry are interchangeable in bidirectional models, and that causal positions form a monotone capacity chain in which later positions strictly dominate earlier ones. The analysis pins down how many positions are needed and which ones, both computable from forward passes alone: any subset suffices in bidirectional models, while the last few positions suffice in causal ones. We translate this into a procedure that backpropagates through only a small subset of positions while running the full forward pass over the rest. Empirically, the procedure matches or exceeds full-sequence fine-tuning across six task families---commonsense reasoning, mathematical reasoning, visual question answering, video question answering, GLUE, and image classification---and five backbones spanning bidirectional and causal architectures. It reduces activation memory substantially, translating to up to ∼ 20% lower total GPU memory (∼ 33GB) on long-context tasks, and composes cleanly with parameter-efficient methods such as LoRA and DoRA.

We study learning-augmented algorithms for scheduling on multiple parallel machines under the objectives of minimizing the makespan or the total weighted completion time. We assume that the algorithm has access to noisy information (the prediction) about an optimal solution, expressed through a solution encoding that implicitly quantifies both the amount of information provided and the nature of noise (errors in the prediction). Our central question is to determine the maximum level of noise that still allows for efficiently recovering this optimal solution. We observe that information guiding greedy scheduling to optimality or revealing the optimal assignment of tasks to machines is very sensitive to noise, and prove that recovering an optimal solution from solution encodings capturing either is computationally hard, even for a small (non-constant) number of errors. On the positive side, we show that an order-based encoding, which leads to errors that are dispersed locally, is robust to noise. We present a dynamic programming framework that utilizes this encoding and solves optimally several classical scheduling problems, including makespan and total weighted completion time minimization on identical or unrelated machines.

Sparse attention is becoming increasingly important for serving large language models (LLMs) as generation lengths continue to grow. However, deploying and evaluating new sparse attention algorithms at scale remains highly engineering-intensive, slowing both human researchers and AI agents in exploring the sparse attention design. To address this challenge, we present Vortex, a system that combines a Python-embedded frontend language atop a page-centric tensor abstraction for expressing a broad range of sparse attention algorithms, with an efficient backend tightly integrated into modern LLM serving stacks. Vortex enables rapid prototyping, deployment, and evaluation of sparse attention algorithms, effectively translating their theoretical efficiency gains into real-world throughput improvements. As a result, Vortex substantially accelerates the design and iteration of sparse attention algorithms. Using Vortex, we further demonstrate AI-agent-driven exploration of the sparse attention, automatically generating and refining diverse algorithms that achieve strong accuracy-throughput trade-offs, with the best generated algorithms delivering up to $3.46\times$ higher throughput than full attention while preserving accuracy.

Anomaly detection (AD) is a critical task in machine learning and scientific discovery. Existing methods that operate on learned embeddings typically define anomalies through local density, isolation heuristics, or distance from a fitted distribution, making them sensitive to neighborhood scale, partitioning choices, or restrictive distributional assumptions. In this work, we introduce a new paradigm by formulating anomaly detection in terms of dataset diversity. We propose the Vendi Anomaly Score (VAS), which detects anomalies by quantifying how much a sample changes the diversity of the dataset, as measured by the Vendi Score, when removed. VAS captures both local redundancy and global data structure without relying on density estimation or partitioning heuristics. Moreover, VAS requires no hyperparameter tuning, instead adapting automatically to the spectral structure of the data. VAS is non-parametric and scales linearly with dataset size. Across 148 benchmark embedding datasets derived from 10 source datasets and 4 embedding architectures, VAS achieves state-of-the-art AD performance. We further validate VAS on large-scale ImageNet experiments, where it remains robust as both contamination rate and dataset scale increase.

As long-video understanding becomes increasingly important, augmented reality (AR) devices offer a natural platform for deploying egocentric video intelligence. Yet first-person videos often contain substantial temporal redundancy, creating heavy memory and compute demands for resource-limited AR hardware. We present~\textit{UTOPI}, a plug-and-play pre-compression module tailored to egocentric long-video processing on AR devices. UTOPI first uses user motion cues to estimate cross-frame overlap and remove redundant content through a viewpoint-shift-robust pruning strategy. It then leverages eye-tracking signals to detect user fixation regions, separating attention-relevant foreground from background and preserving important content through a fine-tuned neural adjustment module. Evaluated across multiple benchmarks and model backbones, UTOPI reduces video tokens by up to $95%$ while maintaining task performance.

Probabilistic weather forecasting requires not only accurate trajectories, but calibrated distributions over plausible atmospheric futures. Recent data-driven systems have achieved remarkable deterministic skill, and diffusion-based ensemble forecasters have substantially improved sample realism and uncertainty quantification. However, their inference cost scales with forecast horizon, ensemble size, and the number of denoising steps required for each transition, making large operational ensembles expensive. To address this, we present Tyche, a one-step conditional flow model for efficient probabilistic weather forecasting. Tyche models the conditional forecast distribution with a destination-aware average-velocity flow that maps Gaussian noise directly to future weather states in a single function evaluation (1-NFE). To make this one-step transport learnable in high-dimensional geophysical fields, we derive a JVP-regularized rectification objective that enforces temporal self-consistency across source and destination flow timesteps without explicitly forming Jacobians. The transport field is parameterized by an isotropic Swin-style transformer that preserves fine-scale spatial structure while remaining scalable on global grids. To improve ensemble reliability under autoregressive forecasting, we further introduce a rollout-based finetuning stage with curriculum CRPS calibration supervision. Experiments on ERA5 at 1.5$^\circ$ and 6-hour resolution show that our Tyche, using merely a single NFE, matches or exceeds the forecast skill and calibration of state-of-the-art multi-step generative baselines and the operational ECMWF IFS ensemble. Codes are available.

Test-time training and test-time policy optimization have emerged as effective tools for improving the robustness of LLMs under distribution shift. A common approach constructs label-free reward signals from multiple sampled rollouts using majority voting, but existing methods typically rely on a fixed, large rollout budget for every query. This fixed allocation is inefficient because many queries reach consensus well before the rollout budget is exhausted. In this work, we introduce \textbf{AdaPO}, which replaces fixed-budget majority voting with an adaptive two-stage voting procedure that moves from electing to confirming when a leader is statistically dominant, and stops once the posterior error of accepting that leader is sufficiently low. AdaPO is plug-and-play with standard policy optimization algorithms such as PPO and GRPO, and it is also compatible with supervised fine-tuning at test time. Our rigorous theoretical results show that, under the symmetric categorical noise model with known likelihood parameters, AdaPO's electing-stage trigger coincides with the classical sequential probability ratio test (SPRT), while the confirming-stage rule yields a principled posterior-threshold stopping mechanism. Across representative reasoning benchmarks, AdaPO substantially reduces compute, achieving over 40\% token savings on GPQA while maintaining competitive accuracy. The source code will be open upon acceptance at \url{https://open-upon-acceptance}.

Large language model (LLM) routing is widely used to advance the cost-quality Pareto frontier of modern serving systems. While coarse-grained routing at the session or query level has been widely adopted in production systems, recent algorithm work shows substantial efficiency and quality advantages of finer token-level routing. However, efficiently serving token-level routed inference poses significant challenges to existing system design. Built on single-LLM assumptions, current systems struggle with step desynchronization and frequent batch admission delays under token-level routing, and they also impose high implementation complexity on developers. To address this, we propose TokenRouter, an efficient and user-friendly serving system for token-level routed LLM inference. TokenRouter shifts from the conventional request-centric paradigm to a model-centric one. Instead of treating each request as a synchronized decoding stream, TokenRouter launches a subserver for each candidate LLM and dispatches requests asynchronously according to routing decisions. Each subserver employs a delayed-batching scheduler, whose hyperparameters are derived mathematically from a throughput model of the system. Across diverse routing algorithms, workloads, and model pairs, TokenRouter achieves 1.83-64.29× higher decoding throughput than existing systems, substantially advancing the serving efficiency of token-level LLM routing.

Pruning has emerged as a promising direction for accelerating large language model (LLM) inference. However, many existing methods rely on offline calibration data, making them sensitive to distribution shifts between calibration data and inference inputs. In this paper, we introduce Token Filtering, a lightweight online pruning method that selectively skips attention computation for redundant tokens during inference, without requiring any calibration data or fine-tuning. Token Filtering identifies redundant tokens based on joint key–value (KV) similarity and bypasses their attention computation. This approach reduces both compute and KV cache size while preserving essential contextual information. To preserve accuracy while meeting the target pruning ratio, we restrict pruning to later layers, which are typically less sensitive to pruning, with a layer-wise threshold that adaptively tracks the target pruning ratio. Extensive experiments on LLaMA3.1-8B, LLaMA3.1-8B-Instruct, and Qwen3-8B demonstrate that Token Filtering consistently achieves better accuracy–efficiency trade-offs than prior methods. In long-context generation, Token Filtering achieves up to 1.5X higher accuracy than the best-performing pruning baseline while reducing latency by up to 44% compared to the dense baseline.

Probabilistic long-term time series forecasting requires estimating a distribution over future trajectories under practical deployment constraints, where inference latency and stability can be as critical as forecast quality. Existing diffusion-based forecasters provide expressive uncertainty modeling, yet their iterative denoising incurs substantial horizon insensitive runtime overhead, which limits real time long-term use. In contrast, one-shot generative models such as VAEs and flows are computationally efficient but often struggle to capture complex multi-modal structure and heavy tailed uncertainty. We propose \textbf{ToLD}, an efficient time‑series forecasting framework built upon \textbf{to}kenized truncated \textbf{l}atent \textbf{d}iffusion to enable fast and reliable long term generation. ToLD adopts a tokenized conditional latent modeling and performs compact token-wise refinement via a few-step truncated latent diffusion process, where the diffusion noise is injected in a context-dependent manner to model non-stationary uncertainty. To reduce train test mismatch in multi-sample forecasting, we further introduce a score distillation scheme that learns a target-free scoring function for test-time candidate ranking. Experiments on eight real-world datasets show that ToLD improves both point accuracy and probabilistic quality over state-of-the-art, achieving up to \textbf{4.3\%} MSE improvement and \textbf{19.3\%} CRPS reduction on long-term tasks with minimal additional inference overhead.

As high-quality public web corpora become increasingly exhausted, clean long-context documents have become a scarce and expensive source of training data for large language models (LLMs). Existing long-context corpora are often proprietary and costly to acquire, synthetically generated, or concentrated in narrow domains such as programming. We introduce the Stanford EDGAR Filings Dataset (SFD), an open reconstruction of SEC filings into layout-faithful MultiMarkdown for financial language modeling and evaluation. SFD makes audited financial statements, risk disclosures, ownership reports, accounting notes, and market-moving event filings usable as long-context pretraining data and as a basis for financial reasoning, forecasting, compliance, and document understanding. The resulting corpus is token-efficient, model-ready, and has less than 0.1\% overlap with Common Crawl-derived corpora. We release SFD-v1, a 152B-token snapshot of a larger 18.4M-filing corpus estimated at 500B tokens. We further introduce two SFD-derived benchmarks: EDGAR-Forecast, which evaluates filing-grounded numerical forecasting after model knowledge cutoffs, and EDGAR-OCR, which evaluates transcription of complex financial tables.

Kernelized attention methods, which aim to replace the $O(N^2)$ complexity of softmax attention with $O(N)$ alternatives, have been the focus of intense research in the last five years. Despite growing interest, systematic evaluations at foundation model scales and fair comparisons across approaches are still missing. We analyze kernelized attention in two halves and highlight prominent limitations on each. On efficiency, FlashAttention achieves lower memory consumption and faster wall-clock time than state-of-the-art kernelized methods across most sequence lengths (with auto-regressive generation memory the only clear exception), directly contradicting the field's core assumption. We also discover that reported efficiency gains often fail to reproduce against modern baselines. On quality, current kernelized methods leave gaps of $2$-$10$ points relative to softmax under a unified distillation protocol, and these gaps do not close with model scale. We show this gap is not intrinsic to kernelization: LARA++, a principled extension of LARA that approximates softmax via importance sampling with adaptive proposal distributions, recovers up to 99\% of softmax accuracy. We establish that the gap is closable; however, the computational overhead remains substantial, leaving the efficiency wall intact.

Spiking neural networks (SNNs) offer a compelling pathway toward energy-efficient artificial intelligence, fundamentally anchored in presynaptic input-driven sparsity. However, a substantial yet chronically overlooked source of computational overhead persists: the continuous subthreshold dynamics of postsynaptic neurons. Quantitative analysis reveals that up to 44\% of neurons in an ImageNet-trained ResNet-18 model reside in a deeply inhibited state, imposing a persistent computational burden despite possessing a negligible probability of firing. To eradicate this inefficiency, the Dormant Spiking Neuron (DSN) model is introduced. By incorporating a state-driven dynamic gate, the DSN allows a neuron to enter a dormant state when its membrane potential falls below a predefined negative threshold, dynamically pruning redundant computations at the instance level. Crucially, this dormancy mechanism is directly integrated into the neuronal dynamics and trained end-to-end via backpropagation through time using surrogate gradients, enabling the network to actively learn a robust dormancy policy that adaptively preserves critical information flow. Extensive experiments across static and neuromorphic datasets demonstrate that the DSN framework consistently reduces computational energy by 40\% to over 70\%. Notably, on the large-scale ImageNet dataset, the DSN outperforms state-of-the-art static pruning techniques by achieving equivalent energy savings while simultaneously improving upon the baseline accuracy by 1.34\%. By establishing a trainable, state-driven sparsity paradigm orthogonal to conventional input-driven methods, the DSN unlocks synergistic efficiency gains, presenting a highly potent approach for advancing neuromorphic computing. Source code will be made publicly available.

Neurobiological circuits may be energy efficient in part because there is a division of labor: different subsystems compute or represent different things. In light of this, it is interesting that a major difference between subsystems is the time scale on which they typically vary. Here, we argue that temporal smoothness constraints---the idea that it may be energetically costly for systems to vary much more quickly or slowly than their typical time scale---imply a certain division of labor with respect to temporal stimulus features. In particular, slow subsystems ought to represent slowly varying stimulus features, and fast subsystems ought to represent quickly varying features. We formulate a novel efficient coding model that we use to investigate this claim, and exactly solve for the optimal division of labor. In addition to finding that slower subsystems ought to represent more slowly varying temporal features, we find surprising structure of mathematical interest: optimal codes have subsystems which represent orthogonal temporal features, and which correspond to the eigenfunctions of a Sturm-Liouville problem.

Video large multimodal models incur substantial inference cost because visual tokens accumulate across sampled frames. Training-free token compression can reduce this cost, but it often degrades video reasoning performance when later frames must contribute new evidence over time. We identify a common failure mode behind this degradation: existing compressors do not track the committed support, namely the token support already retained and delivered to the language model by earlier frames. As a result, selectors with different scoring rules repeatedly retain overlapping supports across neighboring frames, a phenomenon we call Selector Collapse. To address this issue, we propose \tempo{}, a frame-causal token compression framework that combines committed-support memory, residual-energy greedy selection, and parameter-free neighborhood fusion. Across four video understanding benchmarks and multiple backbones, \tempo{} consistently improves the accuracy-efficiency trade-off of training-free compression; at 10\% retention, it preserves 99.8\% of vanilla performance on LLaVA-OneVision and improves fixed-budget frame scaling on Qwen2.5-VL to 109.3\% relative accuracy, without retraining, backbone modification, or additional token budget.

Scaling laws are increasingly used to pick which expensive training run to scale up. The practical question — which candidate is best at a fixed future compute budget $C^\star$ — fits neither classical best-arm identification nor minimum-oriented functional bandits. We formulate it as the *target-budget best-function identification* (TB-BFI) problem, with two natural flavors: TB-BFI-FC asks for a $\delta$-correct certificate of the winner at $C^\star$, and TB-BFI-FB asks for low target-budget simple regret under a hard exploration budget. For the shared-exponent power-law class we prove uniform target-budget validity, a finite-time stopping bound for the cost-aware TB-LUCB rule, and a per-arm-exponent perturbation extension. We then close two structural gaps. **(a)** We extend the theory *beyond* shared-exponent to the joint $(N,D)$ Chinchilla law with shared but *unknown* exponents, via a two-stage protocol with explicit pilot sample complexity and a quadratic-in-$\epsilon$ centred-design perturbation theorem. **(b)** We give the first positive TB-BFI-FB result: a cost-aware Sequential-Halving algorithm whose cost-weighted simple-regret upper bound matches a new information-theoretic lower bound up to the standard $K \log K$ adaptive-vs-oracle gap. Empirically, theorem-aligned synthetic experiments certify the $C^\star$-winner in $95$–$100\%$ of trials; on a char-level transformer learning-rate-crossover our extrapolator picks the right arm in $3/3$ seeds where Successive Halving picks $0/3$; and on a flip-rank Chinchilla instance the joint-law procedure reaches $70$–$87\%$ accuracy where the univariate $c = N \cdot D$ baseline scores $0\%$.

Decentralized Stochastic Approximation (SA) enables cooperative fixed-point solving over mesh networks without a central server. In these systems, local update is relatively cheap compared with coordination, and frequent information exchange makes communication cost the main bottleneck. To alleviate this issue, we propose a decentralized SA method in which each agent performs $H$ local updates before each communication round. By amortizing communications over multiple local SA updates, the method can substantially reduce communication cost. We also study a multiple mixing scheme based on FastMix, which trades a small amount intra-round communications for a stronger consensus effectiveness on poorly connected graphs. Theoretically, we establish finite-time bounds under general contractive norms and Markovian sampling, by combining contraction, consensus error, and delayed Markovian terms in a round-level Lyapunov recursion. The results show trade-offs among computation, communication and sampling. Numerical experiments on decentralized Q-learning, Markovian SGD and linear SA validate the theory and show substantial communication savings for target accuracy.

Decoder-only Large View Synthesis Models (LVSMs) with KV-cache have recently achieved state-of-the-art quality by regressing novel views from neural networks without reconstructing 3D geometry, while Test-Time Training (TTT) layers can be introduced to save computation, but sacrifice quality. We propose **T**est-**T**ime **B**aking (**TTB**), a strategy that learns to bake global scene information from cross-view attention into a lightweight MLP, enabling fast and high-fidelity novel-view decoding. The first key is to repurpose the fast-weight MLP of TTT to learn the cross-view attention mapping explicitly from its input-output pairs, rather than replacing attention entirely, achieving the same rendering efficiency as TTT while delivering substantially higher quality. The second key addresses a modality misalignment between context prefilling and novel-view rendering: we disentangle context tokens into camera-only context rendering tokens to serve as TTB input, and context ground-truth tokens with image content to serve as the TTB mapping target, ensuring the baked MLPs are relieved of the modality generalization challenge between context and novel-view processing. Our method achieves up to **23x** rendering speedup with quality on par with the state-of-the-art across several benchmarks.

Large language models remain vulnerable to jailbreak attacks, yet we still lack a systematic understanding of how jailbreak success scales with attacker effort across methods, model families, and harm types. We initiate a scaling-law framework for jailbreaks by treating each attack as a compute-bounded optimization procedure and measuring progress on a shared FLOPs axis. Our systematic evaluation spans four representative jailbreak paradigms, covering optimization-based attacks, self-refinement prompting, sampling-based selection, and genetic optimization, across multiple model families and scales on a diverse set of harmful goals. We investigate scaling laws that relate attacker budget to attack success score by fitting a simple saturating exponential function to FLOPs--success trajectories, and we derive comparable efficiency summaries from the fitted curves. Empirically, prompting-based paradigms tend to be the most compute-efficient compared to optimization-based methods. To explain this gap, we cast prompt-based updates into an optimization view and show via a same-state comparison that prompt-based attacks more effectively optimize in prompt space. We also show that attacks occupy distinct success--stealthiness operating points with prompting-based methods occupying the high-success, high-stealth region. Finally, we find that vulnerability is strongly goal-dependent: harms involving misinformation are typically easier to elicit than other non-misinformation harms.