合成数据

The *synthetic control method* (SCM) estimates causal effects in panel data with a single-treated unit by constructing a counterfactual outcome as a weighted combination of untreated control units that matches the pre-treatment trajectory. In this paper, we introduce the **targeted synthetic control (TSC)** method, a new two-stage estimator that directly estimates the counterfactual outcome. Specifically, our TSC method (1) yields a targeted debiasing estimator, in the sense that the targeted updating refines the initial weights to produce more stable weights; and (2) ensures that the final counterfactual estimation is a convex combination of observed control outcomes to enable direct interpretation of the synthetic control weights. TSC is flexible and can be instantiated with arbitrary machine learning models. Methodologically, TSC starts from an initial set of synthetic-control weights via a one-dimensional targeted update through the weight-tilting submodel, which calibrates the weights to reduce bias of weights estimation arising from pre-treatment fit. Furthermore, TSC avoids key shortcomings of existing methods (e.g., the augmented SCM), which can produce unbounded counterfactual estimates. Across extensive synthetic and real-world experiments, TSC consistently improves estimation accuracy over state-of-the-art SCM baselines.

Large language models (LLMs) rely on web-scale corpora for pre-training. The noise inherent in these datasets tends to obscure meaningful patterns and ultimately degrade model performance. Data curation mitigates but cannot eliminate such noise, so pre-training corpora remain noisy in practice. We therefore study whether a lightweight pre-pre-training (PPT) stage based on synthetic data with learnable temporal structure helps resist noisy data during the pre-training (PT) stage. % Concretely, we draw PPT sequences from many randomly initialized RNNs and keep the PT recipe unchanged. Across various corruption settings, our method consistently improves robustness to noise during PT, with larger relative gains at higher noise levels. % The gains generalize across corruption types and naturally noisy web corpora. For a 1B-parameter model, a synthetic PPT stage with only 65M tokens achieves the same final loss as the baseline while using up to 49% fewer natural-text PT tokens across different noise levels. Mechanistic analyses suggest PPT does not immediately suppress attention to noisy tokens. Rather, PPT-initialized models gradually downweight attention between corrupted tokens during noisy PT. This indicates that synthetic PPT inhibits noise self-modeling and shapes the subsequent optimization trajectory.

Heterogeneous Federated Learning (HFL) aims to enable collaboration among clients with diverse model architectures and non-IID data distributions, where direct parameter aggregation is often infeasible and prototype-based aggregation may suffer from semantic drift. Existing methods usually construct global references from client-side representations, making the alignment process vulnerable to local statistical bias and noisy or corrupted prototype updates. In this paper, we propose SATPL, a synthetic anchor-assisted prototype alignment framework that introduces externally generated synthetic anchors as auxiliary semantic priors for HFL. Instead of treating Large Language Models (LLMs) as infallible semantic oracles, SATPL uses LLM-generated class descriptions and generative models to construct reproducible reference prototypes for supervised tasks with known class semantics. Each client maps its heterogeneous local representation into a shared anchor space through a lightweight projection head, while an alignment-weighted aggregation rule assigns lower weights to prototypes that are highly inconsistent with the corresponding anchors. The blockchain component is used as an auditable recording layer for anchor commitments and aggregation metadata, rather than as the source of the statistical robustness guarantee. Experiments under architecture heterogeneity, label skew, and random prototype corruption show that SATPL improves accuracy, convergence stability, and robustness over representative HFL baselines. Additional analyses evaluate anchor quality, temperature sensitivity, and system overhead, clarifying both the benefits and limitations of synthetic-anchor-based alignment.

Existing peptide models saturate natural-sequence proteomics benchmarks, but whether they capture the underlying physicochemical dynamics or merely pattern-match on sequence statistics remains unclear. Public peptide datasets offer no axis along which to test this: non-canonical residues and stereochemical variation are essentially absent. We introduce StereoPep, a benchmark of 48,788 chemically synthesized peptides spanning 11 libraries, produced in the laboratory via high-throughput flow synthesis and characterized experimentally by reverse-phase liquid chromatography retention time. Unlike model-generated synthetic data, every peptide in StereoPep is a real molecule with a real chromatographic measurement. The dataset is constructed around matched diastereomeric pairs (sequences identical in primary structure but differing at a single stereocenter), providing a direct test of whether models capture three-dimensional stereochemical information. StereoPep enables progress across multiple fields: it serves as a benchmark for stereochemistry-aware molecular representations and geometric deep learning, supports adapting protein language models to non-canonical residues, and underpins emerging proteomics multiplexing strategies that encode samples through retention-time shifts. Across specialized retention-time models, graph-based architectures, and protein language models, all approaches predict overall retention reasonably well (Pearson $r \approx 0.80$) but collapse on diastereomer pair discrimination (AUC $\approx 0.55$), revealing that current representations encode atomic composition far better than 3D structure.

As IT systems grow in complexity, automated agent-driven incident response becomes increasingly critical. A key capability for these agents is summarizing large log streams into human-readable reports. Further progress requires meaningfully evaluating this function, yet existing benchmarks leave end-to-end summarization from large log streams underexplored due to the prohibitive cost and complexity of collecting real-world data. We introduce SLS-Bench, a benchmark for incident log summarization featuring 90 tasks grounded in real-world incident reports, each with a synthetic log stream and a reference summary. To construct SLS-Bench, we developed a data generation pipeline that converts incident reports into structured incident models, which guide the generation of synthetic logs and reference summaries. This pipeline combines narrative grounding, explicit incident modeling, generator-verifier loops, and statistical comparisons with real-world log streams to produce realistic and controllable observability data. Our evaluation of 14 LLMs on SLS-Bench shows that while frontier models can produce useful summaries, they do so at substantial cost and latency, and open-weight models lag significantly. SLS-Bench provides a framework for systematic evaluation of log analysis agents and a scalable blueprint for future synthetic observability benchmarks.

Deep research agents extend the role of search engines from retrieving keyword-matched pages to synthesizing knowledge, fundamentally changing how humans interact with information. However, frontier systems remain proprietary, while existing open agents often generalize poorly across different task types, leaving unclear how to train a broadly capable deep research agent. We release Quest, a family of open models (ranging from 2B to 35B) that serve as general-purpose deep research agents designed to handle a wide range of search tasks, with strong capabilities in fact seeking, citation grounding, and report synthesis. To build Quest, we propose an effective training recipe combining mid-training, supervised fine-tuning, and reinforcement learning. Central to this recipe is a curated data synthesis pipeline based on unified rubric trees, which applies to different task types and enables synthesizing training data with verifiable rewards without human annotation. In addition, Quest incorporates a built-in context management mechanism that enables effective long-horizon reasoning and knowledge synthesis. Using only 8K synthesized tasks, Quest approaches or even surpasses frontier closed-source agents across eight deep research benchmarks spanning diverse task types, and achieves the best overall performance among recent open-weight agents. Notably, Quest-35B achieves 30.7\% on Mind2Web 2, outperforming OpenAI DeepResearch (28\%) and Gemini DeepResearch (18\%). We plan to release all the resources for further research.

We study *privacy amplification by synthetic data release*, a phenomenon in which differential privacy guarantees improve when releasing only synthetic data rather than the private generative model itself. Recent work by Pierquin et al. (2025) established the first formal amplification guarantees for linear generators, but they apply only in asymptotic regimes where the model dimension far exceeds the number of released synthetic records, limiting practical relevance. In this work, we show that this restriction is not fundamental and establish that, surprisingly, for linear generators, *privacy amplification can persist even when releasing an unbounded number of synthetic records*. To prove this result, we develop new analytical tools for characterizing the privacy loss induced by synthetic data release. In particular, we leverage a sufficient-statistics reduction to characterize privacy leakage, and introduce a novel criterion that upper bounds Rényi divergences in terms of Fisher information. Our analysis provides structural insights that may guide the development of tighter privacy guarantees for more complex release mechanisms. Finally, experiments with variational autoencoders trained using DP-SGD further support the relevance of our theory beyond the linear setting.

We introduce Pion, a spectrum-preserving optimizer for large language model (LLM) training based on orthogonal equivalence transformation. Unlike additive optimizers such as Adam and Muon, Pion updates each weight matrix through left and right orthogonal transformations, preserving its singular values throughout training. This yields an optimization mechanism that modulates the geometry of weight matrices while keeping their spectral norm fixed. We derive the Pion update rule, systematically examine its design choices, and analyze its convergence behavior along with several key properties. Empirical results show that Pion offers a stable and competitive alternative to standard optimizers for both LLM pretraining and finetuning.

Simulating human behavior with Large Language Models (LLMs) offers a scalable laboratory for social science and stress-testing AI products. However, limitations remain in matching LLM outputs against the full breadth of human diversity. Current techniques for creating synthetic humans typically optimize for density matching, which often enforces behavioral uniformity and overlooks the rare, consequential outliers. We argue that robust simulation requires shifting the focus toward support coverage: ensuring synthetic populations span the entire landscape of possible human traits, opinions, and preferences. This paper introduces Persona Generators, functions that can produce diverse synthetic populations tailored to arbitrary contexts. We apply an iterative improvement loop based on AlphaEvolve, using LLMs as mutation operators to evolve our Persona Generator code rather than the personas themselves. The optimization process produces lightweight functions that can automatically expand small descriptions into populations of diverse synthetic personas that are maximizing coverage along relevant diversity axes. We demonstrate that evolved generators substantially outperform existing baselines across six diversity metrics on held-out contexts. Furthermore, by successfully mitigating standard LLM mode collapse, our generated populations actually capture real human trait distributions better than baselines specifically designed to match human statistics.

Large language models have made synthetic data inexpensive, but they can still be biased. When real data are scarce, researchers must decide how much of the real sample to use for direct estimation and how much to spend calibrating the generator. We study this allocation problem and derive the marginal-value condition for the optimal calibration size: $(v_n + c x^{-2\beta})^2 = 2\beta a c x^{-(2\beta+1)}$, where $x$ is the calibration size, $v_n$ is synthetic sampling variance, $a$ is the real-estimator variance constant, and $c x^{-2\beta}$ is squared synthetic bias. This condition has no universal fixed-share solution and yields five allocation regimes. In the main regime, $m_n \asymp n$ and $\beta > 1/2$, the optimal calibration size grows only as $n^{2/(2\beta+1)}$, so the calibration share vanishes; rules that keep a positive fixed fraction of real observations in calibration over-calibrate asymptotically. We also propose an adaptive grid estimator with an oracle inequality and a safety check, $xB_n(x)

Concept bottleneck models predict outcomes from high-level concepts detected in inputs. Although concepts provide a simple way to reap benefits from interpretability, very few datasets include concept labels. This limits researchers' ability to determine which problems are suitable for these models, isolate the factors that drive their performance or lead to failures, or uncover which algorithms perform well. In this paper, we develop synthetic benchmarks for concept-bottleneck models, focusing on their two main use cases: decision support, in which models assist humans in making better decisions, and automation, in which models handle routine tasks without supervision. Our benchmarks can generate labeled datasets while controlling for properties that affect performance, including data modality, concept choice, annotation quality, and completeness. We demonstrate how the benchmarks can be used to evaluate representative classes of concept bottleneck models. Our demonstrations show how the benchmarks can diagnose failure modes and guide follow-up testing.

Sparse Mixture of Experts (MoE) models offer a scalable and efficient architecture for training large neural networks by activating only a subset of parameters (“experts”) for each input. A learned router computes a distribution over these experts, and assigns input tokens to a small subset. However, without auxiliary balancing mechanisms, routers often converge to using only a few experts, severely limiting model capacity and degrading performance. Most current load balancing mechanisms encourage a uniform routing probability across experts. Early in pretraining, this can result in inconsistent routing behavior, resulting in the model spending its capacity learning redundant knowledge. We address this by introducing a novel load balancing loss that helps preserve relational structure, encouraging consistent expert choices for similar inputs during training. Our experimental results show that applying our loss to the router results in 36% faster convergence and lower redundancy compared to a popular load balancing loss.

Synthetic data has become a promising way to scale model training beyond limited human-generated data but it may also induce strong model collapse (Dohmatob et al., 2024), where any fixed fraction of synthetic data prevents model performance from improving under data scaling, leaving a non-vanishing excess risk floor. In this paper, we studied how synthetic data affects the generalization of one-pass SGD in high-dimensional linear regression with model shift. We show that mixed training induces strong model collapse while two-stage training avoids this by using synthetic data only in the first stage, followed by real-data training in the second stage, showing that strong model collapse is not inevitable through a simple data curriculum. We further establish scaling-law upper bounds for both protocols under a random sketch model, showing that larger models amplify synthetic-induced degradation in mixed training and giving an explicit characterization of how high-quality synthetic training may reduce bias in two-stage training. Overall, our results highlight that synthetic data is neither inherently harmful nor beneficial; its effect depends critically on both its quality and the training protocol used to incorporate it.

Current pre-training datasets are derived from web crawls, with all their issues, and were not designed to support mid- and post-training pipelines—for instance, they contain little explicit reasoning. Thus, many frontier labs have begun to develop their own internal datasets, starting from state-of-the-art models, to augment their pre-training data mix, e.g., with reasoning traces to address cold start problems. While demonstratively effective, none of these datasets are public, and the effect of this so-called _synthetic data_ on knowledge and skill acquisition of language models, including small ones, remains poorly understood. We present Synth, the first open-source synthetic corpus derived from 58,698 Wikipedia articles that collapses pre-, mid-, and post-training into a single training stage via structured amplification of curated encyclopedic seeds. We evaluate Synth by training a series of models: a 56M tiny model (Monad), 0.3B--0.6B dense models (Baguettotron), and a 13B-total / 1B-active Mixture-of-Experts. At iso-compute, Synth outperforms filtered web data, and our models remain competitive with similarly-sized open-weight baselines. Because Synth is back-translated from grounded passages, Synth-trained models achieve high factual precision despite 10-140$\times$ fewer training tokens, with memorization targeted by the seed corpus. These results show that synthetic datasets, including our Synth dataset, are capable of producing competitive generalist models at a significantly lower cost, enabling rapid iteration as the frontier advances. These findings open up possibilities for both generalist models with significantly increased data efficiency, as well as domain-specific models where no instruction or conversational data is available. Finally, we publicly release our Synth dataset and the series of Baguettotron models under a permissive license, thus supporting open-source language model development.

Backpropagation is the default learning rule for artificial neural networks and is often treated as the settled approach whenever differentiability is available. In this work, we revisit this convention from the perspective of sample efficiency. We introduce a unified vectorized feedback framework for loss-based and reward-based learning on computational graphs, in which synthetic gradients emerge as a natural alternative to backpropagation. We characterize the conditions under which synthetic gradients can achieve a lower gradient-estimation mean squared error than backpropagation. We construct examples illustrating that this sample efficiency advantage can be arbitrarily large. Experiments on contextual bandits and reinforcement learning tasks demonstrate the potential of our theoretical findings.

ECG, with mature physiological simulators and public benchmarks, is a useful testbed for evaluating what synthetic pretraining supports. We introduce a reusable framework for ECG representation learning that compares patient-free physiological simulators, patient-derived real ECGs, and learned synthetic ECGs from real-ECG-trained generators. Holding the encoder and MAE objective fixed, we study three matched settings. Transfer-protocol analysis compares abnormal-inclusive real ECGs, normal-only real ECGs, and simulator pretraining under frozen probing and full fine-tuning on 26 abnormal-ECG tasks from PTB-XL, G12EC, and CPSC2018 using a single-lead-II pipeline. Compute-normalized PTB-XL scaling tests whether simulator pretraining matches real-data pretraining under equal processed-sample budgets. A fixed 100k-sample generator-class comparison evaluates VAE, DCGAN, SSSD-ECG diffusion variants with and without labels, and two simulators, SimECG-M and SimECG-N. The transfer analysis shows that protocol determines the supported claim: abnormal-inclusive real ECG pretraining yields more linearly separable pathology features under frozen probing, whereas full fine-tuning narrows the gap and can reverse it when the simulator corpus is sufficiently scaled. Under the fixed-100k protocol, task-wise F1 averaged over five runs ranks SimECG-N first on 22/26 tasks; no evaluated learned generator ranks first under this fixed-100k protocol; and SimECG-N obtains higher mean F1 than both diffusion variants on all 26 tasks. These results delineate where abnormal-inclusive real exposure remains beneficial and where patient-free physiological simulation can serve as an effective, protocol-bound substitute under the evaluated single-lead-II MAE pretraining and full-fine-tuning setting. We release pretraining corpora, splits, and code to support reproducible evaluation of synthetic ECG pretraining sources.

Designing DNA libraries is a key challenge from drug design to protein engineering and synthetic biology. Modern generative models offer opportunities to navigate the design space and propose specific sequences predicted to be effective in-silico. Designing deterministic libraries of specific sequences is however limited by the cost of DNA synthesis -- the synthesis barrier. In contrast, high-throughput multiplexed screening can measure the function of billions of biological sequences in parallel. Harnessing this technology requires the design of randomized libraries with specific design constraints to achieve low synthesis costs. In practice, such stochastic libraries are often chosen heuristically, sacrificing control for scale. Is there a way to bridge AI-based in-silico sequence design with high-throughput experimentation? In this work, we introduce Policy Gradients for Library Design (PGLD). PGLD uses a synthesis-aware parametrization of stochastic DNA libraries and optimizes them against a specified objective function. This allows for designing massive, controlled libraries without being limited by synthesis costs. We show how PGLD enables lab-in-the-loop design of multi-round high-throughput experiments, and large-scale in-vitro DNA sampling from generative models. Finally, we use PGLD to design a library of $\sim 10^6$ unique sequences at a cost of $\sim 700$ USD to explore the mutation space of a broadly neutralizing influenza antibody.

We present AnyHand, a large-scale synthetic dataset designed to advance the state of the art in 3D hand pose estimation. While recent works with foundation approaches have shown that scaling training data markedly improves hand pose estimation, existing real-world datasets are limited in coverage, and prior synthetic datasets rarely provide occlusions, arm details, and aligned depth together at scale. To address this bottleneck, our proposed AnyHand contains 2.5M single-hand and 4.1M hand-object interaction RGB-D images, with rich geometric annotations. We show that extending the original training data recipes of existing RGB baselines with AnyHand yields significant gains on multiple benchmarks (FreiHAND and HO-3D), even when keeping the architectures and training schemes fixed. Together with extensive ablations on the scale and composition of the training data setups, these results suggest that training data diversity and quality are as critical as scale for advancing hand pose estimation. We further examine the utility of AnyHand's aligned depth maps in the appendix, showing that scaling RGB-D supervision with AnyHand allows a lightweight depth-fusion variant of existing RGB baselines to outperform prior RGB-D methods.