Foundation Priors

• URL: http://arxiv.org/abs/2512.01107v1
• Date: Sun, 30 Nov 2025 22:09:37 GMT
• Title: Foundation Priors
• Abstract summary: Synthetic data reflects both the model's learned patterns and the user's subjective priors, expectations, and biases.<n>We show how synthetic data and the associated foundation prior can be incorporated into standard statistical and econometrics.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Foundation models, and in particular large language models, can generate highly informative responses, prompting growing interest in using these ''synthetic'' outputs as data in empirical research and decision-making. This paper introduces the idea of a foundation prior, which shows that model-generated outputs are not as real observations, but draws from the foundation prior induced prior predictive distribution. As such synthetic data reflects both the model's learned patterns and the user's subjective priors, expectations, and biases. We model the subjectivity of the generative process by making explicit the dependence of synthetic outputs on the user's anticipated data distribution, the prompt-engineering process, and the trust placed in the foundation model. We derive the foundation prior as an exponential-tilted, generalized Bayesian update of the user's primitive prior, where a trust parameter governs the weight assigned to synthetic data. We then show how synthetic data and the associated foundation prior can be incorporated into standard statistical and econometric workflows, and discuss their use in applications such as refining complex models, informing latent constructs, guiding experimental design, and augmenting random-coefficient and partially linear specifications. By treating generative outputs as structured, explicitly subjective priors rather than as empirical observations, the framework offers a principled way to harness foundation models in empirical work while avoiding the conflation of synthetic ''facts'' with real data.

Related papers

• From Synthetic Priors to Model Behavior: Structural Coverage in Tabular Foundation Models [15.974042867454365]
  We measure how broadly and repeatedly each synthetic prior reaches benchmark tasks using structural coverage and normalized density.<n>We find substantial differences across synthetic pretraining priors.<n>Stronger synthetic-to-benchmark support is generally associated with better relative model performance.
  arXiv  Detail & Related papers  (2026-09-07T01:27:45Z)
• Joint Model and Data Sparsification via the Marginal Likelihood [53.29070892356214]
  We propose jointly learning individual feature and sample relevancies, enabling simultaneous model and data sparsification.<n>This symmetric pruning of model and data offers a natural extension that preserves conjugacy.<n> Empirical results across diverse regression tasks affirm that a joint ARD approach consistently yields both sparse and robust prediction models.
  arXiv  Detail & Related papers  (2026-05-28T13:26:53Z)
• Harnessing Synthetic Data from Generative AI for Statistical Inference [6.0353292419288485]
  This paper reviews the current landscape of synthetic data generation and use from a statistical perspective.<n>We survey major classes of modern generative models, their intended use cases, and the benefits they offer.<n>We examine common pitfalls that arise when synthetic data are treated as surrogates for real observations.
  arXiv  Detail & Related papers  (2026-03-05T17:24:41Z)
• Mitra: Mixed Synthetic Priors for Enhancing Tabular Foundation Models [85.64873567417396]
  We introduce Mitra, a TFM trained on a curated mixture of synthetic priors selected for their diversity, distinctiveness, and performance on real-world data.<n>Mitra consistently outperforms state-of-the-art TFMs, such as TabPFNv2 and TabICL, across both classification and regression benchmarks.
  arXiv  Detail & Related papers  (2025-10-24T07:15:06Z)
• Valid Inference with Imperfect Synthetic Data [39.10587411316875]
  We introduce a new estimator based on generalized method of moments.<n>We find that interactions between the moment residuals of synthetic data and those of real data can greatly improve estimates of the target parameter.
  arXiv  Detail & Related papers  (2025-08-08T18:32:52Z)
• A Generalized Theory of Mixup for Structure-Preserving Synthetic Data [2.184775414778289]
  We show that mixup can distort key statistical properties such as variance, potentially leading to unintended consequences in data synthesis.<n>We propose a novel mixup method that incorporates a generalized and flexible weighting scheme, better preserving the original data's structure.<n> Numerical experiments confirm that the new approach not only preserves the statistical characteristics of the original data but also sustains model performance across repeated synthesis.
  arXiv  Detail & Related papers  (2025-03-03T14:28:50Z)
• A Theoretical Perspective: How to Prevent Model Collapse in Self-consuming Training Loops [55.07063067759609]
  High-quality data is essential for training large generative models, yet the vast reservoir of real data available online has become nearly depleted.<n>Models increasingly generate their own data for further training, forming Self-consuming Training Loops (STLs)<n>Some models degrade or even collapse, while others successfully avoid these failures, leaving a significant gap in theoretical understanding.
  arXiv  Detail & Related papers  (2025-02-26T06:18:13Z)
• Towards a Theoretical Understanding of Synthetic Data in LLM Post-Training: A Reverse-Bottleneck Perspective [9.590540796223715]
  We show that the generalization capability of the post-trained model is critically determined by the information gain derived from the generative model.<n>This analysis serves as a theoretical foundation for synthetic data generation and highlights its connection with the generalization capability of post-trained models.<n>We open-source our code at https://github.com/ZyGan1999/Towards-a-Theoretical-Understanding-of-Synthetic-Data-in-LLM-Post-Train ing.
  arXiv  Detail & Related papers  (2024-10-02T16:32:05Z)
• Diffusion posterior sampling for simulation-based inference in tall data settings [53.17563688225137]
  Simulation-based inference ( SBI) is capable of approximating the posterior distribution that relates input parameters to a given observation. In this work, we consider a tall data extension in which multiple observations are available to better infer the parameters of the model. We compare our method to recently proposed competing approaches on various numerical experiments and demonstrate its superiority in terms of numerical stability and computational cost.
  arXiv  Detail & Related papers  (2024-04-11T09:23:36Z)
• Towards Theoretical Understandings of Self-Consuming Generative Models [56.84592466204185]
  This paper tackles the emerging challenge of training generative models within a self-consuming loop. We construct a theoretical framework to rigorously evaluate how this training procedure impacts the data distributions learned by future models. We present results for kernel density estimation, delivering nuanced insights such as the impact of mixed data training on error propagation.
  arXiv  Detail & Related papers  (2024-02-19T02:08:09Z)
• The Real Deal Behind the Artificial Appeal: Inferential Utility of Tabular Synthetic Data [40.165159490379146]
  We show that the rate of false-positive findings (type 1 error) will be unacceptably high, even when the estimates are unbiased. Despite the use of a previously proposed correction factor, this problem persists for deep generative models.
  arXiv  Detail & Related papers  (2023-12-13T02:04:41Z)
• On the Stability of Iterative Retraining of Generative Models on their own Data [56.153542044045224]
  We study the impact of training generative models on mixed datasets. We first prove the stability of iterative training under the condition that the initial generative models approximate the data distribution well enough. We empirically validate our theory on both synthetic and natural images by iteratively training normalizing flows and state-of-the-art diffusion models.
  arXiv  Detail & Related papers  (2023-09-30T16:41:04Z)
• Utility Theory of Synthetic Data Generation [12.511220449652384]
  This paper bridges the practice-theory gap by establishing relevant utility theory in a statistical learning framework.<n>It considers two utility metrics: generalization and ranking of models trained on synthetic data.
  arXiv  Detail & Related papers  (2023-05-17T07:49:16Z)
• Synthetic data, real errors: how (not) to publish and use synthetic data [86.65594304109567]
  We show how the generative process affects the downstream ML task. We introduce Deep Generative Ensemble (DGE) to approximate the posterior distribution over the generative process model parameters.
  arXiv  Detail & Related papers  (2023-05-16T07:30:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.