Fundamental limits to learning closed-form mathematical models from data
- URL: http://arxiv.org/abs/2204.02704v1
- Date: Wed, 6 Apr 2022 10:00:33 GMT
- Title: Fundamental limits to learning closed-form mathematical models from data
- Authors: Oscar Fajardo-Fontiveros, Ignasi Reichardt, Harry R. De Los Rios,
Jordi Duch, Marta Sales-Pardo, Roger Guimera
- Abstract summary: Given a noisy dataset, when is it possible to learn the true generating model from the data alone?
We show that this problem displays a transition from a low-noise phase in which the true model can be learned, to a phase in which the observation noise is too high for the true model to be learned by any method.
- Score: 0.0
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Given a finite and noisy dataset generated with a closed-form mathematical
model, when is it possible to learn the true generating model from the data
alone? This is the question we investigate here. We show that this
model-learning problem displays a transition from a low-noise phase in which
the true model can be learned, to a phase in which the observation noise is too
high for the true model to be learned by any method. Both in the low-noise
phase and in the high-noise phase, probabilistic model selection leads to
optimal generalization to unseen data. This is in contrast to standard machine
learning approaches, including artificial neural networks, which are limited,
in the low-noise phase, by their ability to interpolate. In the transition
region between the learnable and unlearnable phases, generalization is hard for
all approaches including probabilistic model selection.
Related papers
- Structure learning with Temporal Gaussian Mixture for model-based Reinforcement Learning [0.9217021281095907]
We propose a temporal Gaussian Mixture Model composed of a perception model and a transition model.
The perception model extracts discrete (latent) states from continuous observations using a variational Gaussian mixture likelihood.
The transition model learns the temporal transition between consecutive time steps by taking advantage of the Dirichlet-categorical conjugacy.
Finally, decision making is performed with a variant of Q-learning which is able to learn Q-values from beliefs over states.
arXiv Detail & Related papers (2024-11-18T12:16:03Z) - Heat Death of Generative Models in Closed-Loop Learning [63.83608300361159]
We study the learning dynamics of generative models that are fed back their own produced content in addition to their original training dataset.
We show that, unless a sufficient amount of external data is introduced at each iteration, any non-trivial temperature leads the model to degenerate.
arXiv Detail & Related papers (2024-04-02T21:51:39Z) - Towards Theoretical Understandings of Self-Consuming Generative Models [56.84592466204185]
This paper tackles the emerging challenge of training generative models within a self-consuming loop.
We construct a theoretical framework to rigorously evaluate how this training procedure impacts the data distributions learned by future models.
We present results for kernel density estimation, delivering nuanced insights such as the impact of mixed data training on error propagation.
arXiv Detail & Related papers (2024-02-19T02:08:09Z) - Neural Likelihood Approximation for Integer Valued Time Series Data [0.0]
We construct a neural likelihood approximation that can be trained using unconditional simulation of the underlying model.
We demonstrate our method by performing inference on a number of ecological and epidemiological models.
arXiv Detail & Related papers (2023-10-19T07:51:39Z) - Learning to Jump: Thinning and Thickening Latent Counts for Generative
Modeling [69.60713300418467]
Learning to jump is a general recipe for generative modeling of various types of data.
We demonstrate when learning to jump is expected to perform comparably to learning to denoise, and when it is expected to perform better.
arXiv Detail & Related papers (2023-05-28T05:38:28Z) - Learning from aggregated data with a maximum entropy model [73.63512438583375]
We show how a new model, similar to a logistic regression, may be learned from aggregated data only by approximating the unobserved feature distribution with a maximum entropy hypothesis.
We present empirical evidence on several public datasets that the model learned this way can achieve performances comparable to those of a logistic model trained with the full unaggregated data.
arXiv Detail & Related papers (2022-10-05T09:17:27Z) - Evaluating State-of-the-Art Classification Models Against Bayes
Optimality [106.50867011164584]
We show that we can compute the exact Bayes error of generative models learned using normalizing flows.
We use our approach to conduct a thorough investigation of state-of-the-art classification models.
arXiv Detail & Related papers (2021-06-07T06:21:20Z) - Distilling Interpretable Models into Human-Readable Code [71.11328360614479]
Human-readability is an important and desirable standard for machine-learned model interpretability.
We propose to train interpretable models using conventional methods, and then distill them into concise, human-readable code.
We describe a piecewise-linear curve-fitting algorithm that produces high-quality results efficiently and reliably across a broad range of use cases.
arXiv Detail & Related papers (2021-01-21T01:46:36Z) - Variational Mixture of Normalizing Flows [0.0]
Deep generative models, such as generative adversarial networks autociteGAN, variational autoencoders autocitevaepaper, and their variants, have seen wide adoption for the task of modelling complex data distributions.
Normalizing flows have overcome this limitation by leveraging the change-of-suchs formula for probability density functions.
The present work overcomes this by using normalizing flows as components in a mixture model and devising an end-to-end training procedure for such a model.
arXiv Detail & Related papers (2020-09-01T17:20:08Z) - BayesFlow: Learning complex stochastic models with invertible neural
networks [3.1498833540989413]
We propose a novel method for globally amortized Bayesian inference based on invertible neural networks.
BayesFlow incorporates a summary network trained to embed the observed data into maximally informative summary statistics.
We demonstrate the utility of BayesFlow on challenging intractable models from population dynamics, epidemiology, cognitive science and ecology.
arXiv Detail & Related papers (2020-03-13T13:39:31Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.