Achieving Occam's Razor: Deep Learning for Optimal Model Reduction

• URL: http://arxiv.org/abs/2303.13746v2
• Date: Thu, 7 Sep 2023 02:43:57 GMT
• Title: Achieving Occam's Razor: Deep Learning for Optimal Model Reduction
• Abstract summary: Occam's razor refers to the principle that good models should exclude parameters beyond those minimally required to describe the systems they represent. This is because redundancy can lead to incorrect estimates of model parameters from data, and thus inaccurate or ambiguous conclusions. We show how deep learning can be powerfully leveraged to address Occam's razor.
• Score: 0.07499722271664146
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: All fields of science depend on mathematical models. Occam's razor refers to the principle that good models should exclude parameters beyond those minimally required to describe the systems they represent. This is because redundancy can lead to incorrect estimates of model parameters from data, and thus inaccurate or ambiguous conclusions. Here, we show how deep learning can be powerfully leveraged to address Occam's razor. FixFit, our new method, uses a feedforward deep neural network with a bottleneck layer to characterize and predict the behavior of a given model from its input parameters. FixFit has three major benefits. First, it provides a metric to quantify the original model's degree of complexity. Second, it allows for the unique fitting of data. Third, it provides an unbiased way to discriminate between experimental hypotheses that add value versus those that do not. In two use cases, we demonstrate the broad applicability of this method across scientific domains. To validate the method using a known system, we apply FixFit to recover known composite parameters for the Kepler orbit model. To illustrate how the method can be applied to less well-established fields, we use it to identify parameters for a multi-scale brain model and reduce the search space for viable candidate mechanisms.

Related papers

• Discriminative versus Generative Approaches to Simulation-based Inference [0.19999259391104385]
  Deep learning has enabled unbinned and high-dimensional parameter estimation. We compare two approaches for neural simulation-based inference (N SBI) We find that both the direct likelihood and likelihood ratio estimation are able to effectively extract parameters with reasonable uncertainties.
  arXiv  Detail & Related papers  (2025-03-11T01:38:54Z)
• An Iterative Bayesian Approach for System Identification based on Linear Gaussian Models [86.05414211113627]
  We tackle the problem of system identification, where we select inputs, observe the corresponding outputs from the true system, and optimize the parameters of our model to best fit the data. We propose a flexible and computationally tractable methodology that is compatible with any system and parametric family of models.
  arXiv  Detail & Related papers  (2025-01-28T01:57:51Z)
• On the Effectiveness of Parameter-Efficient Fine-Tuning [79.6302606855302]
  Currently, many research works propose to only fine-tune a small portion of the parameters while keeping most of the parameters shared across different tasks. We show that all of the methods are actually sparse fine-tuned models and conduct a novel theoretical analysis of them. Despite the effectiveness of sparsity grounded by our theory, it still remains an open problem of how to choose the tunable parameters.
  arXiv  Detail & Related papers  (2022-11-28T17:41:48Z)
• Bayesian Learning of Coupled Biogeochemical-Physical Models [28.269731698116257]
  Predictive models for marine ecosystems are used for a variety of needs. Due to sparse measurements and limited understanding of the myriad of ocean processes, there is significant uncertainty. We develop a Bayesian model learning methodology that allows handling in the space of candidate models and discovery of new models.
  arXiv  Detail & Related papers  (2022-11-12T17:49:18Z)
• Neural parameter calibration for large-scale multi-agent models [0.7734726150561089]
  We present a method to retrieve accurate probability densities for parameters using neural equations. The two combined create a powerful tool that can quickly estimate densities on model parameters, even for very large systems.
  arXiv  Detail & Related papers  (2022-09-27T17:36:26Z)
• Human Trajectory Prediction via Neural Social Physics [63.62824628085961]
  Trajectory prediction has been widely pursued in many fields, and many model-based and model-free methods have been explored. We propose a new method combining both methodologies based on a new Neural Differential Equation model. Our new model (Neural Social Physics or NSP) is a deep neural network within which we use an explicit physics model with learnable parameters.
  arXiv  Detail & Related papers  (2022-07-21T12:11:18Z)
• Inverting brain grey matter models with likelihood-free inference: a tool for trustable cytoarchitecture measurements [62.997667081978825]
  characterisation of the brain grey matter cytoarchitecture with quantitative sensitivity to soma density and volume remains an unsolved challenge in dMRI. We propose a new forward model, specifically a new system of equations, requiring a few relatively sparse b-shells. We then apply modern tools from Bayesian analysis known as likelihood-free inference (LFI) to invert our proposed model.
  arXiv  Detail & Related papers  (2021-11-15T09:08:27Z)
• Training Structured Mechanical Models by Minimizing Discrete Euler-Lagrange Residual [36.52097893036073]
  Structured Mechanical Models (SMMs) are a data-efficient black-box parameterization of mechanical systems. We propose a methodology for fitting SMMs to data by minimizing the discrete Euler-Lagrange residual. Experiments show that our methodology learns models that are better in accuracy to those of the conventional schemes for fitting SMMs.
  arXiv  Detail & Related papers  (2021-05-05T00:44:01Z)
• Leveraging Global Parameters for Flow-based Neural Posterior Estimation [90.21090932619695]
  Inferring the parameters of a model based on experimental observations is central to the scientific method. A particularly challenging setting is when the model is strongly indeterminate, i.e., when distinct sets of parameters yield identical observations. We present a method for cracking such indeterminacy by exploiting additional information conveyed by an auxiliary set of observations sharing global parameters.
  arXiv  Detail & Related papers  (2021-02-12T12:23:13Z)
• Design of Dynamic Experiments for Black-Box Model Discrimination [72.2414939419588]
  Consider a dynamic model discrimination setting where we wish to chose: (i) what is the best mechanistic, time-varying model and (ii) what are the best model parameter estimates. For rival mechanistic models where we have access to gradient information, we extend existing methods to incorporate a wider range of problem uncertainty. We replace these black-box models with Gaussian process surrogate models and thereby extend the model discrimination setting to additionally incorporate rival black-box model.
  arXiv  Detail & Related papers  (2021-02-07T11:34:39Z)
• Provable Benefits of Overparameterization in Model Compression: From Double Descent to Pruning Neural Networks [38.153825455980645]
  Recent empirical evidence indicates that the practice of overization not only benefits training large models, but also assists - perhaps counterintuitively - building lightweight models. This paper sheds light on these empirical findings by theoretically characterizing the high-dimensional toolsets of model pruning. We analytically identify regimes in which, even if the location of the most informative features is known, we are better off fitting a large model and then pruning.
  arXiv  Detail & Related papers  (2020-12-16T05:13:30Z)
• The data-driven physical-based equations discovery using evolutionary approach [77.34726150561087]
  We describe the algorithm for the mathematical equations discovery from the given observations data. The algorithm combines genetic programming with the sparse regression. It could be used for governing analytical equation discovery as well as for partial differential equations (PDE) discovery.
  arXiv  Detail & Related papers  (2020-04-03T17:21:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.