Interpretable models for extrapolation in scientific machine learning
- URL: http://arxiv.org/abs/2212.10283v1
- Date: Fri, 16 Dec 2022 19:33:28 GMT
- Title: Interpretable models for extrapolation in scientific machine learning
- Authors: Eric S. Muckley, James E. Saal, Bryce Meredig, Christopher S. Roper,
and John H. Martin
- Abstract summary: Complex machine learning algorithms often outperform simple regressions in interpolative settings.
We examine the trade-off between model performance and interpretability across a broad range of science and engineering problems.
- Score: 0.0
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Data-driven models are central to scientific discovery. In efforts to achieve
state-of-the-art model accuracy, researchers are employing increasingly complex
machine learning algorithms that often outperform simple regressions in
interpolative settings (e.g. random k-fold cross-validation) but suffer from
poor extrapolation performance, portability, and human interpretability, which
limits their potential for facilitating novel scientific insight. Here we
examine the trade-off between model performance and interpretability across a
broad range of science and engineering problems with an emphasis on materials
science datasets. We compare the performance of black box random forest and
neural network machine learning algorithms to that of single-feature linear
regressions which are fitted using interpretable input features discovered by a
simple random search algorithm. For interpolation problems, the average
prediction errors of linear regressions were twice as high as those of black
box models. Remarkably, when prediction tasks required extrapolation, linear
models yielded average error only 5% higher than that of black box models, and
outperformed black box models in roughly 40% of the tested prediction tasks,
which suggests that they may be desirable over complex algorithms in many
extrapolation problems because of their superior interpretability,
computational overhead, and ease of use. The results challenge the common
assumption that extrapolative models for scientific machine learning are
constrained by an inherent trade-off between performance and interpretability.
Related papers
- Structured Radial Basis Function Network: Modelling Diversity for
Multiple Hypotheses Prediction [51.82628081279621]
Multi-modal regression is important in forecasting nonstationary processes or with a complex mixture of distributions.
A Structured Radial Basis Function Network is presented as an ensemble of multiple hypotheses predictors for regression problems.
It is proved that this structured model can efficiently interpolate this tessellation and approximate the multiple hypotheses target distribution.
arXiv Detail & Related papers (2023-09-02T01:27:53Z) - Learning Active Subspaces and Discovering Important Features with Gaussian Radial Basis Functions Neural Networks [0.0]
We show that precious information is contained in the spectrum of the precision matrix that can be extracted once the training of the model is completed.
We conducted numerical experiments for regression, classification, and feature selection tasks.
Our results demonstrate that the proposed model does not only yield an attractive prediction performance compared to the competitors.
arXiv Detail & Related papers (2023-07-11T09:54:30Z) - Amortized Inference for Causal Structure Learning [72.84105256353801]
Learning causal structure poses a search problem that typically involves evaluating structures using a score or independence test.
We train a variational inference model to predict the causal structure from observational/interventional data.
Our models exhibit robust generalization capabilities under substantial distribution shift.
arXiv Detail & Related papers (2022-05-25T17:37:08Z) - Accelerating Understanding of Scientific Experiments with End to End
Symbolic Regression [12.008215939224382]
We develop a deep neural network to address the problem of learning free-form symbolic expressions from raw data.
We train our neural network on a synthetic dataset consisting of data tables of varying length and varying levels of noise.
We validate our technique by running on a public dataset from behavioral science.
arXiv Detail & Related papers (2021-12-07T22:28:53Z) - X-model: Improving Data Efficiency in Deep Learning with A Minimax Model [78.55482897452417]
We aim at improving data efficiency for both classification and regression setups in deep learning.
To take the power of both worlds, we propose a novel X-model.
X-model plays a minimax game between the feature extractor and task-specific heads.
arXiv Detail & Related papers (2021-10-09T13:56:48Z) - Hessian-based toolbox for reliable and interpretable machine learning in
physics [58.720142291102135]
We present a toolbox for interpretability and reliability, extrapolation of the model architecture.
It provides a notion of the influence of the input data on the prediction at a given test point, an estimation of the uncertainty of the model predictions, and an agnostic score for the model predictions.
Our work opens the road to the systematic use of interpretability and reliability methods in ML applied to physics and, more generally, science.
arXiv Detail & Related papers (2021-08-04T16:32:59Z) - Transfer learning suppresses simulation bias in predictive models built
from sparse, multi-modal data [15.587831925516957]
Many problems in science, engineering, and business require making predictions based on very few observations.
To build a robust predictive model, these sparse data may need to be augmented with simulated data, especially when the design space is multidimensional.
We combine recent developments in deep learning to build more robust predictive models from multimodal data.
arXiv Detail & Related papers (2021-04-19T23:28:32Z) - Goal-directed Generation of Discrete Structures with Conditional
Generative Models [85.51463588099556]
We introduce a novel approach to directly optimize a reinforcement learning objective, maximizing an expected reward.
We test our methodology on two tasks: generating molecules with user-defined properties and identifying short python expressions which evaluate to a given target value.
arXiv Detail & Related papers (2020-10-05T20:03:13Z) - Good Classifiers are Abundant in the Interpolating Regime [64.72044662855612]
We develop a methodology to compute precisely the full distribution of test errors among interpolating classifiers.
We find that test errors tend to concentrate around a small typical value $varepsilon*$, which deviates substantially from the test error of worst-case interpolating model.
Our results show that the usual style of analysis in statistical learning theory may not be fine-grained enough to capture the good generalization performance observed in practice.
arXiv Detail & Related papers (2020-06-22T21:12:31Z) - Designing Accurate Emulators for Scientific Processes using
Calibration-Driven Deep Models [33.935755695805724]
Learn-by-Calibrating (LbC) is a novel deep learning approach for designing emulators in scientific applications.
We show that LbC provides significant improvements in generalization error over widely-adopted loss function choices.
LbC achieves high-quality emulators even in small data regimes and more importantly, recovers the inherent noise structure without any explicit priors.
arXiv Detail & Related papers (2020-05-05T16:54:11Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.