Data efficiency and extrapolation trends in neural network interatomic potentials

• URL: http://arxiv.org/abs/2302.05823v2
• Date: Wed, 12 Apr 2023 23:25:14 GMT
• Title: Data efficiency and extrapolation trends in neural network interatomic potentials
• Authors: Joshua A. Vita, Daniel Schwalbe-Koda
• Abstract summary: We show how architectural and optimization choices influence the generalization of neural network interatomic potentials (NNIPs) We show that test errors in NNIP follow a scaling relation and can be robust to noise, but cannot predict MD stability in the high-accuracy regime. Our work provides a deep learning justification for the extrapolation performance of many common NNIPs.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Over the last few years, key architectural advances have been proposed for neural network interatomic potentials (NNIPs), such as incorporating message-passing networks, equivariance, or many-body expansion terms. Although modern NNIP models exhibit small differences in energy/forces errors, improvements in accuracy are still considered the main target when developing new NNIP architectures. In this work, we show how architectural and optimization choices influence the generalization of NNIPs, revealing trends in molecular dynamics (MD) stability, data efficiency, and loss landscapes. Using the 3BPA dataset, we show that test errors in NNIP follow a scaling relation and can be robust to noise, but cannot predict MD stability in the high-accuracy regime. To circumvent this problem, we propose the use of loss landscape visualizations and a metric of loss entropy for predicting the generalization power of NNIPs. With a large-scale study on NequIP and MACE, we show that the loss entropy predicts out-of-distribution error and MD stability despite being computed only on the training set. Using this probe, we demonstrate how the choice of optimizers, loss function weighting, data normalization, and other architectural decisions influence the extrapolation behavior of NNIPs. Finally, we relate loss entropy to data efficiency, demonstrating that flatter landscapes also predict learning curve slopes. Our work provides a deep learning justification for the extrapolation performance of many common NNIPs, and introduces tools beyond accuracy metrics that can be used to inform the development of next-generation models.

Related papers

• DFA-GNN: Forward Learning of Graph Neural Networks by Direct Feedback Alignment [57.62885438406724]
  Graph neural networks are recognized for their strong performance across various applications. BP has limitations that challenge its biological plausibility and affect the efficiency, scalability and parallelism of training neural networks for graph-based tasks. We propose DFA-GNN, a novel forward learning framework tailored for GNNs with a case study of semi-supervised learning.
  arXiv  Detail & Related papers  (2024-06-04T07:24:51Z)
• Deep Neural Networks Tend To Extrapolate Predictably [51.303814412294514]
  neural network predictions tend to be unpredictable and overconfident when faced with out-of-distribution (OOD) inputs. We observe that neural network predictions often tend towards a constant value as input data becomes increasingly OOD. We show how one can leverage our insights in practice to enable risk-sensitive decision-making in the presence of OOD inputs.
  arXiv  Detail & Related papers  (2023-10-02T03:25:32Z)
• On the Generalization of PINNs outside the training domain and the Hyperparameters influencing it [1.3927943269211593]
  PINNs are Neural Network architectures trained to emulate solutions of differential equations without the necessity of solution data. We perform an empirical analysis of the behavior of PINN predictions outside their training domain. We assess whether the algorithmic setup of PINNs can influence their potential for generalization and showcase the respective effect on the prediction.
  arXiv  Detail & Related papers  (2023-02-15T09:51:56Z)
• PINN Training using Biobjective Optimization: The Trade-off between Data Loss and Residual Loss [0.0]
  Physics informed neural networks (PINNs) have proven to be an efficient tool to represent problems for which measured data are available. In this paper, we suggest a multiobjective perspective on the training of PINNs by treating the data loss and the residual loss as two individual objective functions.
  arXiv  Detail & Related papers  (2023-02-03T15:27:50Z)
• Learning Low Dimensional State Spaces with Overparameterized Recurrent Neural Nets [57.06026574261203]
  We provide theoretical evidence for learning low-dimensional state spaces, which can also model long-term memory. Experiments corroborate our theory, demonstrating extrapolation via learning low-dimensional state spaces with both linear and non-linear RNNs.
  arXiv  Detail & Related papers  (2022-10-25T14:45:15Z)
• Adaptive Self-supervision Algorithms for Physics-informed Neural Networks [59.822151945132525]
  Physics-informed neural networks (PINNs) incorporate physical knowledge from the problem domain as a soft constraint on the loss function. We study the impact of the location of the collocation points on the trainability of these models. We propose a novel adaptive collocation scheme which progressively allocates more collocation points to areas where the model is making higher errors.
  arXiv  Detail & Related papers  (2022-07-08T18:17:06Z)
• coVariance Neural Networks [119.45320143101381]
  Graph neural networks (GNN) are an effective framework that exploit inter-relationships within graph-structured data for learning. We propose a GNN architecture, called coVariance neural network (VNN), that operates on sample covariance matrices as graphs. We show that VNN performance is indeed more stable than PCA-based statistical approaches.
  arXiv  Detail & Related papers  (2022-05-31T15:04:43Z)
• Probabilistic AutoRegressive Neural Networks for Accurate Long-range Forecasting [6.295157260756792]
  We introduce the Probabilistic AutoRegressive Neural Networks (PARNN) PARNN is capable of handling complex time series data exhibiting non-stationarity, nonlinearity, non-seasonality, long-range dependence, and chaotic patterns. We evaluate the performance of PARNN against standard statistical, machine learning, and deep learning models, including Transformers, NBeats, and DeepAR.
  arXiv  Detail & Related papers  (2022-04-01T17:57:36Z)
• Robust Learning of Physics Informed Neural Networks [2.86989372262348]
  Physics-informed Neural Networks (PINNs) have been shown to be effective in solving partial differential equations. This paper shows that a PINN can be sensitive to errors in training data and overfit itself in dynamically propagating these errors over the domain of the solution of the PDE.
  arXiv  Detail & Related papers  (2021-10-26T00:10:57Z)
• Entropy-Based Modeling for Estimating Soft Errors Impact on Binarized Neural Network Inference [2.249916681499244]
  We present the relatively-accurate statistical models to delineate the impact of both undertaken single-event upset (SEU) and multi-bit upset (MBU) across layers and per each layer of the selected convolution neural network. These models can be used for evaluating the error-resiliency magnitude of NN topology before adopting them in the safety-critical applications.
  arXiv  Detail & Related papers  (2020-04-10T16:10:24Z)
• Rectified Linear Postsynaptic Potential Function for Backpropagation in Deep Spiking Neural Networks [55.0627904986664]
  Spiking Neural Networks (SNNs) usetemporal spike patterns to represent and transmit information, which is not only biologically realistic but also suitable for ultra-low-power event-driven neuromorphic implementation. This paper investigates the contribution of spike timing dynamics to information encoding, synaptic plasticity and decision making, providing a new perspective to design of future DeepSNNs and neuromorphic hardware systems.
  arXiv  Detail & Related papers  (2020-03-26T11:13:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.