Graph neural networks informed locally by thermodynamics

• URL: http://arxiv.org/abs/2405.13093v1
• Date: Tue, 21 May 2024 12:57:10 GMT
• Title: Graph neural networks informed locally by thermodynamics
• Authors: Alicia Tierz, Iciar Alfaro, David González, Francisco Chinesta, Elías Cueto,
• Abstract summary: Thermodynamics-informed neural networks employ inductive biases for enforcement of thermodynamics. A metriplectic evolution of the system is assumed, which provides excellent results. A local version of the metriplectic biases has been developed, which avoids the aforementioned matrix assembly. We apply this framework for examples in the fields of solid and fluid mechanics.
• Score: 3.495246564946556
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Thermodynamics-informed neural networks employ inductive biases for the enforcement of the first and second principles of thermodynamics. To construct these biases, a metriplectic evolution of the system is assumed. This provides excellent results, when compared to uninformed, black box networks. While the degree of accuracy can be increased in one or two orders of magnitude, in the case of graph networks, this requires assembling global Poisson and dissipation matrices, which breaks the local structure of such networks. In order to avoid this drawback, a local version of the metriplectic biases has been developed in this work, which avoids the aforementioned matrix assembly, thus preserving the node-by-node structure of the graph networks. We apply this framework for examples in the fields of solid and fluid mechanics. Our approach demonstrates significant computational efficiency and strong generalization capabilities, accurately making inferences on examples significantly different from those encountered during training.

Related papers

• Generalization emerges from local optimization in a self-organized learning network [0.0]
  We design and analyze a new paradigm for building supervised learning networks, driven only by local optimization rules without relying on a global error function. Our network stores new knowledge in the nodes accurately and instantaneously, in the form of a lookup table. We show on numerous examples of classification tasks that the networks generated by our algorithm systematically reach such a state of perfect generalization when the number of learned examples becomes sufficiently large. We report on the dynamics of the change of state and show that it is abrupt and has the distinctive characteristics of a first order phase transition, a phenomenon already observed for traditional learning networks and known as grokking.
  arXiv  Detail & Related papers  (2024-10-03T15:32:08Z)
• Centered Self-Attention Layers [89.21791761168032]
  The self-attention mechanism in transformers and the message-passing mechanism in graph neural networks are repeatedly applied. We show that this application inevitably leads to oversmoothing, i.e., to similar representations at the deeper layers. We present a correction term to the aggregating operator of these mechanisms.
  arXiv  Detail & Related papers  (2023-06-02T15:19:08Z)
• Rank Diminishing in Deep Neural Networks [71.03777954670323]
  Rank of neural networks measures information flowing across layers. It is an instance of a key structural condition that applies across broad domains of machine learning. For neural networks, however, the intrinsic mechanism that yields low-rank structures remains vague and unclear.
  arXiv  Detail & Related papers  (2022-06-13T12:03:32Z)
• Learning Graph Structure from Convolutional Mixtures [119.45320143101381]
  We propose a graph convolutional relationship between the observed and latent graphs, and formulate the graph learning task as a network inverse (deconvolution) problem. In lieu of eigendecomposition-based spectral methods, we unroll and truncate proximal gradient iterations to arrive at a parameterized neural network architecture that we call a Graph Deconvolution Network (GDN) GDNs can learn a distribution of graphs in a supervised fashion, perform link prediction or edge-weight regression tasks by adapting the loss function, and they are inherently inductive.
  arXiv  Detail & Related papers  (2022-05-19T14:08:15Z)
• Thermodynamics-informed graph neural networks [0.09332987715848712]
  We propose using both geometric and thermodynamic inductive biases to improve accuracy and generalization of the resulting integration scheme. The first is achieved with Graph Neural Networks, which induces a non-Euclidean geometrical prior and permutation invariant node and edge update functions. The second bias is forced by learning the GENERIC structure of the problem, an extension of the Hamiltonian formalism, to model more general non-conservative dynamics.
  arXiv  Detail & Related papers  (2022-03-03T17:30:44Z)
• The Implicit Bias of Gradient Descent on Generalized Gated Linear Networks [3.3946853660795893]
  We derive the infinite-time training limit of a mathematically tractable class of deep nonlinear neural networks (GLNs) We show how architectural constraints and the implicit bias of gradient descent affect performance. By making the inductive bias explicit, our framework is poised to inform the development of more efficient, biologically plausible, and robust learning algorithms.
  arXiv  Detail & Related papers  (2022-02-05T22:37:39Z)
• The Principles of Deep Learning Theory [19.33681537640272]
  This book develops an effective theory approach to understanding deep neural networks of practical relevance. We explain how these effectively-deep networks learn nontrivial representations from training. We show that the depth-to-width ratio governs the effective model complexity of the ensemble of trained networks.
  arXiv  Detail & Related papers  (2021-06-18T15:00:00Z)
• A neural anisotropic view of underspecification in deep learning [60.119023683371736]
  We show that the way neural networks handle the underspecification of problems is highly dependent on the data representation. Our results highlight that understanding the architectural inductive bias in deep learning is fundamental to address the fairness, robustness, and generalization of these systems.
  arXiv  Detail & Related papers  (2021-04-29T14:31:09Z)
• Towards Deeper Graph Neural Networks [63.46470695525957]
  Graph convolutions perform neighborhood aggregation and represent one of the most important graph operations. Several recent studies attribute this performance deterioration to the over-smoothing issue. We propose Deep Adaptive Graph Neural Network (DAGNN) to adaptively incorporate information from large receptive fields.
  arXiv  Detail & Related papers  (2020-07-18T01:11:14Z)
• Input-to-State Representation in linear reservoirs dynamics [15.491286626948881]
  Reservoir computing is a popular approach to design recurrent neural networks. The working principle of these networks is not fully understood. A novel analysis of the dynamics of such networks is proposed.
  arXiv  Detail & Related papers  (2020-03-24T00:14:25Z)
• Molecule Property Prediction and Classification with Graph Hypernetworks [113.38181979662288]
  We show that the replacement of the underlying networks with hypernetworks leads to a boost in performance. A major difficulty in the application of hypernetworks is their lack of stability. A recent work has tackled the training instability of hypernetworks in the context of error correcting codes.
  arXiv  Detail & Related papers  (2020-02-01T16:44:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.