Stochastic Thermodynamics of Learning Parametric Probabilistic Models

• URL: http://arxiv.org/abs/2310.19802v5
• Date: Wed, 17 Jan 2024 14:45:45 GMT
• Title: Stochastic Thermodynamics of Learning Parametric Probabilistic Models
• Authors: Shervin Sadat Parsi
• Abstract summary: We introduce two information-theoretic metrics: Memorized-information (M-info) and Learned-information (L-info), which trace the flow of information during the learning process of Parametric Probabilistic Models (PPMs) We demonstrate that the accumulation of L-info during the learning process is associated with entropy production, and parameters serve as a heat reservoir in this process, capturing learned information in the form of M-info.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: We have formulated a family of machine learning problems as the time evolution of Parametric Probabilistic Models (PPMs), inherently rendering a thermodynamic process. Our primary motivation is to leverage the rich toolbox of thermodynamics of information to assess the information-theoretic content of learning a probabilistic model. We first introduce two information-theoretic metrics: Memorized-information (M-info) and Learned-information (L-info), which trace the flow of information during the learning process of PPMs. Then, we demonstrate that the accumulation of L-info during the learning process is associated with entropy production, and parameters serve as a heat reservoir in this process, capturing learned information in the form of M-info.

Related papers

• Thermodynamic Prediction Enabled by Automatic Dataset Building and Machine Learning [0.0]
  New discoveries in chemistry and materials science provide opportunities for machine learning (ML) to take critical roles in accelerating research efficiency.<n>We demonstrate (1) the use of large language models (LLMs) for automated literature reviews, and (2) the training of an ML model to predict chemical knowledge (thermodynamic parameters)<n>This work highlights the transformative potential of integrated ML approaches to reshape chemistry and materials science research.
  arXiv  Detail & Related papers  (2025-07-09T21:33:25Z)
• Model-free quantification of completeness, uncertainties, and outliers in atomistic machine learning using information theory [4.59916193837551]
  atomistic machine learning (ML) often relies on unsupervised learning or model predictions to analyze information contents. Here, we introduce a theoretical framework that provides a rigorous, model-free tool to quantify information contents in atomistic simulations.
  arXiv  Detail & Related papers  (2024-04-18T17:50:15Z)
• Reward Shaping via Diffusion Process in Reinforcement Learning [1.0878040851638]
  I leverage the principles of thermodynamics and system dynamics to explore reward shaping via diffusion processes. This article sheds light on relationships between information entropy, system dynamics, and their influences on entropy production.
  arXiv  Detail & Related papers  (2023-06-20T20:58:33Z)
• A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation Analysis [128.0532113800092]
  We present a mechanistic interpretation of Transformer-based LMs on arithmetic questions. This provides insights into how information related to arithmetic is processed by LMs.
  arXiv  Detail & Related papers  (2023-05-24T11:43:47Z)
• Prediction of liquid fuel properties using machine learning models with Gaussian processes and probabilistic conditional generative learning [56.67751936864119]
  The present work aims to construct cheap-to-compute machine learning (ML) models to act as closure equations for predicting the physical properties of alternative fuels. Those models can be trained using the database from MD simulations and/or experimental measurements in a data-fusion-fidelity approach. The results show that ML models can predict accurately the fuel properties of a wide range of pressure and temperature conditions.
  arXiv  Detail & Related papers  (2021-10-18T14:43:50Z)
• Meta-learning using privileged information for dynamics [66.32254395574994]
  We extend the Neural ODE Process model to use additional information within the Learning Using Privileged Information setting. We validate our extension with experiments showing improved accuracy and calibration on simulated dynamics tasks.
  arXiv  Detail & Related papers  (2021-04-29T12:18:02Z)
• Using Data Assimilation to Train a Hybrid Forecast System that Combines Machine-Learning and Knowledge-Based Components [52.77024349608834]
  We consider the problem of data-assisted forecasting of chaotic dynamical systems when the available data is noisy partial measurements. We show that by using partial measurements of the state of the dynamical system, we can train a machine learning model to improve predictions made by an imperfect knowledge-based model.
  arXiv  Detail & Related papers  (2021-02-15T19:56:48Z)
• Using machine-learning modelling to understand macroscopic dynamics in a system of coupled maps [0.0]
  We consider a case study the macroscopic motion emerging from a system of globally coupled maps. We build a coarse-grained Markov process for the macroscopic dynamics both with a machine learning approach and with a direct numerical computation of the transition probability of the coarse-grained process. We are able to infer important information about the effective dimension of the attractor, the persistence of memory effects and the multi-scale structure of the dynamics.
  arXiv  Detail & Related papers  (2020-11-08T15:38:12Z)
• Learning Manifold Implicitly via Explicit Heat-Kernel Learning [63.354671267760516]
  We propose the concept of implicit manifold learning, where manifold information is implicitly obtained by learning the associated heat kernel. The learned heat kernel can be applied to various kernel-based machine learning models, including deep generative models (DGM) for data generation and Stein Variational Gradient Descent for Bayesian inference.
  arXiv  Detail & Related papers  (2020-10-05T03:39:58Z)
• Learning Dynamical Systems with Side Information [2.28438857884398]
  We present a framework for the problem of learning a dynamical system from noisy observations of a few trajectories. We identify six types of side information that arise naturally in many applications. We demonstrate the added value of side information for learning the dynamics of basic models in physics and cell biology, as well as for learning and controlling the dynamics of a model in epidemiology.
  arXiv  Detail & Related papers  (2020-08-23T23:30:48Z)
• Physics-informed machine learning for composition-process-property alloy design: shape memory alloy demonstration [0.0]
  Machine learning (ML) is shown to predict new alloys and their performances in a high dimensional, multiple-target-property design space. A physics-informed featured engineering approach is shown to enable otherwise poorly performing ML models to perform well with the same data.
  arXiv  Detail & Related papers  (2020-03-04T03:53:55Z)
• A Near-Optimal Gradient Flow for Learning Neural Energy-Based Models [93.24030378630175]
  We propose a novel numerical scheme to optimize the gradient flows for learning energy-based models (EBMs) We derive a second-order Wasserstein gradient flow of the global relative entropy from Fokker-Planck equation. Compared with existing schemes, Wasserstein gradient flow is a smoother and near-optimal numerical scheme to approximate real data densities.
  arXiv  Detail & Related papers  (2019-10-31T02:26:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.