Neural Thermodynamic Laws for Large Language Model Training

• URL: http://arxiv.org/abs/2505.10559v1
• Date: Thu, 15 May 2025 17:59:22 GMT
• Title: Neural Thermodynamic Laws for Large Language Model Training
• Authors: Ziming Liu, Yizhou Liu, Jeff Gore, Max Tegmark,
• Abstract summary: We introduce Neural Thermodynamic Laws (NTL) -- a new framework that offers fresh insights into large language models.<n>On the theoretical side, we demonstrate that key thermodynamic quantities (e.g., temperature, entropy, heat capacity, thermal conduction) naturally emerge under river-valley loss landscape assumptions.<n>On the practical side, this scientific perspective yields intuitive guidelines for designing learning rate schedules.
• Score: 13.83966606346023
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Beyond neural scaling laws, little is known about the laws underlying large language models (LLMs). We introduce Neural Thermodynamic Laws (NTL) -- a new framework that offers fresh insights into LLM training dynamics. On the theoretical side, we demonstrate that key thermodynamic quantities (e.g., temperature, entropy, heat capacity, thermal conduction) and classical thermodynamic principles (e.g., the three laws of thermodynamics and the equipartition theorem) naturally emerge under river-valley loss landscape assumptions. On the practical side, this scientific perspective yields intuitive guidelines for designing learning rate schedules.

Related papers

• Pioneer: Physics-informed Riemannian Graph ODE for Entropy-increasing Dynamics [61.70424540412608]
  We present a physics-informed graph ODE for a wide range of entropy-increasing dynamic systems.<n>We report the provable entropy non-decreasing of our formulation, obeying the physics laws.<n> Empirical results show the superiority of Pioneer on real datasets.
  arXiv  Detail & Related papers  (2025-02-05T14:54:30Z)
• Laws of thermodynamics for exponential families [2.092840468195572]
  We develop the laws of thermodynamics in terms of general exponential families.<n>We extend the well-known way in which exponential families characterize thermodynamic and learning equilibria.<n>These ideas have broad implications for quantifying and addressing distribution shift.
  arXiv  Detail & Related papers  (2025-01-03T19:36:09Z)
• Thermodynamics-Consistent Graph Neural Networks [50.0791489606211]
  We propose excess Gibbs free energy graph neural networks (GE-GNNs) for predicting composition-dependent activity coefficients of binary mixtures. The GE-GNN architecture ensures thermodynamic consistency by predicting the molar excess Gibbs free energy. We demonstrate high accuracy and thermodynamic consistency of the activity coefficient predictions.
  arXiv  Detail & Related papers  (2024-07-08T06:58:56Z)
• Learning Neural Constitutive Laws From Motion Observations for Generalizable PDE Dynamics [97.38308257547186]
  Many NN approaches learn an end-to-end model that implicitly models both the governing PDE and material models. We argue that the governing PDEs are often well-known and should be explicitly enforced rather than learned. We introduce a new framework termed "Neural Constitutive Laws" (NCLaw) which utilizes a network architecture that strictly guarantees standard priors.
  arXiv  Detail & Related papers  (2023-04-27T17:42:24Z)
• Gauge Quantum Thermodynamics of Time-local non-Markovian Evolutions [77.34726150561087]
  We deal with a generic time-local non-Markovian master equation. We define current and power to be process-dependent as in classical thermodynamics. Applying the theory to quantum thermal engines, we show that gauge transformations can change the machine efficiency.
  arXiv  Detail & Related papers  (2022-04-06T17:59:15Z)
• Maximum entropy quantum state distributions [58.720142291102135]
  We go beyond traditional thermodynamics and condition on the full distribution of the conserved quantities. The result are quantum state distributions whose deviations from thermal states' get more pronounced in the limit of wide input distributions.
  arXiv  Detail & Related papers  (2022-03-23T17:42:34Z)
• Open-system approach to nonequilibrium quantum thermodynamics at arbitrary coupling [77.34726150561087]
  We develop a general theory describing the thermodynamical behavior of open quantum systems coupled to thermal baths. Our approach is based on the exact time-local quantum master equation for the reduced open system states.
  arXiv  Detail & Related papers  (2021-09-24T11:19:22Z)
• Gauge invariant quantum thermodynamics: consequences for the first law [0.0]
  Information theory plays a major role in the identification of thermodynamic functions. We explicitly construct physically motivated gauge transformations which encode a gentle variant of coarse-graining behind thermodynamics. As a consequence, we reinterpret quantum work and heat, as well as the role of quantum coherence.
  arXiv  Detail & Related papers  (2021-04-20T17:53:16Z)
• The information-theoretic foundation of thermodynamic work extraction [0.0]
  I show that if it is possible to extract work deterministically from a physical system prepared in any one of a set of states, then those states must be distinguishable from one another. This result is formulated independently of scale and of particular dynamical laws. It also provides a novel connection between thermodynamics and information theory, established via the law of conservation of energy.
  arXiv  Detail & Related papers  (2020-09-09T21:58:03Z)
• Tangible phenomenological thermodynamics [4.87717454493713]
  A new rigorous basis for thermodynamics is laid out in the main text and presented in full detail in the appendix. All relevant concepts, such as work, heat, internal energy, heat reservoirs, reversibility, absolute temperature and entropy, are introduced on an abstract level.
  arXiv  Detail & Related papers  (2020-02-20T19:00:04Z)
• First and Second Law of Quantum Thermodynamics: A Consistent Derivation Based on a Microscopic Definition of Entropy [0.0]
  This tutorial focuses on the derivation of the first and second law for closed and open quantum systems far from equilibrium. The derivation is based on a microscopic definition of five essential quantities: internal energy, thermodynamic entropy, work, heat and temperature.
  arXiv  Detail & Related papers  (2020-02-20T15:54:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.