Thermodynamics of Reinforcement Learning Curricula

• URL: http://arxiv.org/abs/2603.12324v1
• Date: Thu, 12 Mar 2026 18:00:03 GMT
• Title: Thermodynamics of Reinforcement Learning Curricula
• Abstract summary: We use non-equilibrium thermodynamics to formalize curriculum learning in reinforcement learning.<n>In particular, we propose a geometric framework for RL by interpreting reward parameters as coordinates on a task manifold.<n>We show that, by minimizing the excess thermodynamic work, optimal curricula correspond to geodesics in this task space.
• Score: 5.008597638379228
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Connections between statistical mechanics and machine learning have repeatedly proven fruitful, providing insight into optimization, generalization, and representation learning. In this work, we follow this tradition by leveraging results from non-equilibrium thermodynamics to formalize curriculum learning in reinforcement learning (RL). In particular, we propose a geometric framework for RL by interpreting reward parameters as coordinates on a task manifold. We show that, by minimizing the excess thermodynamic work, optimal curricula correspond to geodesics in this task space. As an application of this framework, we provide an algorithm, "MEW" (Minimum Excess Work), to derive a principled schedule for temperature annealing in maximum-entropy RL.

Related papers

• Bridging Control, Inference, Transport, and Thermodynamics: From Theory to Applications in Learning [25.727025115930825]
  The last decade has seen the development of powerful methods for learning complex structure from high-dimensional data.<n>These advances have brought to the foreground fundamental connections between subdisciplines of physics, applied mathematics, and machine learning.<n>We highlight a conceptual thread that links five distinct fields: control theory, optimal transport, probabilistic inference, non-equilibrium thermodynamics, and machine learning.
  arXiv  Detail & Related papers  (2026-09-14T17:16:33Z)
• On the Entropy Dynamics in Reinforcement Fine-Tuning of Large Language Models [54.61810451777578]
  Entropy serves as a critical metric for measuring the diversity of outputs generated by large language models.<n>Recent studies increasingly focus on monitoring and adjusting entropy to better balance exploration and exploitation in reinforcement fine-tuning.
  arXiv  Detail & Related papers  (2026-02-03T11:14:58Z)
• Thermodynamically Optimal Regularization under Information-Geometric Constraints [0.6345523830122167]
  Modern machine learning relies on a collection of empirically successful but theoretically heterogeneous regularization techniques.<n>We propose a unifying theoretical framework connecting thermodynamic optimality, information geometry, and regularization.
  arXiv  Detail & Related papers  (2026-01-24T06:26:18Z)
• Training thermodynamic computers by gradient descent [0.0]
  We show how to adjust the parameters of a thermodynamic computer by descent in order to perform a desired computation.<n>We estimate the thermodynamic advantage -- the ratio of energy costs of the digital and thermodynamic implementations -- to exceed seven orders of magnitude.
  arXiv  Detail & Related papers  (2025-09-18T18:12:55Z)
• Performance of machine-learning-assisted Monte Carlo in sampling from simple statistical physics models [0.38233569758620045]
  We study the widely-used Sequential Tempering procedure applied to a shallow MADE architecture for the Curie-Weiss model.<n>This work establishes a clear theoretical basis for the integration of machine learning techniques into Monte Carlo sampling and optimization.
  arXiv  Detail & Related papers  (2025-05-28T17:13:11Z)
• Enhancing Spectrum Efficiency in 6G Satellite Networks: A GAIL-Powered Policy Learning via Asynchronous Federated Inverse Reinforcement Learning [67.95280175998792]
  A novel adversarial imitation learning (GAIL)-powered policy learning approach is proposed for optimizing beamforming, spectrum allocation, and remote user equipment (RUE) association ins. We employ inverse RL (IRL) to automatically learn reward functions without manual tuning. We show that the proposed MA-AL method outperforms traditional RL approaches, achieving a $14.6%$ improvement in convergence and reward value.
  arXiv  Detail & Related papers  (2024-09-27T13:05:02Z)
• Temperature Balancing, Layer-wise Weight Analysis, and Neural Network Training [58.20089993899729]
  This paper proposes TempBalance, a straightforward yet effective layerwise learning rate method. We show that TempBalance significantly outperforms ordinary SGD and carefully-tuned spectral norm regularization. We also show that TempBalance outperforms a number of state-of-the-art metrics and schedulers.
  arXiv  Detail & Related papers  (2023-12-01T05:38:17Z)
• Provable Reward-Agnostic Preference-Based Reinforcement Learning [61.39541986848391]
  Preference-based Reinforcement Learning (PbRL) is a paradigm in which an RL agent learns to optimize a task using pair-wise preference-based feedback over trajectories. We propose a theoretical reward-agnostic PbRL framework where exploratory trajectories that enable accurate learning of hidden reward functions are acquired.
  arXiv  Detail & Related papers  (2023-05-29T15:00:09Z)
• Entropy Regularized Reinforcement Learning Using Large Deviation Theory [3.058685580689605]
  In this paper, we establish a mapping between entropy-regularized RL and research in non-equilibrium statistical mechanics. We apply approaches from large deviation theory to derive exact analytical results for the optimal policy and optimal dynamics. The results lead to a novel analytical and computational framework for entropy-regularized RL which is validated by simulations.
  arXiv  Detail & Related papers  (2021-06-07T19:42:06Z)
• Energy-Efficient and Federated Meta-Learning via Projected Stochastic Gradient Ascent [79.58680275615752]
  We propose an energy-efficient federated meta-learning framework. We assume each task is owned by a separate agent, so a limited number of tasks is used to train a meta-model.
  arXiv  Detail & Related papers  (2021-05-31T08:15:44Z)
• Weighted Entropy Modification for Soft Actor-Critic [95.37322316673617]
  We generalize the existing principle of the maximum Shannon entropy in reinforcement learning (RL) to weighted entropy by characterizing the state-action pairs with some qualitative weights. We propose an algorithm motivated for self-balancing exploration with the introduced weight function, which leads to state-of-the-art performance on Mujoco tasks despite its simplicity in implementation.
  arXiv  Detail & Related papers  (2020-11-18T04:36:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.