Thermodynamically Optimal Regularization under Information-Geometric Constraints

• URL: http://arxiv.org/abs/2601.17330v1
• Date: Sat, 24 Jan 2026 06:26:18 GMT
• Title: Thermodynamically Optimal Regularization under Information-Geometric Constraints
• Authors: Laurent Caraffa,
• Abstract summary: Modern machine learning relies on a collection of empirically successful but theoretically heterogeneous regularization techniques.<n>We propose a unifying theoretical framework connecting thermodynamic optimality, information geometry, and regularization.
• Score: 0.6345523830122167
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Modern machine learning relies on a collection of empirically successful but theoretically heterogeneous regularization techniques, such as weight decay, dropout, and exponential moving averages. At the same time, the rapidly increasing energetic cost of training large models raises the question of whether learning algorithms approach any fundamental efficiency bound. In this work, we propose a unifying theoretical framework connecting thermodynamic optimality, information geometry, and regularization. Under three explicit assumptions -- (A1) that optimality requires an intrinsic, parametrization-invariant measure of information, (A2) that belief states are modeled by maximum-entropy distributions under known constraints, and (A3) that optimal processes are quasi-static -- we prove a conditional optimality theorem. Specifically, the Fisher--Rao metric is the unique admissible geometry on belief space, and thermodynamically optimal regularization corresponds to minimizing squared Fisher--Rao distance to a reference state. We derive the induced geometries for Gaussian and circular belief models, yielding hyperbolic and von Mises manifolds, respectively, and show that classical regularization schemes are structurally incapable of guaranteeing thermodynamic optimality. We introduce a notion of thermodynamic efficiency of learning and propose experimentally testable predictions. This work provides a principled geometric and thermodynamic foundation for regularization in machine learning.

Related papers

• Physics Informed Viscous Value Representations [18.60946729267083]
  We propose a physics-informed regularization of the viscosity solution of the Hamilton-Jacobi-Bellhikeman equation.<n>Our approach grounds the learning process in optimal control theory, explicitly regularizing and bounding updates during value iterations.<n> Experiments demonstrate that our method improves geometric consistency, making it broadly applicable to navigation and high-dimensional, complex manipulation tasks.
  arXiv  Detail & Related papers  (2026-02-26T17:53:46Z)
• ODELoRA: Training Low-Rank Adaptation by Solving Ordinary Differential Equations [54.886931928255564]
  Low-rank adaptation (LoRA) has emerged as a widely adopted parameter-efficient fine-tuning method in deep transfer learning.<n>We propose a novel continuous-time optimization dynamic for LoRA factor matrices in the form of an ordinary differential equation (ODE)<n>We show that ODELoRA achieves stable feature learning, a property that is crucial for training deep neural networks at different scales of problem dimensionality.
  arXiv  Detail & Related papers  (2026-02-07T10:19:36Z)
• Variational Entropic Optimal Transport [67.76725267984578]
  We propose Variational Entropic Optimal Transport (VarEOT) for domain translation problems.<n>VarEOT is based on an exact variational reformulation of the log-partition $log mathbbE[exp(cdot)$ as a tractable generalization over an auxiliary positive normalizer.<n> Experiments on synthetic data and unpaired image-to-image translation demonstrate competitive or improved translation quality.
  arXiv  Detail & Related papers  (2026-02-02T15:48:44Z)
• Learning Geometry: A Framework for Building Adaptive Manifold Models through Metric Optimization [8.201374511929538]
  This paper proposes a novel paradigm for machine learning that moves beyond traditional parameter optimization.<n>We optimize the metric tensor field on a manifold with a predefined topology, thereby dynamically shaping the geometric structure of the model space.<n>This work lays a solid foundation for constructing fully dynamic "meta-learners" capable of autonomously evolving their geometry and topology.
  arXiv  Detail & Related papers  (2025-10-30T01:53:32Z)
• Differentiable Entropy Regularization for Geometry and Neural Networks [6.908972852063454]
  We introduce a differentiable estimator of range-partition entropy, a recent concept from computational geometry.<n>We design EntropyNet, a neural module that restructures data into low-entropy forms to accelerate downstream instance-optimal algorithms.<n>Across tasks, we demonstrate that differentiable entropy improves efficiency without degrading correctness.
  arXiv  Detail & Related papers  (2025-09-03T21:38:22Z)
• Thermodynamic Constraints on the Emergence of Intersubjectivity in Quantum Systems [41.94295877935867]
  Ideal quantum measurement requires divergent thermodynamic resources.<n>This work bridges quantum thermodynamics and the emergence of classicality in the form of intersubjectivity.
  arXiv  Detail & Related papers  (2025-07-28T11:39:10Z)
• Asymptotically Optimal Change Detection for Unnormalized Pre- and Post-Change Distributions [65.38208224389027]
  This paper addresses the problem of detecting changes when only unnormalized pre- and post-change distributions are accessible.<n>Our approach is based on the estimation of the Cumulative Sum statistics, which is known to produce optimal performance.
  arXiv  Detail & Related papers  (2024-10-18T17:13:29Z)
• Learning Generalized Statistical Mechanics with Matrix Product States [41.94295877935867]
  We introduce a variational algorithm based on Matrix Product States that is trained by minimizing a generalized free energy defined using Tsallis entropy instead of the standard Gibbs entropy. As a result, our model can generate the probability distributions associated with generalized statistical mechanics.
  arXiv  Detail & Related papers  (2024-09-12T18:30:45Z)
• Thermodynamics-Consistent Graph Neural Networks [50.0791489606211]
  We propose excess Gibbs free energy graph neural networks (GE-GNNs) for predicting composition-dependent activity coefficients of binary mixtures. The GE-GNN architecture ensures thermodynamic consistency by predicting the molar excess Gibbs free energy. We demonstrate high accuracy and thermodynamic consistency of the activity coefficient predictions.
  arXiv  Detail & Related papers  (2024-07-08T06:58:56Z)
• Discovering Interpretable Physical Models using Symbolic Regression and Discrete Exterior Calculus [55.2480439325792]
  We propose a framework that combines Symbolic Regression (SR) and Discrete Exterior Calculus (DEC) for the automated discovery of physical models. DEC provides building blocks for the discrete analogue of field theories, which are beyond the state-of-the-art applications of SR to physical problems. We prove the effectiveness of our methodology by re-discovering three models of Continuum Physics from synthetic experimental data.
  arXiv  Detail & Related papers  (2023-10-10T13:23:05Z)
• TANGO: Time-Reversal Latent GraphODE for Multi-Agent Dynamical Systems [43.39754726042369]
  We propose a simple-yet-effective self-supervised regularization term as a soft constraint that aligns the forward and backward trajectories predicted by a continuous graph neural network-based ordinary differential equation (GraphODE) It effectively imposes time-reversal symmetry to enable more accurate model predictions across a wider range of dynamical systems under classical mechanics. Experimental results on a variety of physical systems demonstrate the effectiveness of our proposed method.
  arXiv  Detail & Related papers  (2023-10-10T08:52:16Z)
• Thermodynamic geometry of ideal quantum gases: a general framework and a geometric picture of BEC-enhanced heat engines [0.0]
  We show that the standard approach of equilibrium physics can be extended to the slow driving regime in a thermodynamically consistent way. We use a Lindblad-type quantum master equation to work out a dynamical model of a quantum many-body engine using a harmonically trapped Bose gas. Our work paves the way for a more general thermodynamic framework that makes it possible to systematically assess the impact of quantum many-body effects on the performance of thermal machines.
  arXiv  Detail & Related papers  (2022-12-22T23:14:00Z)
• Fractal Structure and Generalization Properties of Stochastic Optimization Algorithms [71.62575565990502]
  We prove that the generalization error of an optimization algorithm can be bounded on the complexity' of the fractal structure that underlies its generalization measure. We further specialize our results to specific problems (e.g., linear/logistic regression, one hidden/layered neural networks) and algorithms.
  arXiv  Detail & Related papers  (2021-06-09T08:05:36Z)
• Jointly Modeling and Clustering Tensors in High Dimensions [6.072664839782975]
  We consider the problem of jointly benchmarking and clustering of tensors. We propose an efficient high-maximization algorithm that converges geometrically to a neighborhood that is within statistical precision.
  arXiv  Detail & Related papers  (2021-04-15T21:06:16Z)
• Free Energy Minimization: A Unified Framework for Modelling, Inference, Learning,and Optimization [42.275148861039895]
  Free energy minimization is first introduced, here and historically, as a thermodynamic principle. The mentioned applications to modelling, inference, learning, and optimization are covered starting from basic principles.
  arXiv  Detail & Related papers  (2020-11-25T11:29:03Z)
• Physics-constrained Bayesian inference of state functions in classical density-functional theory [0.6445605125467573]
  We develop a novel data-driven approach to the inverse problem of classical statistical mechanics. We develop an efficient learning algorithm which characterises the construction of approximate free energy functionals. We consider excluded volume particle interactions, which are ubiquitous in nature, whilst being highly challenging for modelling in terms of free energy.
  arXiv  Detail & Related papers  (2020-10-07T12:43:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.