Thermodynamic Isomorphism of Transformers: A Lagrangian Approach to Attention Dynamics

• URL: http://arxiv.org/abs/2602.08216v2
• Date: Fri, 13 Feb 2026 05:45:50 GMT
• Title: Thermodynamic Isomorphism of Transformers: A Lagrangian Approach to Attention Dynamics
• Authors: Gunn Kim,
• Abstract summary: We show that within the Shannon--Boltzmann entropy framework, the Softmax function arises as a stationary solution minimizing a Helmholtz free energy functional.<n> Extending this mapping to macroscopic observables, we define an effective specific heat associated with fluctuations of the attention energy landscape.<n>Our framework provides a unified statistical-mechanical perspective on attention scaling, training dynamics, and positional encoding.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We propose an effective field-theoretic framework for analyzing Transformer attention through a thermodynamic lens. By constructing a Lagrangian on the information manifold equipped with the Fisher metric, we show that, within the Shannon--Boltzmann entropy framework, the Softmax function arises as a stationary solution minimizing a Helmholtz free energy functional. This establishes a formal correspondence between scaled dot-product attention and canonical ensemble statistics. Extending this mapping to macroscopic observables, we define an effective specific heat associated with fluctuations of the attention energy landscape. In controlled experiments on the modular addition task ($p = 19$--$113$), we observe a robust peak in this fluctuation measure that consistently precedes the onset of generalization. While no asymptotic power-law divergence is detected in this finite-depth regime, the reproducible enhancement of energy variance suggests a critical-like crossover accompanying representational reorganization. Our framework provides a unified statistical-mechanical perspective on attention scaling, training dynamics, and positional encoding, interpreting the phenomena as emergent properties of an effective thermodynamic system rather than isolated heuristics. Although the present results indicate finite-size crossover behavior rather than a strict phase transition, they motivate further investigation into scaling limits of deep architectures through fluctuation-based observables.

Related papers

• Universal Sensitivity Bound for Thermal Quantum Dynamic Sensing [7.847010035336918]
  We show that the dynamic quantum Fisher information for a thermal probe state is upper bounded by the degree of non-commutation between the transformed local generator and the Hamiltonian for the thermal state.<n>This upper bound scales as the square of the product of the inverse temperature and the evolution time.<n>In the low-temperature limit, we establish an additional upper bound expressed as the seminorm of the commutator divided by the energy gap.
  arXiv  Detail & Related papers  (2025-12-02T03:19:21Z)
• Information Physics of Intelligence: Unifying Logical Depth and Entropy under Thermodynamic Constraints [7.411478588468014]
  We propose a theoretical framework that treats information processing as an enabling mapping from ontological states to carrier states.<n>We introduce a novel metric, Derivation Entropy, which quantifies the effective work required to compute a target state from a given logical depth.<n>Our findings suggest that the minimization of Derivation Entropy is a governing principle for the evolution of both biological and artificial intelligence.
  arXiv  Detail & Related papers  (2025-11-24T14:24:08Z)
• Mimicking the Physicist's Eye:A VLM-centric Approach for Physics Formula Discovery [98.58830663687911]
  VIPERR-aq1 is a multimodal model that performs Visual Induction for Equation Reasoning.<n>It integrates visual perception, trajectory data, and symbolic reasoning to emulate the scientific discovery process.<n>It consistently outperforms state-of-the-art VLM baselines in accuracy and interpretability.
  arXiv  Detail & Related papers  (2025-08-24T14:34:21Z)
• Thermodynamic Analysis for Harmonic Oscillator with Position-Dependent Mass [0.0]
  We show that increasing alpha leads to a decrease in entropy and specific heat, reflecting a confinement-induced reduction in the number of accessible states.<n>This study underscores the influence of mass deformation on quantum thermal responses and demonstrates that, while the overall thermodynamic trends are consistent with those reported in the literature, certain distinctive features emerge due to the specific form of the deformation.
  arXiv  Detail & Related papers  (2025-05-21T20:00:44Z)
• Eigenstate Thermalization Hypothesis correlations via non-linear Hydrodynamics [0.0]
  We provide a prediction for the late-time behavior of time-ordered free cumulants in the thermodynamic limit.<n>Good agreement is observed in both infinite and finite-temperature regimes.
  arXiv  Detail & Related papers  (2025-05-11T06:35:16Z)
• Pioneer: Physics-informed Riemannian Graph ODE for Entropy-increasing Dynamics [61.70424540412608]
  We present a physics-informed graph ODE for a wide range of entropy-increasing dynamic systems.<n>We report the provable entropy non-decreasing of our formulation, obeying the physics laws.<n> Empirical results show the superiority of Pioneer on real datasets.
  arXiv  Detail & Related papers  (2025-02-05T14:54:30Z)
• Quantum thermodynamics as a gauge theory [0.0]
  A gauge theory for quantum thermodynamics was introduced, defining gauge invariant work and heat.<n>We extend that theory in two significant ways, incorporating energy spectrum degeneracies, which were previously overlooked.<n>This results in a complete framework for quantum thermodynamics grounded in the principle of gauge invariance.
  arXiv  Detail & Related papers  (2024-09-12T00:46:48Z)
• Heat and Work in Quantum Thermodynamics: a Cybernetic Approach [0.0]
  We present a new proposal for distinguishing heat from work based on a control-theoretic observability decomposition. We derive a Hermitian operator representing instantaneous dissipation of observable energy, and suggest a generalization of the von-Neumann entropy.
  arXiv  Detail & Related papers  (2024-03-04T13:26:48Z)
• Physics-Encoded Graph Neural Networks for Deformation Prediction under Contact [87.69278096528156]
  In robotics, it's crucial to understand object deformation during tactile interactions. We introduce a method using Physics-Encoded Graph Neural Networks (GNNs) for such predictions. We've made our code and dataset public to advance research in robotic simulation and grasping.
  arXiv  Detail & Related papers  (2024-02-05T19:21:52Z)
• Dynamically Emergent Quantum Thermodynamics: Non-Markovian Otto Cycle [49.1574468325115]
  We revisit the thermodynamic behavior of the quantum Otto cycle with a focus on memory effects and strong system-bath couplings. Our investigation is based on an exact treatment of non-Markovianity by means of an exact quantum master equation.
  arXiv  Detail & Related papers  (2023-08-18T11:00:32Z)
• Quantum Effects on the Synchronization Dynamics of the Kuramoto Model [62.997667081978825]
  We show that quantum fluctuations hinder the emergence of synchronization, albeit not entirely suppressing it. We derive an analytical expression for the critical coupling, highlighting its dependence on the model parameters.
  arXiv  Detail & Related papers  (2023-06-16T16:41:16Z)
• Spreading of a local excitation in a Quantum Hierarchical Model [62.997667081978825]
  We study the dynamics of the quantum Dyson hierarchical model in its paramagnetic phase. An initial state made by a local excitation of the paramagnetic ground state is considered. A localization mechanism is found and the excitation remains close to its initial position at arbitrary times.
  arXiv  Detail & Related papers  (2022-07-14T10:05:20Z)
• Photoinduced prethermal order parameter dynamics in the two-dimensional large-$N$ Hubbard-Heisenberg model [77.34726150561087]
  We study the microscopic dynamics of competing ordered phases in a two-dimensional correlated electron model. We simulate the light-induced transition between two competing phases.
  arXiv  Detail & Related papers  (2022-05-13T13:13:31Z)
• Accessing the topological Mott insulator in cold atom quantum simulators with realistic Rydberg dressing [58.720142291102135]
  We investigate a realistic scenario for the quantum simulation of such systems using cold Rydberg-dressed atoms in optical lattices. We perform a detailed analysis of the phase diagram at half- and incommensurate fillings, in the mean-field approximation. We furthermore study the stability of the phases with respect to temperature within the mean-field approximation.
  arXiv  Detail & Related papers  (2022-03-28T14:55:28Z)
• Open-system approach to nonequilibrium quantum thermodynamics at arbitrary coupling [77.34726150561087]
  We develop a general theory describing the thermodynamical behavior of open quantum systems coupled to thermal baths. Our approach is based on the exact time-local quantum master equation for the reduced open system states.
  arXiv  Detail & Related papers  (2021-09-24T11:19:22Z)
• The problem of engines in statistical physics [62.997667081978825]
  Engines are open systems that can generate work cyclically, at the expense of an external disequilibrium. Recent advances in the theory of open quantum systems point to a more realistic description of autonomous engines. We show how the external loading force and the thermal noise may be incorporated into the relevant equations of motion.
  arXiv  Detail & Related papers  (2021-08-17T03:59:09Z)
• Out-of-time-order correlations and the fine structure of eigenstate thermalisation [58.720142291102135]
  Out-of-time-orderors (OTOCs) have become established as a tool to characterise quantum information dynamics and thermalisation. We show explicitly that the OTOC is indeed a precise tool to explore the fine details of the Eigenstate Thermalisation Hypothesis (ETH) We provide an estimation of the finite-size scaling of $omega_textrmGOE$ for the general class of observables composed of sums of local operators in the infinite-temperature regime.
  arXiv  Detail & Related papers  (2021-03-01T17:51:46Z)
• Driven-dissipative Ising Model: An exact field-theoretical analysis [0.0]
  Driven-dissipative many-body systems are difficult to analyze analytically due to their non-equilibrium dynamics, dissipation and many-body interactions. We develop an exact field-theoretical analysis and a diagrammatic representation of the spin model that can be understood from a simple scattering picture.
  arXiv  Detail & Related papers  (2021-01-13T19:00:21Z)
• Probing eigenstate thermalization in quantum simulators via fluctuation-dissipation relations [77.34726150561087]
  The eigenstate thermalization hypothesis (ETH) offers a universal mechanism for the approach to equilibrium of closed quantum many-body systems. Here, we propose a theory-independent route to probe the full ETH in quantum simulators by observing the emergence of fluctuation-dissipation relations. Our work presents a theory-independent way to characterize thermalization in quantum simulators and paves the way to quantum simulate condensed matter pump-probe experiments.
  arXiv  Detail & Related papers  (2020-07-20T18:00:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.