Applications of Entropy in Data Analysis and Machine Learning: A Review

• URL: http://arxiv.org/abs/2503.02921v1
• Date: Tue, 04 Mar 2025 17:43:48 GMT
• Title: Applications of Entropy in Data Analysis and Machine Learning: A Review
• Authors: Salomé A. Sepúveda Fontaine, José M. Amigó,
• Abstract summary: The concept of entropy has permeated other fields of physics and mathematics.<n>The subject of this review is their applications in data analysis and machine learning.
• Score: 0.25782420501870285
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Since its origin in the thermodynamics of the 19th century, the concept of entropy has also permeated other fields of physics and mathematics, such as Classical and Quantum Statistical Mechanics, Information Theory, Probability Theory, Ergodic Theory and the Theory of Dynamical Systems. Specifically, we are referring to the classical entropies: the Boltzmann-Gibbs, von Neumann, Shannon, Kolmogorov-Sinai and topological entropies. In addition to their common name, which is historically justified (as we briefly describe in this review), other commonality of the classical entropies is the important role that they have played and are still playing in the theory and applications of their respective fields and beyond. Therefore, it is not surprising that, in the course of time, many other instances of the overarching concept of entropy have been proposed, most of them tailored to specific purposes. Following the current usage, we will refer to all of them, whether classical or new, simply as entropies. Precisely, the subject of this review is their applications in data analysis and machine learning. The reason for these particular applications is that entropies are very well suited to characterize probability mass distributions, typically generated by finite-state processes or symbolized signals. Therefore, we will focus on entropies defined as positive functionals on probability mass distributions and provide an axiomatic characterization that goes back to Shannon and Khinchin. Given the plethora of entropies in the literature, we have selected a representative group, including the classical ones. The applications summarized in this review finely illustrate the power and versatility of entropy in data analysis and machine learning.

Related papers

• Unification of observational entropy with maximum entropy principles [2.9127054707887967]
  We introduce a definition of coarse-grained entropy that unifies measurement-based (observational entropy) and max-entropy-based (Jaynes) approaches to coarse-graining. We study the dynamics of this entropy in a quantum random matrix model and a classical hard sphere gas.
  arXiv  Detail & Related papers  (2025-03-19T18:00:30Z)
• Pioneer: Physics-informed Riemannian Graph ODE for Entropy-increasing Dynamics [61.70424540412608]
  We present a physics-informed graph ODE for a wide range of entropy-increasing dynamic systems. We report the provable entropy non-decreasing of our formulation, obeying the physics laws. Empirical results show the superiority of Pioneer on real datasets.
  arXiv  Detail & Related papers  (2025-02-05T14:54:30Z)
• Quantum Entropy Prover [17.38531785155932]
  We derive a framework for quantum systems based on the strong sub-additivity and weak monotonicity inequalities for the von-Neumann entropy.<n>Our main contribution is the Python package qITIP, for which we present the theory and demonstrate its capabilities.
  arXiv  Detail & Related papers  (2025-01-27T13:10:23Z)
• Fully quantum stochastic entropy production [2.3895981099137535]
  Building on the approach of thermodynamics, we define entropy production for arbitrary quantum processes.<n>We show that the classical expression for average entropy production involves only comparisons of statistics at the input or output.<n>We construct an entropy production operator, that generalizes the value of entropy to the non-commutative case.
  arXiv  Detail & Related papers  (2024-12-17T02:45:10Z)
• How to Explore with Belief: State Entropy Maximization in POMDPs [40.82741665804367]
  We develop a memory and efficient *policy* method to address a first-order relaxation of the objective defined on ** states. This paper aims to generalize state entropy to more realistic domains that meet the challenges of applications.
  arXiv  Detail & Related papers  (2024-06-04T13:16:34Z)
• Which entropy for general physical theories? [44.99833362998488]
  We address the problem of quantifying the information content of a source for an arbitrary information theory. The functions that solve this problem in classical and quantum theory are Shannon's and von Neumann's entropy, respectively. In a general information theory there are three different functions that extend the notion of entropy, and this opens the question as to whether any of them can universally play the role of the quantifier for the information content.
  arXiv  Detail & Related papers  (2023-02-03T10:55:13Z)
• Bridging the gap between classical and quantum many-body information dynamics [0.0]
  Study sheds light on the nature of information spreading in classical and quantum systems. It opens new avenues for quantum-inspired classical approaches across physics, information theory, and statistics.
  arXiv  Detail & Related papers  (2022-04-06T18:03:29Z)
• Why we should interpret density matrices as moment matrices: the case of (in)distinguishable particles and the emergence of classical reality [69.62715388742298]
  We introduce a formulation of quantum theory (QT) as a general probabilistic theory but expressed via quasi-expectation operators (QEOs) We will show that QT for both distinguishable and indistinguishable particles can be formulated in this way. We will show that finitely exchangeable probabilities for a classical dice are as weird as QT.
  arXiv  Detail & Related papers  (2022-03-08T14:47:39Z)
• Linear growth of the entanglement entropy for quadratic Hamiltonians and arbitrary initial states [11.04121146441257]
  We prove that the entanglement entropy of any pure initial state of a bosonic quantum system grows linearly in time. We discuss several applications of our results to physical systems with (weakly) interacting Hamiltonians and periodically driven quantum systems.
  arXiv  Detail & Related papers  (2021-07-23T07:55:38Z)
• Emergence of classical behavior in the early universe [68.8204255655161]
  Three notions are often assumed to be essentially equivalent, representing different facets of the same phenomenon. We analyze them in general Friedmann-Lemaitre- Robertson-Walker space-times through the lens of geometric structures on the classical phase space. The analysis shows that: (i) inflation does not play an essential role; classical behavior can emerge much more generally; (ii) the three notions are conceptually distinct; classicality can emerge in one sense but not in another.
  arXiv  Detail & Related papers  (2020-04-22T16:38:25Z)
• On Entropy for general quantum systems [0.0]
  We will provide a consistent treatment of entropy which can be applied within the recently developed Orlicz space based approach to large systems. This means that the proposed approach successfully provides a refined framework for the treatment of entropy in each of classical statistical physics, Dirac's formalism of Quantum Mechanics, large systems of quantum statistical physics, and finally also for Quantum Field Theory.
  arXiv  Detail & Related papers  (2018-04-16T09:52:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.