Understanding Deep Learning via Entropy Space Theory

• URL: http://arxiv.org/abs/2608.29279v1
• Date: Sat, 29 Aug 2026 14:05:57 GMT
• Title: Understanding Deep Learning via Entropy Space Theory
• Abstract summary: The entropy space can cover all the possibilities of any deep learning model by topological structure.<n>The theory offers a novel priori framework for mathematical fundamentals of deep learning.
• Score: 14.12103715873018
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deep learning is often criticized for its theoretical research lagging behind practice. To make deep learning easier to understand, the entropy space theory is first introduced here. The entropy space can cover all the possibilities of any deep learning model by topological structure. It is independent of network parameters. Through the designed fundamental operations and norm, entropy space is proven to be a normed space within the formal axiomatic framework. Based on the theory, a unified coordinate system is proposed. It can coordinatize every state of a model and rank them by compression of the maximal value of information entropy. The theory offers a novel priori framework for mathematical fundamentals of deep learning.

Related papers

• Informational Frustration in Neural Manifolds: Shannon Bottlenecks and the Limits of Learnability [0.0]
  This paper introduces a unified framework that links information theory, topology, and statistical mechanics to map the hard limits of deep learning.<n>Central to our approach is the Entropic Learnability Horizon (ELH), a law stating that a network can only truly learn a target function if the Shannon entropy of the data manifold outpaces the topological entropy of the function's decision boundary.<n>We show that the enigmatic phenomenon of "grokking" is actually an Entropic Release, where weights abruptly reorganise to unlock the bottleneck.
  arXiv  Detail & Related papers  (2026-06-29T16:20:02Z)
• There Will Be a Scientific Theory of Deep Learning [62.690977616250954]
  We make the case that a scientific theory of deep learning is emerging.<n>We identify five growing bodies of work that point toward such a theory.<n>We argue that the emerging theory is best thought of as a mechanics of the learning process.
  arXiv  Detail & Related papers  (2026-04-23T13:58:12Z)
• Newton to Einstein: Axiom-Based Discovery via Game Design [55.30047000068118]
  We propose a game design framework in which scientific inquiry is recast as a rule-evolving system.<n>Unlike conventional ML approaches that operate within fixed assumptions, our method enables the discovery of new theoretical structures.
  arXiv  Detail & Related papers  (2025-09-05T18:59:18Z)
• Deep Learning is Not So Mysterious or Different [54.5330466151362]
  We argue that anomalous generalization behaviour is not distinct to neural networks.<n>We present soft inductive biases as a key unifying principle in explaining these phenomena.<n>We also highlight how deep learning is relatively distinct in other ways, such as its ability for representation learning.
  arXiv  Detail & Related papers  (2025-03-03T22:56:04Z)
• Envisioning Future Deep Learning Theories: Some Basic Concepts and Characteristics [30.365274034429508]
  We argue that a future deep learning theory should inherit three characteristics: a textitarchhierically structured network architecture, parameters textititeratively optimized using gradient-based methods, and information from the data that evolves textitcompressively We integrate these characteristics into a graphical model called textitneurashed, which effectively explains some common empirical patterns in deep learning.
  arXiv  Detail & Related papers  (2021-12-17T19:51:26Z)
• Unified Field Theory for Deep and Recurrent Neural Networks [56.735884560668985]
  We present a unified and systematic derivation of the mean-field theory for both recurrent and deep networks. We find that convergence towards the mean-field theory is typically slower for recurrent networks than for deep networks. Our method exposes that Gaussian processes are but the lowest order of a systematic expansion in $1/n$.
  arXiv  Detail & Related papers  (2021-12-10T15:06:11Z)
• An Empirical Investigation into Deep and Shallow Rule Learning [0.0]
  In this paper, we empirically compare deep and shallow rule learning with a uniform general algorithm. Our experiments on both artificial and real-world benchmark data indicate that deep rule networks outperform shallow networks.
  arXiv  Detail & Related papers  (2021-06-18T17:43:17Z)
• Credit Assignment in Neural Networks through Deep Feedback Control [59.14935871979047]
  Deep Feedback Control (DFC) is a new learning method that uses a feedback controller to drive a deep neural network to match a desired output target and whose control signal can be used for credit assignment. The resulting learning rule is fully local in space and time and approximates Gauss-Newton optimization for a wide range of connectivity patterns. To further underline its biological plausibility, we relate DFC to a multi-compartment model of cortical pyramidal neurons with a local voltage-dependent synaptic plasticity rule, consistent with recent theories of dendritic processing.
  arXiv  Detail & Related papers  (2021-06-15T05:30:17Z)
• The Autodidactic Universe [0.8795040582681388]
  We present an approach to cosmology in which the Universe learns its own physical laws. We discover maps that put each of these matrix models in correspondence with both a gauge/gravity theory and a mathematical model of a learning machine. We discuss in detail what it means to say that learning takes place in autodidactic systems, where there is no supervision.
  arXiv  Detail & Related papers  (2021-03-29T02:25:02Z)
• Recent advances in deep learning theory [104.01582662336256]
  This paper reviews and organizes the recent advances in deep learning theory. The literature is categorized in six groups: (1) complexity and capacity-based approaches for analysing the generalizability of deep learning; (2) differential equations and their dynamic systems for modelling gradient descent and its variants; (3) the geometrical structures of the loss landscape that drives the trajectories of the dynamic systems; and (5) theoretical foundations of several special structures in network architectures.
  arXiv  Detail & Related papers  (2020-12-20T14:16:41Z)
• Deep Learning is Singular, and That's Good [31.985399645173022]
  In singular models, the optimal set of parameters forms an analytic set with singularities and classical statistical inference cannot be applied. This is significant for deep learning as neural networks are singular and thus "dividing" by the determinant of the Hessian or employing the Laplace approximation are not appropriate. Despite its potential for addressing fundamental issues in deep learning, singular learning theory appears to have made little inroads into the developing canon of deep learning theory.
  arXiv  Detail & Related papers  (2020-10-22T09:33:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.