Informational Frustration in Neural Manifolds: Shannon Bottlenecks and the Limits of Learnability

• URL: http://arxiv.org/abs/2606.30512v1
• Date: Mon, 29 Jun 2026 16:20:02 GMT
• Title: Informational Frustration in Neural Manifolds: Shannon Bottlenecks and the Limits of Learnability
• Abstract summary: This paper introduces a unified framework that links information theory, topology, and statistical mechanics to map the hard limits of deep learning.<n>Central to our approach is the Entropic Learnability Horizon (ELH), a law stating that a network can only truly learn a target function if the Shannon entropy of the data manifold outpaces the topological entropy of the function's decision boundary.<n>We show that the enigmatic phenomenon of "grokking" is actually an Entropic Release, where weights abruptly reorganise to unlock the bottleneck.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Why overparameterised deep networks generalise so remarkably well remains one of the most stubborn open questions in machine learning theory. Classical frameworks like VC dimension and Rademacher complexity predict catastrophic overfitting in modern models, leaving a massive theoretical gap between theory and reality. In this paper, we bridge this divide by introducing a unified framework that links information theory, topology, and statistical mechanics to map the hard limits of deep learning. Central to our approach is the Entropic Learnability Horizon (ELH): a fundamental law stating that a network can only truly learn a target function if the Shannon entropy of the data manifold outpaces the topological entropy of the function's decision boundary, balanced by the von Neumann entropy of the network's weight space. We establish the Shannon-Topological Bottleneck Theorem, proving that when a target boundary's geometric complexity exceeds this informational horizon, the system undergoes a sudden entropic phase transition. It falls into a state of Informational Frustration - a glassy, rigid memorization phase where generalization becomes thermodynamically impossible. Using this lens, we show that the enigmatic phenomenon of "grokking" is actually an Entropic Release, where weights abruptly reorganise to unlock the bottleneck. Finally, we translate this theory into practice with Entropic Gradient Descent (EGD), an optimization algorithm that dynamically manages weight entropy to keep learning on track. Ultimately, this work repositions entropy not just as a tool for tracking uncertainty but as the fundamental physical currency that dictates whether a machine can learn.

Related papers

• Understanding Deep Learning via Entropy Space Theory [14.12103715873018]
  The entropy space can cover all the possibilities of any deep learning model by topological structure.<n>The theory offers a novel priori framework for mathematical fundamentals of deep learning.
  arXiv  Detail & Related papers  (2026-08-29T14:05:57Z)
• Human-Centered Learning Mechanics: A Dynamical Framework for Entropy-Regulated Representation Learning [1.1675763847424785]
  We propose a dynamical and information-theoretic framework for open and controlled learning systems.<n>The central idea is that entropy regularization is useful only when the chosen entropy surrogate generates a non-degenerate information force.<n> Controlled representation-learning experiments support the hypothesis that geometric entropy surrogates induce stronger and more stable information forces than softmax-normalized entropy.
  arXiv  Detail & Related papers  (2026-05-21T18:16:39Z)
• Information as Maximum-Caliber Deviation: A bridge between Integrated Information Theory and the Free Energy Principle [51.56484100374058]
  This work proposes that information can be defined as the deviation $$ of realized dynamics from a constrained maximum-caliber (MaxCal) path ensemble over a finite time horizon.<n>Under this definition, each of the cause/effect repertoires central to IIT 3.0 emerge directly from MaxCal variational principles.<n>Information $$ is shown to be equivalent to prediction error under accompanying predictive coding models.
  arXiv  Detail & Related papers  (2026-05-03T07:22:55Z)
• Brain-inspired AI for Edge Intelligence: a systematic review [40.25885510107807]
  Spiking Neural Networks (SNNs) promise to circumvent the severe Size, Weight, and Power (SWaP) constraints of edge intelligence.<n>Currently, theoretical energy gains are frequently negated by the inefficiencies of mapping asynchronous, event-driven dynamics onto traditional von Neumann substrates.<n>This survey adopts a rigorous system-level hardware-software co-design perspective to examine the 2020-2025 trajectory.
  arXiv  Detail & Related papers  (2026-03-19T08:13:49Z)
• Information Physics of Intelligence: Unifying Logical Depth and Entropy under Thermodynamic Constraints [7.411478588468014]
  We propose a theoretical framework that treats information processing as an enabling mapping from ontological states to carrier states.<n>We introduce a novel metric, Derivation Entropy, which quantifies the effective work required to compute a target state from a given logical depth.<n>Our findings suggest that the minimization of Derivation Entropy is a governing principle for the evolution of both biological and artificial intelligence.
  arXiv  Detail & Related papers  (2025-11-24T14:24:08Z)
• Transformers Provably Learn Chain-of-Thought Reasoning with Length Generalization [53.89723291716722]
  A crucial question about AI reasoning is whether models can extrapolate learned reasoning patterns to solve harder tasks with longer chain-of-thought (CoT)<n>We mathematically prove how the algebraic structure of state-tracking problems governs the degree of extrapolation of the learned CoT.<n>We provide the first optimization guarantee that constant-depth transformers provably learn $mathsfNC1$-complete problems with CoT.
  arXiv  Detail & Related papers  (2025-11-10T18:40:24Z)
• Barriers for Learning in an Evolving World: Mathematical Understanding of Loss of Plasticity [41.584807567389625]
  Loss of plasticity (LoP) is the degradation of deep learning models' ability to learn in the future.<n>This work presents a first-principles investigation of LoP in gradient-based learning.
  arXiv  Detail & Related papers  (2025-09-30T21:49:50Z)
• Over-squashing in Spatiotemporal Graph Neural Networks [43.811698922164055]
  Graph Neural Networks (GNNs) have achieved remarkable success across various domains.<n>Recent theoretical advances have identified fundamental limitations in their information propagation, such as over-squashing, where distant nodes fail to effectively exchange information.<n>In this work, we formalize the over-squashing problem and demonstrate its distinct characteristics compared to the static case.<n>Our analysis reveals that, counterintuitively, convolutional STGNNs favor information propagation from points temporally distant rather than close in time.
  arXiv  Detail & Related papers  (2025-06-18T14:45:06Z)
• Binarized Neural Networks Converge Toward Algorithmic Simplicity: Empirical Support for the Learning-as-Compression Hypothesis [33.73453802399709]
  We propose a shift toward algorithmic information theory, using Binarized Neural Networks (BNNs) as a first proxy.<n>We apply the Block Decomposition Method (BDM) and demonstrate it more closely tracks structural changes during training than entropy.<n>These results support the view of training as a process of algorithmic compression, where learning corresponds to the progressive internalization of structured regularities.
  arXiv  Detail & Related papers  (2025-05-27T02:51:36Z)
• Is Grokking a Computational Glass Relaxation? [11.327748614675885]
  We study the phenomenon of grokking, where neural networks abruptly generalize long after the training performance reaches a near-perfect level.<n>We find NO entropy barrier in the memorization-to-generalization transition of grokking, challenging previous theory that defines grokking as a first-order phase transition.<n>Inspired by grokking's far-from-equilibrium nature, we develop a toy WanD based on Wang-landau molecular dynamics, which can eliminate grokking without any constraints and find high-norm generalizing solutions.
  arXiv  Detail & Related papers  (2025-05-16T16:20:02Z)
• The Expressivity of Classical and Quantum Neural Networks on Entanglement Entropy [0.3299672391663526]
  Von Neumann entropy from R'enyi entropies is a challenging problem in quantum field theory. We propose a general framework to tackle this problem using classical and quantum neural networks with supervised learning. Our proposed methods can accurately predict the von Neumann and R'enyi entropies numerically.
  arXiv  Detail & Related papers  (2023-05-01T18:00:01Z)
• Physics informed neural networks for continuum micromechanics [68.8204255655161]
  Recently, physics informed neural networks have successfully been applied to a broad variety of problems in applied mathematics and engineering. Due to the global approximation, physics informed neural networks have difficulties in displaying localized effects and strong non-linear solutions by optimization. It is shown, that the domain decomposition approach is able to accurately resolve nonlinear stress, displacement and energy fields in heterogeneous microstructures obtained from real-world $mu$CT-scans.
  arXiv  Detail & Related papers  (2021-10-14T14:05:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.