There Will Be a Scientific Theory of Deep Learning

• URL: http://arxiv.org/abs/2604.21691v1
• Date: Thu, 23 Apr 2026 13:58:12 GMT
• Title: There Will Be a Scientific Theory of Deep Learning
• Abstract summary: We make the case that a scientific theory of deep learning is emerging.<n>We identify five growing bodies of work that point toward such a theory.<n>We argue that the emerging theory is best thought of as a mechanics of the learning process.
• Score: 62.690977616250954
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this paper, we make the case that a scientific theory of deep learning is emerging. By this we mean a theory which characterizes important properties and statistics of the training process, hidden representations, final weights, and performance of neural networks. We pull together major strands of ongoing research in deep learning theory and identify five growing bodies of work that point toward such a theory: (a) solvable idealized settings that provide intuition for learning dynamics in realistic systems; (b) tractable limits that reveal insights into fundamental learning phenomena; (c) simple mathematical laws that capture important macroscopic observables; (d) theories of hyperparameters that disentangle them from the rest of the training process, leaving simpler systems behind; and (e) universal behaviors shared across systems and settings which clarify which phenomena call for explanation. Taken together, these bodies of work share certain broad traits: they are concerned with the dynamics of the training process; they primarily seek to describe coarse aggregate statistics; and they emphasize falsifiable quantitative predictions. We argue that the emerging theory is best thought of as a mechanics of the learning process, and suggest the name learning mechanics. We discuss the relationship between this mechanics perspective and other approaches for building a theory of deep learning, including the statistical and information-theoretic perspectives. In particular, we anticipate a symbiotic relationship between learning mechanics and mechanistic interpretability. We also review and address common arguments that fundamental theory will not be possible or is not important. We conclude with a portrait of important open directions in learning mechanics and advice for beginners. We host further introductory materials, perspectives, and open questions at learningmechanics.pub.

Related papers

• Bridging Control, Inference, Transport, and Thermodynamics: From Theory to Applications in Learning [25.727025115930825]
  The last decade has seen the development of powerful methods for learning complex structure from high-dimensional data.<n>These advances have brought to the foreground fundamental connections between subdisciplines of physics, applied mathematics, and machine learning.<n>We highlight a conceptual thread that links five distinct fields: control theory, optimal transport, probabilistic inference, non-equilibrium thermodynamics, and machine learning.
  arXiv  Detail & Related papers  (2026-09-14T17:16:33Z)
• From Observation to Insight: Mechanistic World Models and the Quest for Autonomous Discovery [60.11725132433485]
  We argue that scientific discovery is fundamentally a problem of knowledge organisation.<n>We introduce Mechanistic World Models, a new design paradigm that places reusable mechanisms at the centre of representation, computation and learning.
  arXiv  Detail & Related papers  (2026-07-14T08:03:27Z)
• From Approximation to Emergence: A Theory of Deep Learning [5.863264019032883]
  From Approximation to Emergence is a unified, proof-oriented account of modern deep learning theory.<n>Written for researchers, graduate students, and mathematically trained practitioners, it offers a rigorous map of deep learning theory.
  arXiv  Detail & Related papers  (2026-07-01T17:26:19Z)
• An Introduction to Causal Reinforcement Learning [58.680653905480284]
  Causal inference and reinforcement learning operate over different aspects of the same building block, counterfactual relations.<n>We put under a unifying treatment different modes of learning, including online, off-policy, and causal calculus learning.<n>Specifically, we introduce and discuss through causal lenses generalized policy learning, where to intervene, imitation learning, and counterfactual learning.
  arXiv  Detail & Related papers  (2026-06-23T05:28:33Z)
• Is the end of Insight in Sight ? [0.0]
  A physics-informed neural network (PINN) trained on a rarefied gas dynamics problem governed by the Boltzmann equation.<n>Despite the system's clear structure and well-understood governing laws, the trained network's weights resemble Gaussian-distributed random matrices.<n>This suggests that deep learning and traditional simulation may follow distinct cognitive paths to the same outcome.
  arXiv  Detail & Related papers  (2025-05-07T19:57:36Z)
• Information-Theoretic Foundations for Machine Learning [20.617552198581024]
  We propose a theoretical framework which attempts to provide rigor to existing practices in machine learning.<n>We provide a framework rooted in Bayesian statistics and Shannon's information theory which is general enough to unify the analysis of many phenomena in machine learning.<n>Unlike existing analyses that weaken with increasing data complexity, our theoretical tools provide accurate insights across diverse machine learning settings.
  arXiv  Detail & Related papers  (2024-07-17T03:18:40Z)
• Neural Causal Abstractions [63.21695740637627]
  We develop a new family of causal abstractions by clustering variables and their domains. We show that such abstractions are learnable in practical settings through Neural Causal Models. Our experiments support the theory and illustrate how to scale causal inferences to high-dimensional settings involving image data.
  arXiv  Detail & Related papers  (2024-01-05T02:00:27Z)
• Machine Psychology [54.287802134327485]
  We argue that a fruitful direction for research is engaging large language models in behavioral experiments inspired by psychology. We highlight theoretical perspectives, experimental paradigms, and computational analysis techniques that this approach brings to the table. It paves the way for a "machine psychology" for generative artificial intelligence (AI) that goes beyond performance benchmarks.
  arXiv  Detail & Related papers  (2023-03-24T13:24:41Z)
• Intrinsic Physical Concepts Discovery with Object-Centric Predictive Models [86.25460882547581]
  We introduce the PHYsical Concepts Inference NEtwork (PHYCINE), a system that infers physical concepts in different abstract levels without supervision. We show that object representations containing the discovered physical concepts variables could help achieve better performance in causal reasoning tasks.
  arXiv  Detail & Related papers  (2023-03-03T11:52:21Z)
• Envisioning Future Deep Learning Theories: Some Basic Concepts and Characteristics [30.365274034429508]
  We argue that a future deep learning theory should inherit three characteristics: a textitarchhierically structured network architecture, parameters textititeratively optimized using gradient-based methods, and information from the data that evolves textitcompressively We integrate these characteristics into a graphical model called textitneurashed, which effectively explains some common empirical patterns in deep learning.
  arXiv  Detail & Related papers  (2021-12-17T19:51:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.