Statistical Properties of Training & Generalization

• URL: http://arxiv.org/abs/2606.20299v1
• Date: Thu, 18 Jun 2026 14:35:53 GMT
• Title: Statistical Properties of Training & Generalization
• Abstract summary: We investigate the key features and surprises of deep learning from a physics-informed perspective.<n>We review the phenomenon of neural scaling laws and discuss their interplay with the constraints and inductive biases which may be present when applying machine learning to problems in physics.
• Score: 3.3135760457470718
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Deep learning has managed to evade numerous intuitions from classical statistics to achieve unprecedented performance on a number of real-world tasks. In this article, we investigate the key features and surprises of deep learning from a physics-informed perspective, taking care to point out and justify where possible the many choices inherent in constructing a deep learning model. In particular, we review the phenomenon of neural scaling laws and discuss their interplay with the constraints and inductive biases which may be present when applying machine learning to problems in physics.

Related papers

• There Will Be a Scientific Theory of Deep Learning [62.690977616250954]
  We make the case that a scientific theory of deep learning is emerging.<n>We identify five growing bodies of work that point toward such a theory.<n>We argue that the emerging theory is best thought of as a mechanics of the learning process.
  arXiv  Detail & Related papers  (2026-04-23T13:58:12Z)
• Can Data-Driven Dynamics Reveal Hidden Physics? There Is A Need for Interpretable Neural Operators [10.591168773809635]
  We classify neural operators into two types: (1) Spatial domain models that learn on grids and (2) Functional domain models that learn with function bases.<n> Specifically, we provide a way to explain the prediction-making process of neural operators and show that neural operator can learn hidden physical patterns from data.<n>Next, we show that a simple dual-space multi-scale model can achieve SOTA performance and we believe that dual-space multi-spatio-scale models hold significant potential to learn complex physics.
  arXiv  Detail & Related papers  (2025-10-03T02:50:21Z)
• Is the end of Insight in Sight ? [0.0]
  A physics-informed neural network (PINN) trained on a rarefied gas dynamics problem governed by the Boltzmann equation.<n>Despite the system's clear structure and well-understood governing laws, the trained network's weights resemble Gaussian-distributed random matrices.<n>This suggests that deep learning and traditional simulation may follow distinct cognitive paths to the same outcome.
  arXiv  Detail & Related papers  (2025-05-07T19:57:36Z)
• Double descent in quantum kernel methods [2.4704291770020617]
  We analytically demonstrate that linear regression models in quantum feature spaces can exhibit double descent behavior.<n>We also confirm the existence of a test error peak, a characteristic feature of double descent.
  arXiv  Detail & Related papers  (2025-01-17T09:49:46Z)
• Deep Learning Through A Telescoping Lens: A Simple Model Provides Empirical Insights On Grokking, Gradient Boosting & Beyond [61.18736646013446]
  In pursuit of a deeper understanding of its surprising behaviors, we investigate the utility of a simple yet accurate model of a trained neural network. Across three case studies, we illustrate how it can be applied to derive new empirical insights on a diverse range of prominent phenomena.
  arXiv  Detail & Related papers  (2024-10-31T22:54:34Z)
• High-dimensional learning of narrow neural networks [1.7094064195431147]
  This manuscript reviews the tools and ideas underlying recent progress in machine learning.<n>We introduce a generic model -- the sequence multi-index model -- which encompasses numerous previously studied models as special instances.<n>We explicate in full detail the analysis of the learning of sequence multi-index models, using statistical physics techniques such as the replica method and approximate message-passing algorithms.
  arXiv  Detail & Related papers  (2024-09-20T21:20:04Z)
• ShadowNet for Data-Centric Quantum System Learning [188.683909185536]
  We propose a data-centric learning paradigm combining the strength of neural-network protocols and classical shadows. Capitalizing on the generalization power of neural networks, this paradigm can be trained offline and excel at predicting previously unseen systems. We present the instantiation of our paradigm in quantum state tomography and direct fidelity estimation tasks and conduct numerical analysis up to 60 qubits.
  arXiv  Detail & Related papers  (2023-08-22T09:11:53Z)
• Brain-inspired Computational Intelligence via Predictive Coding [73.42407863671565]
  Predictive coding (PC) has shown promising properties that make it potentially valuable for the machine learning community.<n>PC-like algorithms are starting to be present in multiple sub-fields of machine learning and AI at large.
  arXiv  Detail & Related papers  (2023-08-15T16:37:16Z)
• Exponential separations between classical and quantum learners [2.209921757303168]
  We discuss how subtle differences in definitions can result in significantly different requirements and tasks for the learner to meet and solve. We present two new learning separations where the classical difficulty primarily lies in identifying the function generating the data.
  arXiv  Detail & Related papers  (2023-06-28T08:55:56Z)
• Privacy-preserving machine learning with tensor networks [37.01494003138908]
  We show that tensor network architectures have especially prospective properties for privacy-preserving machine learning. First, we describe a new privacy vulnerability that is present in feedforward neural networks, illustrating it in synthetic and real-world datasets. We rigorously prove that such conditions are satisfied by tensor-network architectures.
  arXiv  Detail & Related papers  (2022-02-24T19:04:35Z)
• Deep Active Learning by Leveraging Training Dynamics [57.95155565319465]
  We propose a theory-driven deep active learning method (dynamicAL) which selects samples to maximize training dynamics. We show that dynamicAL not only outperforms other baselines consistently but also scales well on large deep learning models.
  arXiv  Detail & Related papers  (2021-10-16T16:51:05Z)
• Quantum Continual Learning Overcoming Catastrophic Forgetting [1.1344265020822928]
  Catastrophic forgetting describes the fact that machine learning models will likely forget the knowledge of previously learned tasks after the learning process of a new one. We find that, similar to those classical learning models based on neural networks, quantum learning systems likewise suffer from such forgetting problem in classification tasks emerging from various application scenes.
  arXiv  Detail & Related papers  (2021-08-05T18:00:03Z)
• A neural anisotropic view of underspecification in deep learning [60.119023683371736]
  We show that the way neural networks handle the underspecification of problems is highly dependent on the data representation. Our results highlight that understanding the architectural inductive bias in deep learning is fundamental to address the fairness, robustness, and generalization of these systems.
  arXiv  Detail & Related papers  (2021-04-29T14:31:09Z)
• Plausible Counterfactuals: Auditing Deep Learning Classifiers with Realistic Adversarial Examples [84.8370546614042]
  Black-box nature of Deep Learning models has posed unanswered questions about what they learn from data. Generative Adversarial Network (GAN) and multi-objectives are used to furnish a plausible attack to the audited model. Its utility is showcased within a human face classification task, unveiling the enormous potential of the proposed framework.
  arXiv  Detail & Related papers  (2020-03-25T11:08:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.