GenEFT: Understanding Statics and Dynamics of Model Generalization via Effective Theory

• URL: http://arxiv.org/abs/2402.05916v2
• Date: Thu, 20 Mar 2025 16:31:41 GMT
• Title: GenEFT: Understanding Statics and Dynamics of Model Generalization via Effective Theory
• Abstract summary: We present an effective theory framework for shedding light on the statics and dynamics of neural network generalization.<n>We find generalization in a Goldilocks zone where the decoder is neither too weak nor too powerful.<n>We then introduce an effective theory for the dynamics of representation learning, where latent-space representations are modeled as interacting particles (repons)
• Score: 15.01642959193149
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present GenEFT: an effective theory framework for shedding light on the statics and dynamics of neural network generalization, and illustrate it with graph learning examples. We first investigate the generalization phase transition as data size increases, comparing experimental results with information-theory-based approximations. We find generalization in a Goldilocks zone where the decoder is neither too weak nor too powerful. We then introduce an effective theory for the dynamics of representation learning, where latent-space representations are modeled as interacting particles (repons), and find that it explains our experimentally observed phase transition between generalization and overfitting as encoder and decoder learning rates are scanned. This highlights the power of physics-inspired effective theories for bridging the gap between theoretical predictions and practice in machine learning.

Related papers

• A Theoretical Analysis of Generalization Dynamics in Neural Networks under Gradient Descent with Weight Decay [32.27793372712302]
  We study a class of neural networks trained under the $ell2$ loss by gradient descent (GD) with weight decay.<n>We decompose the population error into data error, optimization error, and prediction variation error, and bound them separately.<n>For the prediction variation error, we propose (local) approximate homogeneity and derive explicit cellwise and layerwise bounds.
  arXiv  Detail & Related papers  (2026-09-07T16:59:16Z)
• Information Spreading in Diffusion Models from Effective Field Theory [0.0]
  We study score-matching diffusion models with a convolutional architecture.<n>We argue that the machinery of effective field theory from physics can be usefully applied to describe the denoising dynamics.
  arXiv  Detail & Related papers  (2026-08-14T13:45:31Z)
• Generalization Performance of Hypergraph Neural Networks [21.483543928698676]
  We develop margin-based generalization bounds for four representative classes of hypergraph neural networks. Our results reveal the manner in which hypergraph structure and spectral norms of the learned weights can affect the generalization bounds. Our empirical study examines the relationship between the practical performance and theoretical bounds of the models over synthetic and real-world datasets.
  arXiv  Detail & Related papers  (2025-01-22T00:20:26Z)
• An Effective Theory of Bias Amplification [18.648588509429167]
  Machine learning models may capture and amplify biases present in data, leading to disparate test performance across social groups. We propose a precise analytical theory in the context of ridge regression, where the former models neural networks in a simplified regime. Our theory offers a unified and rigorous explanation of machine learning bias, providing insights into phenomena such as bias amplification and minority-group bias.
  arXiv  Detail & Related papers  (2024-10-07T08:43:22Z)
• Foundations and Frontiers of Graph Learning Theory [81.39078977407719]
  Recent advancements in graph learning have revolutionized the way to understand and analyze data with complex structures. Graph Neural Networks (GNNs), i.e. neural network architectures designed for learning graph representations, have become a popular paradigm. This article provides a comprehensive summary of the theoretical foundations and breakthroughs concerning the approximation and learning behaviors intrinsic to prevalent graph learning models.
  arXiv  Detail & Related papers  (2024-07-03T14:07:41Z)
• What Improves the Generalization of Graph Transformers? A Theoretical Dive into the Self-attention and Positional Encoding [67.59552859593985]
  Graph Transformers, which incorporate self-attention and positional encoding, have emerged as a powerful architecture for various graph learning tasks. This paper introduces first theoretical investigation of a shallow Graph Transformer for semi-supervised classification.
  arXiv  Detail & Related papers  (2024-06-04T05:30:16Z)
• Injecting Hamiltonian Architectural Bias into Deep Graph Networks for Long-Range Propagation [55.227976642410766]
  dynamics of information diffusion within graphs is a critical open issue that heavily influences graph representation learning. Motivated by this, we introduce (port-)Hamiltonian Deep Graph Networks. We reconcile under a single theoretical and practical framework both non-dissipative long-range propagation and non-conservative behaviors.
  arXiv  Detail & Related papers  (2024-05-27T13:36:50Z)
• Towards Understanding the Generalization of Graph Neural Networks [9.217947432437546]
  Graph neural networks (GNNs) are the most widely adopted model in graph-structured data oriented learning and representation. We first establish high probability bounds of generalization gap and gradients in transductive learning. The theoretical results reveal the architecture specific factors affecting the generalization gap.
  arXiv  Detail & Related papers  (2023-05-14T03:05:14Z)
• DIFFormer: Scalable (Graph) Transformers Induced by Energy Constrained Diffusion [66.21290235237808]
  We introduce an energy constrained diffusion model which encodes a batch of instances from a dataset into evolutionary states. We provide rigorous theory that implies closed-form optimal estimates for the pairwise diffusion strength among arbitrary instance pairs. Experiments highlight the wide applicability of our model as a general-purpose encoder backbone with superior performance in various tasks.
  arXiv  Detail & Related papers  (2023-01-23T15:18:54Z)
• Beyond spectral gap (extended): The role of the topology in decentralized learning [58.48291921602417]
  In data-parallel optimization of machine learning models, workers collaborate to improve their estimates of the model. Current theory does not explain that collaboration enables larger learning rates than training alone. This paper aims to paint an accurate picture of sparsely-connected distributed optimization.
  arXiv  Detail & Related papers  (2023-01-05T16:53:38Z)
• A Theoretical Study of Inductive Biases in Contrastive Learning [32.98250585760665]
  We provide the first theoretical analysis of self-supervised learning that incorporates the effect of inductive biases originating from the model class. We show that when the model has limited capacity, contrastive representations would recover certain special clustering structures that are compatible with the model architecture.
  arXiv  Detail & Related papers  (2022-11-27T01:53:29Z)
• A Scaling Law for Synthetic-to-Real Transfer: A Measure of Pre-Training [52.93808218720784]
  Synthetic-to-real transfer learning is a framework in which we pre-train models with synthetically generated images and ground-truth annotations for real tasks. Although synthetic images overcome the data scarcity issue, it remains unclear how the fine-tuning performance scales with pre-trained models. We observe a simple and general scaling law that consistently describes learning curves in various tasks, models, and complexities of synthesized pre-training data.
  arXiv  Detail & Related papers  (2021-08-25T02:29:28Z)
• Learning While Dissipating Information: Understanding the Generalization Capability of SGLD [9.328633662865682]
  We derive an algorithm-dependent generalization bound by analyzing gradient Langevin dynamics (SGLD) Our analysis reveals an intricate trade-off between learning and information dissipation.
  arXiv  Detail & Related papers  (2021-02-05T03:18:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.