Applying statistical learning theory to deep learning
- URL: http://arxiv.org/abs/2311.15404v2
- Date: Mon, 25 Mar 2024 22:55:43 GMT
- Title: Applying statistical learning theory to deep learning
- Authors: Cédric Gerbelot, Avetik Karagulyan, Stefani Karp, Kavya Ravichandran, Menachem Stern, Nathan Srebro,
- Abstract summary: The goal of these lectures is to provide an overview of some of the main questions that arise when attempting to understand deep learning.
We discuss implicit bias in the context of benign overfitting.
We provide a detailed study of the implicit bias of gradient descent on linear diagonal networks for various regression tasks.
- Score: 21.24637996678039
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Although statistical learning theory provides a robust framework to understand supervised learning, many theoretical aspects of deep learning remain unclear, in particular how different architectures may lead to inductive bias when trained using gradient based methods. The goal of these lectures is to provide an overview of some of the main questions that arise when attempting to understand deep learning from a learning theory perspective. After a brief reminder on statistical learning theory and stochastic optimization, we discuss implicit bias in the context of benign overfitting. We then move to a general description of the mirror descent algorithm, showing how we may go back and forth between a parameter space and the corresponding function space for a given learning problem, as well as how the geometry of the learning problem may be represented by a metric tensor. Building on this framework, we provide a detailed study of the implicit bias of gradient descent on linear diagonal networks for various regression tasks, showing how the loss function, scale of parameters at initialization and depth of the network may lead to various forms of implicit bias, in particular transitioning between kernel or feature learning.
Related papers
- Geometric Understanding of Discriminability and Transferability for Visual Domain Adaptation [27.326817457760725]
Invariant representation learning for unsupervised domain adaptation (UDA) has made significant advances in computer vision and pattern recognition communities.
Recently, empirical connections between transferability and discriminability have received increasing attention.
In this work, we systematically analyze the essentials of transferability and discriminability from the geometric perspective.
arXiv Detail & Related papers (2024-06-24T13:31:08Z) - Rethinking Dimensional Rationale in Graph Contrastive Learning from Causal Perspective [15.162584339143239]
Graph contrastive learning is a general learning paradigm excelling at capturing invariant information from diverse perturbations in graphs.
Recent works focus on exploring the structural rationale from graphs, thereby increasing the discriminability of the invariant information.
We propose the dimensional rationale-aware graph contrastive learning approach, which introduces a learnable dimensional rationale acquiring network and a redundancy reduction constraint.
arXiv Detail & Related papers (2023-12-16T10:05:18Z) - Learned Regularization for Inverse Problems: Insights from a Spectral Model [1.4963011898406866]
This chapter provides a theoretically founded investigation of state-of-the-art learning approaches for inverse problems.
We give an extended definition of regularization methods and their convergence in terms of the underlying data distributions.
arXiv Detail & Related papers (2023-12-15T14:50:14Z) - On the Joint Interaction of Models, Data, and Features [82.60073661644435]
We introduce a new tool, the interaction tensor, for empirically analyzing the interaction between data and model through features.
Based on these observations, we propose a conceptual framework for feature learning.
Under this framework, the expected accuracy for a single hypothesis and agreement for a pair of hypotheses can both be derived in closed-form.
arXiv Detail & Related papers (2023-06-07T21:35:26Z) - On the Benefits of Large Learning Rates for Kernel Methods [110.03020563291788]
We show that a phenomenon can be precisely characterized in the context of kernel methods.
We consider the minimization of a quadratic objective in a separable Hilbert space, and show that with early stopping, the choice of learning rate influences the spectral decomposition of the obtained solution.
arXiv Detail & Related papers (2022-02-28T13:01:04Z) - A neural anisotropic view of underspecification in deep learning [60.119023683371736]
We show that the way neural networks handle the underspecification of problems is highly dependent on the data representation.
Our results highlight that understanding the architectural inductive bias in deep learning is fundamental to address the fairness, robustness, and generalization of these systems.
arXiv Detail & Related papers (2021-04-29T14:31:09Z) - Nonparametric Estimation of Heterogeneous Treatment Effects: From Theory
to Learning Algorithms [91.3755431537592]
We analyze four broad meta-learning strategies which rely on plug-in estimation and pseudo-outcome regression.
We highlight how this theoretical reasoning can be used to guide principled algorithm design and translate our analyses into practice.
arXiv Detail & Related papers (2021-01-26T17:11:40Z) - Learning What Makes a Difference from Counterfactual Examples and
Gradient Supervision [57.14468881854616]
We propose an auxiliary training objective that improves the generalization capabilities of neural networks.
We use pairs of minimally-different examples with different labels, a.k.a counterfactual or contrasting examples, which provide a signal indicative of the underlying causal structure of the task.
Models trained with this technique demonstrate improved performance on out-of-distribution test sets.
arXiv Detail & Related papers (2020-04-20T02:47:49Z) - Disentangling Adaptive Gradient Methods from Learning Rates [65.0397050979662]
We take a deeper look at how adaptive gradient methods interact with the learning rate schedule.
We introduce a "grafting" experiment which decouples an update's magnitude from its direction.
We present some empirical and theoretical retrospectives on the generalization of adaptive gradient methods.
arXiv Detail & Related papers (2020-02-26T21:42:49Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.