Predicting Generalization in Deep Learning via Local Measures of Distortion

• URL: http://arxiv.org/abs/2012.06969v2
• Date: Wed, 16 Dec 2020 02:22:46 GMT
• Title: Predicting Generalization in Deep Learning via Local Measures of Distortion
• Authors: Abhejit Rajagopal, Vamshi C. Madala, Shivkumar Chandrasekaran, Peder E. Z. Larson
• Abstract summary: We study generalization in deep learning by appealing to complexity measures originally developed in approximation and information theory. We show that simple vector quantization approaches such as PCA, GMMs, and SVMs capture their spirit when applied layer-wise to deep extracted features.
• Score: 7.806155368334511
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We study generalization in deep learning by appealing to complexity measures originally developed in approximation and information theory. While these concepts are challenged by the high-dimensional and data-defined nature of deep learning, we show that simple vector quantization approaches such as PCA, GMMs, and SVMs capture their spirit when applied layer-wise to deep extracted features giving rise to relatively inexpensive complexity measures that correlate well with generalization performance. We discuss our results in 2020 NeurIPS PGDL challenge.

Related papers

• Topological Generalization Bounds for Discrete-Time Stochastic Optimization Algorithms [15.473123662393169]
  Deep neural networks (DNNs) show remarkable generalization properties. The source of these capabilities remains elusive, defying the established statistical learning theory. Recent studies have revealed that properties of training trajectories can be indicative of generalization.
  arXiv  Detail & Related papers  (2024-07-11T17:56:03Z)
• On the Importance of Exploration for Generalization in Reinforcement Learning [89.63074327328765]
  We propose EDE: Exploration via Distributional Ensemble, a method that encourages exploration of states with high uncertainty. Our algorithm is the first value-based approach to achieve state-of-the-art on both Procgen and Crafter.
  arXiv  Detail & Related papers  (2023-06-08T18:07:02Z)
• GEC: A Unified Framework for Interactive Decision Making in MDP, POMDP, and Beyond [101.5329678997916]
  We study sample efficient reinforcement learning (RL) under the general framework of interactive decision making. We propose a novel complexity measure, generalized eluder coefficient (GEC), which characterizes the fundamental tradeoff between exploration and exploitation. We show that RL problems with low GEC form a remarkably rich class, which subsumes low Bellman eluder dimension problems, bilinear class, low witness rank problems, PO-bilinear class, and generalized regular PSR.
  arXiv  Detail & Related papers  (2022-11-03T16:42:40Z)
• Provable Generalization of Overparameterized Meta-learning Trained with SGD [62.892930625034374]
  We study the generalization of a widely used meta-learning approach, Model-Agnostic Meta-Learning (MAML) We provide both upper and lower bounds for the excess risk of MAML, which captures how SGD dynamics affect these generalization bounds. Our theoretical findings are further validated by experiments.
  arXiv  Detail & Related papers  (2022-06-18T07:22:57Z)
• Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observability [92.95794652625496]
  Generalization is a central challenge for the deployment of reinforcement learning systems. We show that generalization to unseen test conditions from a limited number of training conditions induces implicit partial observability. We recast the problem of generalization in RL as solving the induced partially observed Markov decision process.
  arXiv  Detail & Related papers  (2021-07-13T17:59:25Z)
• Evaluation of Complexity Measures for Deep Learning Generalization in Medical Image Analysis [77.34726150561087]
  PAC-Bayes flatness-based and path norm-based measures produce the most consistent explanation for the combination of models and data. We also investigate the use of multi-task classification and segmentation approach for breast images.
  arXiv  Detail & Related papers  (2021-03-04T20:58:22Z)
• Predicting Generalization in Deep Learning via Metric Learning -- PGDL Shared task [0.0]
  The report presents the solution that was submitted by the user emphsmeznar which achieved the eight place in the competition. In the proposed approach, we create simple metrics and find their best combination with automatic testing on the provided dataset, exploring how combinations of various properties of the input neural network architectures can be used for the prediction of their generalization.
  arXiv  Detail & Related papers  (2020-12-16T17:59:13Z)
• NeurIPS 2020 Competition: Predicting Generalization in Deep Learning [0.0]
  Understanding generalization in deep learning is arguably one of the most important questions in deep learning. We invite the community to propose complexity measures that can accurately predict generalization of models.
  arXiv  Detail & Related papers  (2020-12-14T22:21:37Z)
• Representation Based Complexity Measures for Predicting Generalization in Deep Learning [0.0]
  Deep Neural Networks can generalize despite being significantly overparametrized. Recent research has tried to examine this phenomenon from various view points. We provide an interpretation of generalization from the perspective of quality of internal representations.
  arXiv  Detail & Related papers  (2020-12-04T18:53:44Z)
• Disentangling Adaptive Gradient Methods from Learning Rates [65.0397050979662]
  We take a deeper look at how adaptive gradient methods interact with the learning rate schedule. We introduce a "grafting" experiment which decouples an update's magnitude from its direction. We present some empirical and theoretical retrospectives on the generalization of adaptive gradient methods.
  arXiv  Detail & Related papers  (2020-02-26T21:42:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.