A Survey Analyzing Generalization in Deep Reinforcement Learning

• URL: http://arxiv.org/abs/2401.02349v2
• Date: Wed, 30 Oct 2024 16:18:57 GMT
• Title: A Survey Analyzing Generalization in Deep Reinforcement Learning
• Authors: Ezgi Korkmaz,
• Abstract summary: We will formalize and analyze generalization in deep reinforcement learning. We will explain the fundamental reasons why deep reinforcement learning policies encounter overfitting problems that limit their generalization capabilities.
• Score: 14.141453107129403
• License:
• Abstract: Reinforcement learning research obtained significant success and attention with the utilization of deep neural networks to solve problems in high dimensional state or action spaces. While deep reinforcement learning policies are currently being deployed in many different fields from medical applications to large language models, there are still ongoing questions the field is trying to answer on the generalization capabilities of deep reinforcement learning policies. In this paper, we will formalize and analyze generalization in deep reinforcement learning. We will explain the fundamental reasons why deep reinforcement learning policies encounter overfitting problems that limit their generalization capabilities. Furthermore, we will categorize and explain the manifold solution approaches to increase generalization, and overcome overfitting in deep reinforcement learning policies. From exploration to adversarial analysis and from regularization to robustness our paper provides an analysis on a wide range of subfields within deep reinforcement learning with a broad scope and in-depth view. We believe our study can provide a compact guideline for the current advancements in deep reinforcement learning, and help to construct robust deep neural policies with higher generalization skills.

Related papers

• Causal Reinforcement Learning: A Survey [57.368108154871]
  Reinforcement learning is an essential paradigm for solving sequential decision problems under uncertainty. One of the main obstacles is that reinforcement learning agents lack a fundamental understanding of the world. Causality offers a notable advantage as it can formalize knowledge in a systematic manner.
  arXiv  Detail & Related papers  (2023-07-04T03:00:43Z)
• A Comprehensive Survey of Continual Learning: Theory, Method and Application [64.23253420555989]
  We present a comprehensive survey of continual learning, seeking to bridge the basic settings, theoretical foundations, representative methods, and practical applications. We summarize the general objectives of continual learning as ensuring a proper stability-plasticity trade-off and an adequate intra/inter-task generalizability in the context of resource efficiency.
  arXiv  Detail & Related papers  (2023-01-31T11:34:56Z)
• Adversarial Robust Deep Reinforcement Learning Requires Redefining Robustness [7.6146285961466]
  We show that high sensitivity directions are more abundant in the deep neural policy landscape and can be found via more natural means in a black-box setting. We show that vanilla training techniques intriguingly result in learning more robust policies compared to the policies learnt via the state-of-the-art adversarial training techniques.
  arXiv  Detail & Related papers  (2023-01-17T16:54:33Z)
• Understanding the Evolution of Linear Regions in Deep Reinforcement Learning [21.53394095184201]
  We study how observed region counts and their densities evolve during deep reinforcement learning. We find that the region density increases only moderately throughout training, as measured along fixed trajectories coming from the final policy. Our findings suggest that the complexity of deep reinforcement learning policies does not principally emerge from a significant growth in the complexity of functions observed on-and-around trajectories of the policy.
  arXiv  Detail & Related papers  (2022-10-24T21:22:12Z)
• Learning Dynamics and Generalization in Reinforcement Learning [59.530058000689884]
  We show theoretically that temporal difference learning encourages agents to fit non-smooth components of the value function early in training. We show that neural networks trained using temporal difference algorithms on dense reward tasks exhibit weaker generalization between states than randomly networks and gradient networks trained with policy methods.
  arXiv  Detail & Related papers  (2022-06-05T08:49:16Z)
• Embracing the Dark Knowledge: Domain Generalization Using Regularized Knowledge Distillation [65.79387438988554]
  Lack of generalization capability in the absence of sufficient and representative data is one of the challenges that hinder their practical application. We propose a simple, effective, and plug-and-play training strategy named Knowledge Distillation for Domain Generalization (KDDG) We find that both the richer dark knowledge" from the teacher network, as well as the gradient filter we proposed, can reduce the difficulty of learning the mapping.
  arXiv  Detail & Related papers  (2021-07-06T14:08:54Z)
• The Modern Mathematics of Deep Learning [8.939008609565368]
  We describe the new field of mathematical analysis of deep learning. This field emerged around a list of research questions that were not answered within the classical of learning theory. For selected approaches, we describe the main ideas in more detail.
  arXiv  Detail & Related papers  (2021-05-09T21:30:42Z)
• NeurIPS 2020 Competition: Predicting Generalization in Deep Learning [0.0]
  Understanding generalization in deep learning is arguably one of the most important questions in deep learning. We invite the community to propose complexity measures that can accurately predict generalization of models.
  arXiv  Detail & Related papers  (2020-12-14T22:21:37Z)
• Optimism in the Face of Adversity: Understanding and Improving Deep Learning through Adversarial Robustness [63.627760598441796]
  We provide an in-depth review of the field of adversarial robustness in deep learning. We highlight the intuitive connection between adversarial examples and the geometry of deep neural networks. We provide an overview of the main emerging applications of adversarial robustness beyond security.
  arXiv  Detail & Related papers  (2020-10-19T16:03:46Z)
• Generalization in Deep Learning [103.91623583928852]
  This paper provides theoretical insights into why and how deep learning can generalize well, despite its large capacity, complexity, possible algorithmic instability, nonrobustness, and sharp minima. We also discuss approaches to provide non-vacuous generalization guarantees for deep learning.
  arXiv  Detail & Related papers  (2017-10-16T02:21:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.