Common Challenges of Deep Reinforcement Learning Applications Development: An Empirical Study

• URL: http://arxiv.org/abs/2310.09575v3
• Date: Sun, 19 May 2024 00:20:48 GMT
• Title: Common Challenges of Deep Reinforcement Learning Applications Development: An Empirical Study
• Authors: Mohammad Mehdi Morovati, Florian Tambon, Mina Taraghi, Amin Nikanjam, Foutse Khomh,
• Abstract summary: Deep Reinforcement Learning (DRL) is a subdomain of Machine Learning (ML) used to produce intelligent agents. Despite recent developments in DRL technology, the main challenges that developers face in the development of DRL applications are still unknown.
• Score: 10.705440907493452
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Machine Learning (ML) is increasingly being adopted in different industries. Deep Reinforcement Learning (DRL) is a subdomain of ML used to produce intelligent agents. Despite recent developments in DRL technology, the main challenges that developers face in the development of DRL applications are still unknown. To fill this gap, in this paper, we conduct a large-scale empirical study of 927 DRL-related posts extracted from Stack Overflow, the most popular Q&A platform in the software community. Through the process of labeling and categorizing extracted posts, we created a taxonomy of common challenges encountered in the development of DRL applications, along with their corresponding popularity levels. This taxonomy has been validated through a survey involving 65 DRL developers. Results show that at least 45% of developers experienced 18 of the 21 challenges identified in the taxonomy. The most frequent source of difficulty during the development of DRL applications are Comprehension, API usage, and Design problems, while Parallel processing, and DRL libraries/frameworks are classified as the most difficult challenges to address, with respect to the time required to receive an accepted answer. We hope that the research community will leverage this taxonomy to develop efficient strategies to address the identified challenges and improve the quality of DRL applications.

Related papers

• What's Wrong with Your Code Generated by Large Language Models? An Extensive Study [80.18342600996601]
  Large language models (LLMs) produce code that is shorter yet more complicated as compared to canonical solutions. We develop a taxonomy of bugs for incorrect codes that includes three categories and 12 sub-categories, and analyze the root cause for common bug types. We propose a novel training-free iterative method that introduces self-critique, enabling LLMs to critique and correct their generated code based on bug types and compiler feedback.
  arXiv  Detail & Related papers  (2024-07-08T17:27:17Z)
• Generative AI for Deep Reinforcement Learning: Framework, Analysis, and Use Cases [60.30995339585003]
  Deep reinforcement learning (DRL) has been widely applied across various fields and has achieved remarkable accomplishments. DRL faces certain limitations, including low sample efficiency and poor generalization. We present how to leverage generative AI (GAI) to address these issues and enhance the performance of DRL algorithms.
  arXiv  Detail & Related papers  (2024-05-31T01:25:40Z)
• Deploying Deep Reinforcement Learning Systems: A Taxonomy of Challenges [13.39623605590729]
  We propose an empirical study on Stack Overflow (SO), the most popular Q&A forum for developers, to uncover and understand the challenges practitioners faced when deploying DRL systems. After filtering and manual analysis, we examined 357 SO posts about DRL deployment, investigated the current state, and identified the challenges related to deploying DRL systems. Results show that the general interest in DRL deployment is growing, confirming the study's relevance and importance.
  arXiv  Detail & Related papers  (2023-08-23T21:44:09Z)
• Prevalence of Code Smells in Reinforcement Learning Projects [1.7218973692320518]
  Reinforcement Learning (RL) is being increasingly used to learn and adapt application behavior in many domains, including large-scale and safety critical systems. With the advent of plug-n-play RL libraries, its applicability has further increased, enabling integration of RL algorithms by users. We note, however, that the majority of such code is not developed by RL engineers, which as a consequence, may lead to poor program quality yielding bugs, suboptimal performance, maintainability, and evolution problems for RL-based projects.
  arXiv  Detail & Related papers  (2023-03-17T20:25:13Z)
• A Survey of Meta-Reinforcement Learning [69.76165430793571]
  We cast the development of better RL algorithms as a machine learning problem itself in a process called meta-RL. We discuss how, at a high level, meta-RL research can be clustered based on the presence of a task distribution and the learning budget available for each individual task. We conclude by presenting the open problems on the path to making meta-RL part of the standard toolbox for a deep RL practitioner.
  arXiv  Detail & Related papers  (2023-01-19T12:01:41Z)
• Automated Reinforcement Learning (AutoRL): A Survey and Open Problems [92.73407630874841]
  Automated Reinforcement Learning (AutoRL) involves not only standard applications of AutoML but also includes additional challenges unique to RL. We provide a common taxonomy, discuss each area in detail and pose open problems which would be of interest to researchers going forward.
  arXiv  Detail & Related papers  (2022-01-11T12:41:43Z)
• Towards Standardizing Reinforcement Learning Approaches for Stochastic Production Scheduling [77.34726150561087]
  reinforcement learning can be used to solve scheduling problems. Existing studies rely on (sometimes) complex simulations for which the code is unavailable. There is a vast array of RL designs to choose from. standardization of model descriptions - both production setup and RL design - and validation scheme are a prerequisite.
  arXiv  Detail & Related papers  (2021-04-16T16:07:10Z)
• Faults in Deep Reinforcement Learning Programs: A Taxonomy and A Detection Approach [13.57291726431012]
  Deep Reinforcement Learning (DRL) is the application of Deep Learning in the domain of Reinforcement Learning (RL) In this paper, we present the first attempt to categorize faults occurring in DRL programs. We have defined a meta-model of DRL programs and developed DRLinter, a model-based fault detection approach.
  arXiv  Detail & Related papers  (2021-01-01T01:49:03Z)
• Deep Reinforcement Learning and Transportation Research: A Comprehensive Review [0.0]
  We offer an overview of the DRL mathematical background, popular and promising DRL algorithms, and some highly effective DRL extensions. Building on this review, we examine the applicability, strengths, shortcomings, and common and application-specific issues of DRL techniques with regard to their applications in transportation.
  arXiv  Detail & Related papers  (2020-10-13T05:23:11Z)
• What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study [50.79125250286453]
  On-policy reinforcement learning (RL) has been successfully applied to many different continuous control tasks. But state-of-the-art implementations take numerous low- and high-level design decisions that strongly affect the performance of the resulting agents. These choices are usually not extensively discussed in the literature, leading to discrepancy between published descriptions of algorithms and their implementations. We implement >50 such choices'' in a unified on-policy RL framework, allowing us to investigate their impact in a large-scale empirical study.
  arXiv  Detail & Related papers  (2020-06-10T17:59:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.