An Experience Report on Machine Learning Reproducibility: Guidance for Practitioners and TensorFlow Model Garden Contributors

• URL: http://arxiv.org/abs/2107.00821v1
• Date: Fri, 2 Jul 2021 04:32:18 GMT
• Title: An Experience Report on Machine Learning Reproducibility: Guidance for Practitioners and TensorFlow Model Garden Contributors
• Authors: Vishnu Banna and Akhil Chinnakotla and Zhengxin Yan and Ani Vegesana and Naveen Vivek and Kruthi Krishnappa and Wenxin Jiang and Yung-Hsiang Lu and George K. Thiruvathukal and James C. Davis
• Abstract summary: This report is to define a process for reproducing a state-of-the-art machine learning model at a level of quality suitable for inclusion in the Model Garden. We report on our experiences implementing the YOLO model family with a team of 26 student researchers, share the tools we developed, and describe the lessons we learned along the way.
• Score: 1.177923904173852
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Machine learning techniques are becoming a fundamental tool for scientific and engineering progress. These techniques are applied in contexts as diverse as astronomy and spam filtering. However, correctly applying these techniques requires careful engineering. Much attention has been paid to the technical potential; relatively little attention has been paid to the software engineering process required to bring research-based machine learning techniques into practical utility. Technology companies have supported the engineering community through machine learning frameworks such as TensorFLow and PyTorch, but the details of how to engineer complex machine learning models in these frameworks have remained hidden. To promote best practices within the engineering community, academic institutions and Google have partnered to launch a Special Interest Group on Machine Learning Models (SIGMODELS) whose goal is to develop exemplary implementations of prominent machine learning models in community locations such as the TensorFlow Model Garden (TFMG). The purpose of this report is to define a process for reproducing a state-of-the-art machine learning model at a level of quality suitable for inclusion in the TFMG. We define the engineering process and elaborate on each step, from paper analysis to model release. We report on our experiences implementing the YOLO model family with a team of 26 student researchers, share the tools we developed, and describe the lessons we learned along the way.

Related papers

• Application-Driven Innovation in Machine Learning [56.85396167616353]
  We describe the paradigm of application-driven research in machine learning. We show how this approach can productively synergize with methods-driven work. Despite these benefits, we find that reviewing, hiring, and teaching practices in machine learning often hold back application-driven innovation.
  arXiv  Detail & Related papers  (2024-03-26T04:59:27Z)
• Utilizing Online and Open-Source Machine Learning Toolkits to Leverage the Future of Sustainable Engineering [8.641860292533023]
  Edge Impulse has designed an open-source TinyML-enabled Arduino education tool kit for engineering disciplines. This paper discusses the various applications and approaches engineering educators have taken to utilize ML toolkits in the classroom.
  arXiv  Detail & Related papers  (2023-04-21T17:50:21Z)
• Requirements Engineering for Machine Learning: A Review and Reflection [39.01716712094724]
  This paper aims to provide an overview of the requirements engineering process for machine learning applications. An example case of industrial data-driven intelligence applications is also discussed in relation to the aforementioned steps.
  arXiv  Detail & Related papers  (2022-10-03T12:24:39Z)
• Enabling Automated Machine Learning for Model-Driven AI Engineering [60.09869520679979]
  We propose a novel approach to enable Model-Driven Software Engineering and Model-Driven AI Engineering. In particular, we support Automated ML, thus assisting software engineers without deep AI knowledge in developing AI-intensive systems.
  arXiv  Detail & Related papers  (2022-03-06T10:12:56Z)
• SELM: Software Engineering of Machine Learning Models [0.19116784879310023]
  In this article, we present a SELM framework for Software Engineering of machine Learning Models. Using the SELM framework, we can improve a machine learning process efficiency and provide more accuracy in learning. This issue highlights the importance of an interdisciplinary approach to machine learning.
  arXiv  Detail & Related papers  (2021-03-20T21:43:24Z)
• Designing Machine Learning Toolboxes: Concepts, Principles and Patterns [0.0]
  We provide an overview of key patterns in the design of AI modeling toolboxes. Our analysis can not only explain the design of existing toolboxes, but also guide the development of new ones.
  arXiv  Detail & Related papers  (2021-01-13T08:55:15Z)
• Technology Readiness Levels for Machine Learning Systems [107.56979560568232]
  Development and deployment of machine learning systems can be executed easily with modern tools, but the process is typically rushed and means-to-an-end. We have developed a proven systems engineering approach for machine learning development and deployment. Our "Machine Learning Technology Readiness Levels" framework defines a principled process to ensure robust, reliable, and responsible systems.
  arXiv  Detail & Related papers  (2021-01-11T15:54:48Z)
• Technology Readiness Levels for AI & ML [79.22051549519989]
  Development of machine learning systems can be executed easily with modern tools, but the process is typically rushed and means-to-an-end. Engineering systems follow well-defined processes and testing standards to streamline development for high-quality, reliable results. We propose a proven systems engineering approach for machine learning development and deployment.
  arXiv  Detail & Related papers  (2020-06-21T17:14:34Z)
• Machine Learning for Software Engineering: A Systematic Mapping [73.30245214374027]
  The software development industry is rapidly adopting machine learning for transitioning modern day software systems towards highly intelligent and self-learning systems. No comprehensive study exists that explores the current state-of-the-art on the adoption of machine learning across software engineering life cycle stages. This study introduces a machine learning for software engineering (MLSE) taxonomy classifying the state-of-the-art machine learning techniques according to their applicability to various software engineering life cycle stages.
  arXiv  Detail & Related papers  (2020-05-27T11:56:56Z)
• Towards CRISP-ML(Q): A Machine Learning Process Model with Quality Assurance Methodology [53.063411515511056]
  We propose a process model for the development of machine learning applications. The first phase combines business and data understanding as data availability oftentimes affects the feasibility of the project. The sixth phase covers state-of-the-art approaches for monitoring and maintenance of a machine learning applications.
  arXiv  Detail & Related papers  (2020-03-11T08:25:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.