On the Interaction between Software Engineers and Data Scientists when building Machine Learning-Enabled Systems

• URL: http://arxiv.org/abs/2402.05334v1
• Date: Thu, 8 Feb 2024 00:27:56 GMT
• Title: On the Interaction between Software Engineers and Data Scientists when building Machine Learning-Enabled Systems
• Authors: Gabriel Busquim, Hugo Villamizar, Maria Julia Lima, Marcos Kalinowski
• Abstract summary: Machine Learning (ML) components have been increasingly integrated into the core systems of organizations. One of the key challenges is the effective interaction between actors with different backgrounds who need to work closely together. This paper presents an exploratory case study to understand the current interaction and collaboration dynamics between these roles in ML projects.
• Score: 1.2184324428571227
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In recent years, Machine Learning (ML) components have been increasingly integrated into the core systems of organizations. Engineering such systems presents various challenges from both a theoretical and practical perspective. One of the key challenges is the effective interaction between actors with different backgrounds who need to work closely together, such as software engineers and data scientists. This paper presents an exploratory case study to understand the current interaction and collaboration dynamics between these roles in ML projects. We conducted semi-structured interviews with four practitioners with experience in software engineering and data science of a large ML-enabled system project and analyzed the data using reflexive thematic analysis. Our findings reveal several challenges that can hinder collaboration between software engineers and data scientists, including differences in technical expertise, unclear definitions of each role's duties, and the lack of documents that support the specification of the ML-enabled system. We also indicate potential solutions to address these challenges, such as fostering a collaborative culture, encouraging team communication, and producing concise system documentation. This study contributes to understanding the complex dynamics between software engineers and data scientists in ML projects and provides insights for improving collaboration and communication in this context. We encourage future studies investigating this interaction in other projects.

Related papers

• Towards Effective Collaboration between Software Engineers and Data Scientists developing Machine Learning-Enabled Systems [1.1153433121962064]
  Development of Machine Learning (ML)-enabled systems encompasses several social and technical challenges. This paper has the objective of understanding how to enhance the collaboration between two key actors in building these systems: software engineers and data scientists. Our research has found that collaboration between these actors is important for effectively developing ML-enabled systems.
  arXiv  Detail & Related papers  (2024-07-22T17:35:18Z)
• Retrieval-Enhanced Machine Learning: Synthesis and Opportunities [60.34182805429511]
  Retrieval-enhancement can be extended to a broader spectrum of machine learning (ML) This work introduces a formal framework of this paradigm, Retrieval-Enhanced Machine Learning (REML), by synthesizing the literature in various domains in ML with consistent notations which is missing from the current literature. The goal of this work is to equip researchers across various disciplines with a comprehensive, formally structured framework of retrieval-enhanced models, thereby fostering interdisciplinary future research.
  arXiv  Detail & Related papers  (2024-07-17T20:01:21Z)
• Vision+X: A Survey on Multimodal Learning in the Light of Data [64.03266872103835]
  multimodal machine learning that incorporates data from various sources has become an increasingly popular research area. We analyze the commonness and uniqueness of each data format mainly ranging from vision, audio, text, and motions. We investigate the existing literature on multimodal learning from both the representation learning and downstream application levels.
  arXiv  Detail & Related papers  (2022-10-05T13:14:57Z)
• Foundations and Recent Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions [68.6358773622615]
  This paper provides an overview of the computational and theoretical foundations of multimodal machine learning. We propose a taxonomy of 6 core technical challenges: representation, alignment, reasoning, generation, transference, and quantification. Recent technical achievements will be presented through the lens of this taxonomy, allowing researchers to understand the similarities and differences across new approaches.
  arXiv  Detail & Related papers  (2022-09-07T19:21:19Z)
• Assessing the Quality of Computational Notebooks for a Frictionless Transition from Exploration to Production [1.332560004325655]
  Data scientists must transition from the explorative phase of Machine Learning projects to their production phase. To narrow the gap between these two phases, tools and practices adopted by data scientists might be improved by incorporating consolidated software engineering solutions. In my research project, I study the best practices for collaboration with computational notebooks and propose proof-of-concept tools to foster guidelines compliance.
  arXiv  Detail & Related papers  (2022-05-24T10:13:38Z)
• More Engineering, No Silos: Rethinking Processes and Interfaces in Collaboration between Interdisciplinary Teams for Machine Learning Projects [4.482886054198202]
  We identify key collaboration challenges that teams face when building and deploying machine learning systems into production. We report on common collaboration points in the development of production ML systems for requirements, data, and integration, as well as corresponding team patterns and challenges.
  arXiv  Detail & Related papers  (2021-10-19T20:03:20Z)
• Human-Robot Collaboration and Machine Learning: A Systematic Review of Recent Research [69.48907856390834]
  Human-robot collaboration (HRC) is the approach that explores the interaction between a human and a robot. This paper proposes a thorough literature review of the use of machine learning techniques in the context of HRC.
  arXiv  Detail & Related papers  (2021-10-14T15:14:33Z)
• Artificial Intelligence for IT Operations (AIOPS) Workshop White Paper [50.25428141435537]
  Artificial Intelligence for IT Operations (AIOps) is an emerging interdisciplinary field arising in the intersection between machine learning, big data, streaming analytics, and the management of IT operations. Main aim of the AIOPS workshop is to bring together researchers from both academia and industry to present their experiences, results, and work in progress in this field.
  arXiv  Detail & Related papers  (2021-01-15T10:43:10Z)
• Technology Readiness Levels for Machine Learning Systems [107.56979560568232]
  Development and deployment of machine learning systems can be executed easily with modern tools, but the process is typically rushed and means-to-an-end. We have developed a proven systems engineering approach for machine learning development and deployment. Our "Machine Learning Technology Readiness Levels" framework defines a principled process to ensure robust, reliable, and responsible systems.
  arXiv  Detail & Related papers  (2021-01-11T15:54:48Z)
• Enabling collaborative data science development with the Ballet framework [9.424574945499844]
  We present a novel conceptual framework and ML programming model to address challenges to scaling data science collaborations. We instantiate these ideas in Ballet, a lightweight software framework for collaborative open-source data science.
  arXiv  Detail & Related papers  (2020-12-14T18:51:23Z)
• Interactive Machine Learning of Musical Gesture [1.370633147306388]
  This chapter presents an overview of Interactive Machine Learning (IML) techniques applied to the analysis and design of musical gestures. We discuss how different algorithms may be used to accomplish different tasks, including interacting with complex synthesis techniques. We conclude the chapter with a description of how some of these techniques were employed by the authors for the development of four musical pieces, thus outlining the implications that IML have for musical practice.
  arXiv  Detail & Related papers  (2020-11-26T22:44:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.