Industrial Practices of Requirements Engineering for ML-Enabled Systems in Brazil

• URL: http://arxiv.org/abs/2407.18977v1
• Date: Mon, 22 Jul 2024 17:25:27 GMT
• Title: Industrial Practices of Requirements Engineering for ML-Enabled Systems in Brazil
• Authors: Antonio Pedro Santos Alves, Marcos Kalinowski, Daniel Mendez, Hugo Villamizar, Kelly Azevedo, Tatiana Escovedo, Helio Lopes,
• Abstract summary: 41% of companies in Brazil use machine learning (ML) to some extent. However, unrealistic customer expectations and vagueness in ML problem specifications have been reported. This study aims to investigate the application of Requirements Engineering in developing ML-enabled systems in Brazil.
• Score: 0.9342717102482063
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: [Context] In Brazil, 41% of companies use machine learning (ML) to some extent. However, several challenges have been reported when engineering ML-enabled systems, including unrealistic customer expectations and vagueness in ML problem specifications. Literature suggests that Requirements Engineering (RE) practices and tools may help to alleviate these issues, yet there is insufficient understanding of RE's practical application and its perception among practitioners. [Goal] This study aims to investigate the application of RE in developing ML-enabled systems in Brazil, creating an overview of current practices, perceptions, and problems in the Brazilian industry. [Method] To this end, we extracted and analyzed data from an international survey focused on ML-enabled systems, concentrating specifically on responses from practitioners based in Brazil. We analyzed RE-related answers gathered from 72 practitioners involved in data-driven projects. We conducted quantitative statistical analyses on contemporary practices using bootstrapping with confidence intervals and qualitative studies on the reported problems involving open and axial coding procedures. [Results] Our findings highlight distinct RE implementation aspects in Brazil's ML projects. For instance, (i) RE-related tasks are predominantly conducted by data scientists; (ii) the most common techniques for eliciting requirements are interviews and workshop meetings; (iii) there is a prevalence of interactive notebooks in requirements documentation; (iv) practitioners report problems that include a poor understanding of the problem to solve and the business domain, low customer engagement, and difficulties managing stakeholders expectations. [Conclusion] These results provide an understanding of RE-related practices in the Brazilian ML industry, helping to guide research toward improving the maturity of RE for ML-enabled systems.

Related papers

• Gaps Between Research and Practice When Measuring Representational Harms Caused by LLM-Based Systems [88.35461485731162]
  We identify four types of challenges that prevent practitioners from effectively using publicly available instruments for measuring representational harms. Our goal is to advance the development of instruments for measuring representational harms that are well-suited to practitioner needs.
  arXiv  Detail & Related papers  (2024-11-23T22:13:38Z)
• Exploring Knowledge Boundaries in Large Language Models for Retrieval Judgment [56.87031484108484]
  Large Language Models (LLMs) are increasingly recognized for their practical applications. Retrieval-Augmented Generation (RAG) tackles this challenge and has shown a significant impact on LLMs. By minimizing retrieval requests that yield neutral or harmful results, we can effectively reduce both time and computational costs.
  arXiv  Detail & Related papers  (2024-11-09T15:12:28Z)
• EEE-Bench: A Comprehensive Multimodal Electrical And Electronics Engineering Benchmark [10.265704144939503]
  Large language models (LLMs) and large multimodal models (LMMs) have demonstrated promising skills in various domains including science and mathematics. We propose EEE-Bench, a multimodal benchmark aimed at assessing LMMs' capabilities in solving practical engineering tasks. Our benchmark consists of 2860 carefully curated problems spanning 10 essential such as analog circuits, control systems, etc.
  arXiv  Detail & Related papers  (2024-11-03T09:17:56Z)
• Evaluating Cultural and Social Awareness of LLM Web Agents [113.49968423990616]
  We introduce CASA, a benchmark designed to assess large language models' sensitivity to cultural and social norms. Our approach evaluates LLM agents' ability to detect and appropriately respond to norm-violating user queries and observations. Experiments show that current LLMs perform significantly better in non-agent environments.
  arXiv  Detail & Related papers  (2024-10-30T17:35:44Z)
• A Reality check of the benefits of LLM in business [1.9181612035055007]
  Large language models (LLMs) have achieved remarkable performance in language understanding and generation tasks. This paper thoroughly examines the usefulness and readiness of LLMs for business processes.
  arXiv  Detail & Related papers  (2024-06-09T02:36:00Z)
• Naming the Pain in Machine Learning-Enabled Systems Engineering [8.092979562919878]
  Machine learning (ML)-enabled systems are being increasingly adopted by companies. This paper aims to deliver a comprehensive overview of the current status quo of engineering ML-enabled systems.
  arXiv  Detail & Related papers  (2024-05-20T06:59:20Z)
• Towards Generating Executable Metamorphic Relations Using Large Language Models [46.26208489175692]
  We propose an approach for automatically deriving executable MRs from requirements using large language models (LLMs) To assess the feasibility of our approach, we conducted a questionnaire-based survey in collaboration with Siemens Industry Software.
  arXiv  Detail & Related papers  (2024-01-30T13:52:47Z)
• Status Quo and Problems of Requirements Engineering for Machine Learning: Results from an International Survey [7.164324501049983]
  Requirements Engineering (RE) can help address many problems when engineering Machine Learning-enabled systems. We conducted a survey to gather practitioner insights into the status quo and problems of RE in ML-enabled systems. We found significant differences in RE practices within ML projects.
  arXiv  Detail & Related papers  (2023-10-10T15:53:50Z)
• Information Retrieval Meets Large Language Models: A Strategic Report from Chinese IR Community [180.28262433004113]
  Large Language Models (LLMs) have demonstrated exceptional capabilities in text understanding, generation, and knowledge inference. LLMs and humans form a new technical paradigm that is more powerful for information seeking. To thoroughly discuss the transformative impact of LLMs on IR research, the Chinese IR community conducted a strategic workshop in April 2023.
  arXiv  Detail & Related papers  (2023-07-19T05:23:43Z)
• Machine Learning Application Development: Practitioners' Insights [18.114724750441724]
  We report about a survey that aimed to understand the challenges and best practices of ML application development. We synthesize the results obtained from 80 practitioners into 17 findings; outlining challenges and best practices for ML application development. We hope that the reported challenges will inform the research community about topics that need to be investigated to improve the engineering process and the quality of ML-based applications.
  arXiv  Detail & Related papers  (2021-12-31T03:38:37Z)
• Understanding the Usability Challenges of Machine Learning In High-Stakes Decision Making [67.72855777115772]
  Machine learning (ML) is being applied to a diverse and ever-growing set of domains. In many cases, domain experts -- who often have no expertise in ML or data science -- are asked to use ML predictions to make high-stakes decisions. We investigate the ML usability challenges present in the domain of child welfare screening through a series of collaborations with child welfare screeners.
  arXiv  Detail & Related papers  (2021-03-02T22:50:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.