Towards Guidelines for Assessing Qualities of Machine Learning Systems
- URL: http://arxiv.org/abs/2008.11007v1
- Date: Tue, 25 Aug 2020 13:45:54 GMT
- Title: Towards Guidelines for Assessing Qualities of Machine Learning Systems
- Authors: Julien Siebert, Lisa Joeckel, Jens Heidrich, Koji Nakamichi, Kyoko
Ohashi, Isao Namba, Rieko Yamamoto, Mikio Aoyama
- Abstract summary: This article presents the construction of a quality model for an ML system based on an industrial use case.
In the future, we want to learn how the term quality differs between different types of ML systems.
- Score: 1.715032913622871
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Nowadays, systems containing components based on machine learning (ML)
methods are becoming more widespread. In order to ensure the intended behavior
of a software system, there are standards that define necessary quality aspects
of the system and its components (such as ISO/IEC 25010). Due to the different
nature of ML, we have to adjust quality aspects or add additional ones (such as
trustworthiness) and be very precise about which aspect is really relevant for
which object of interest (such as completeness of training data), and how to
objectively assess adherence to quality requirements. In this article, we
present the construction of a quality model (i.e., evaluation objects, quality
aspects, and metrics) for an ML system based on an industrial use case. This
quality model enables practitioners to specify and assess quality requirements
for such kinds of ML systems objectively. In the future, we want to learn how
the term quality differs between different types of ML systems and come up with
general guidelines for specifying and assessing qualities of ML systems.
Related papers
- Benchmarks as Microscopes: A Call for Model Metrology [76.64402390208576]
Modern language models (LMs) pose a new challenge in capability assessment.
To be confident in our metrics, we need a new discipline of model metrology.
arXiv Detail & Related papers (2024-07-22T17:52:12Z) - A Fuzzy Logic-Based Quality Model For Identifying Microservices With Low Maintainability [0.0]
This paper proposes a hierarchical quality model based on fuzzy logic to measure and evaluate the maintainability of MSAs.
We use a fuzzification technique to transform crisp values of code metrics into fuzzy levels and apply them as inputs to our quality model.
arXiv Detail & Related papers (2024-06-20T16:53:37Z) - Bringing Quantum Algorithms to Automated Machine Learning: A Systematic
Review of AutoML Frameworks Regarding Extensibility for QML Algorithms [1.4469725791865982]
This work describes the selection approach and analysis of existing AutoML frameworks regarding their capability of incorporating Quantum Machine Learning (QML) algorithms.
For that, available open-source tools are condensed into a market overview and suitable frameworks are systematically selected on a multi-phase, multi-criteria approach.
We build an extended Automated Quantum Machine Learning (AutoQML) framework with QC-specific pipeline steps and decision characteristics for hardware and software constraints.
arXiv Detail & Related papers (2023-10-06T13:21:16Z) - Identifying Concerns When Specifying Machine Learning-Enabled Systems: A
Perspective-Based Approach [1.2184324428571227]
PerSpecML is a perspective-based approach for specifying ML-enabled systems.
It helps practitioners identify which attributes, including ML and non-ML components, are important to contribute to the overall system's quality.
arXiv Detail & Related papers (2023-09-14T18:31:16Z) - Machine Learning-Enabled Software and System Architecture Frameworks [48.87872564630711]
The stakeholders with data science and Machine Learning related concerns, such as data scientists and data engineers, are yet to be included in existing architecture frameworks.
We surveyed 61 subject matter experts from over 25 organizations in 10 countries.
arXiv Detail & Related papers (2023-08-09T21:54:34Z) - QI2 -- an Interactive Tool for Data Quality Assurance [63.379471124899915]
The planned AI Act from the European commission defines challenging legal requirements for data quality.
We introduce a novel approach that supports the data quality assurance process of multiple data quality aspects.
arXiv Detail & Related papers (2023-07-07T07:06:38Z) - Towards Perspective-Based Specification of Machine Learning-Enabled
Systems [1.3406258114080236]
This paper describes our work towards a perspective-based approach for specifying ML-enabled systems.
The approach involves analyzing a set of 45 ML concerns grouped into five perspectives: objectives, user experience, infrastructure, model, and data.
The main contribution of this paper is to provide two new artifacts that can be used to help specifying ML-enabled systems.
arXiv Detail & Related papers (2022-06-20T13:09:23Z) - Panoramic Learning with A Standardized Machine Learning Formalism [116.34627789412102]
This paper presents a standardized equation of the learning objective, that offers a unifying understanding of diverse ML algorithms.
It also provides guidance for mechanic design of new ML solutions, and serves as a promising vehicle towards panoramic learning with all experiences.
arXiv Detail & Related papers (2021-08-17T17:44:38Z) - Quality Assurance Challenges for Machine Learning Software Applications
During Software Development Life Cycle Phases [1.4213973379473654]
The paper conducts an in-depth review of literature on the quality assurance of Machine Learning (ML) models.
We develop a taxonomy of MLSA quality assurance issues by mapping the various ML adoption challenges across different phases of software development life cycles (SDLC)
This mapping can help prioritize quality assurance efforts of MLSAs where the adoption of ML models can be considered crucial.
arXiv Detail & Related papers (2021-05-03T22:29:23Z) - Technology Readiness Levels for AI & ML [79.22051549519989]
Development of machine learning systems can be executed easily with modern tools, but the process is typically rushed and means-to-an-end.
Engineering systems follow well-defined processes and testing standards to streamline development for high-quality, reliable results.
We propose a proven systems engineering approach for machine learning development and deployment.
arXiv Detail & Related papers (2020-06-21T17:14:34Z) - Unsupervised Quality Estimation for Neural Machine Translation [63.38918378182266]
Existing approaches require large amounts of expert annotated data, computation and time for training.
We devise an unsupervised approach to QE where no training or access to additional resources besides the MT system itself is required.
We achieve very good correlation with human judgments of quality, rivalling state-of-the-art supervised QE models.
arXiv Detail & Related papers (2020-05-21T12:38:06Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.