Variability-Aware Machine Learning Model Selection: Feature Modeling, Instantiation, and Experimental Case Study

• URL: http://arxiv.org/abs/2501.00532v1
• Date: Tue, 31 Dec 2024 16:29:37 GMT
• Title: Variability-Aware Machine Learning Model Selection: Feature Modeling, Instantiation, and Experimental Case Study
• Authors: Cristina Tavares, Nathalia Nascimento, Paulo Alencar, Donald Cowan,
• Abstract summary: We present a variability-aware ML algorithm selection approach that considers the commonalities and variations in the model selection process. The proposed approach can be seen as a step towards providing a more explicit, adaptive, transparent, interpretable, and automated basis for model selection.
• Score: 0.6999740786886538
• License:
• Abstract: The emergence of machine learning (ML) has led to a transformative shift in software techniques and guidelines for building software applications that support data analysis process activities such as data ingestion, modeling, and deployment. Specifically, this shift is impacting ML model selection, which is one of the key phases in this process. There have been several advances in model selection from the standpoint of core ML methods, including basic probability measures and resampling methods. However, from a software engineering perspective, this selection is still an ad hoc and informal process, is not supported by a design approach and representation formalism that explicitly captures the selection process and can not support the specification of existing model selection procedures. The selection adapts to a variety of contextual factors that affect the model selection, such as data characteristics, number of features, prediction type, and their intricate dependencies. Further, it does not provide an explanation for selecting a model and does not consider the contextual factors and their interdependencies when selecting a technique. Although the current literature provides a wide variety of ML techniques and algorithms, there is a lack of design approaches to support algorithm selection. In this paper, we present a variability-aware ML algorithm selection approach that considers the commonalities and variations in the model selection process. The approach's applicability is illustrated by an experimental case study based on the Scikit-Learn heuristics, in which existing model selections presented in the literature are compared with selections suggested by the approach. The proposed approach can be seen as a step towards providing a more explicit, adaptive, transparent, interpretable, and automated basis for model selection.

Related papers

• A General Bayesian Framework for Informative Input Design in System Identification [86.05414211113627]
  We tackle the problem of informative input design for system identification. We select inputs, observe the corresponding outputs from the true system, and optimize the parameters of our model to best fit the data. Our method outperforms model-free baselines with various linear and nonlinear dynamics.
  arXiv  Detail & Related papers  (2025-01-28T01:57:51Z)
• Revisiting Demonstration Selection Strategies in In-Context Learning [66.11652803887284]
  Large language models (LLMs) have shown an impressive ability to perform a wide range of tasks using in-context learning (ICL) In this work, we first revisit the factors contributing to this variance from both data and model aspects, and find that the choice of demonstration is both data- and model-dependent. We propose a data- and model-dependent demonstration selection method, textbfTopK + ConE, based on the assumption that textitthe performance of a demonstration positively correlates with its contribution to the model's understanding of the test samples.
  arXiv  Detail & Related papers  (2024-01-22T16:25:27Z)
• Extending Variability-Aware Model Selection with Bias Detection in Machine Learning Projects [0.7646713951724013]
  This paper describes work on extending an adaptive variability-aware model selection method with bias detection in machine learning projects. The proposed approach aims to advance the state of the art by making explicit factors that influence model selection, particularly those related to bias, as well as their interactions.
  arXiv  Detail & Related papers  (2023-11-23T22:08:29Z)
• A Statistical-Modelling Approach to Feedforward Neural Network Model Selection [0.8287206589886881]
  Feedforward neural networks (FNNs) can be viewed as non-linear regression models. A novel model selection method is proposed using the Bayesian information criterion (BIC) for FNNs. The choice of BIC over out-of-sample performance leads to an increased probability of recovering the true model.
  arXiv  Detail & Related papers  (2022-07-09T11:07:04Z)
• An Additive Instance-Wise Approach to Multi-class Model Interpretation [53.87578024052922]
  Interpretable machine learning offers insights into what factors drive a certain prediction of a black-box system. Existing methods mainly focus on selecting explanatory input features, which follow either locally additive or instance-wise approaches. This work exploits the strengths of both methods and proposes a global framework for learning local explanations simultaneously for multiple target classes.
  arXiv  Detail & Related papers  (2022-07-07T06:50:27Z)
• Switchable Representation Learning Framework with Self-compatibility [50.48336074436792]
  We propose a Switchable representation learning Framework with Self-Compatibility (SFSC) SFSC generates a series of compatible sub-models with different capacities through one training process. SFSC achieves state-of-the-art performance on the evaluated datasets.
  arXiv  Detail & Related papers  (2022-06-16T16:46:32Z)
• MACE: An Efficient Model-Agnostic Framework for Counterfactual Explanation [132.77005365032468]
  We propose a novel framework of Model-Agnostic Counterfactual Explanation (MACE) In our MACE approach, we propose a novel RL-based method for finding good counterfactual examples and a gradient-less descent method for improving proximity. Experiments on public datasets validate the effectiveness with better validity, sparsity and proximity.
  arXiv  Detail & Related papers  (2022-05-31T04:57:06Z)
• Model-agnostic and Scalable Counterfactual Explanations via Reinforcement Learning [0.5729426778193398]
  We propose a deep reinforcement learning approach that transforms the optimization procedure into an end-to-end learnable process. Our experiments on real-world data show that our method is model-agnostic, relying only on feedback from model predictions.
  arXiv  Detail & Related papers  (2021-06-04T16:54:36Z)
• Interpretable Multi-dataset Evaluation for Named Entity Recognition [110.64368106131062]
  We present a general methodology for interpretable evaluation for the named entity recognition (NER) task. The proposed evaluation method enables us to interpret the differences in models and datasets, as well as the interplay between them. By making our analysis tool available, we make it easy for future researchers to run similar analyses and drive progress in this area.
  arXiv  Detail & Related papers  (2020-11-13T10:53:27Z)
• Feature Selection Methods for Uplift Modeling and Heterogeneous Treatment Effect [1.349645012479288]
  Uplift modeling is a causal learning technique that estimates subgroup-level treatment effects. Traditional methods for doing feature selection are not fit for the task. We introduce a set of feature selection methods explicitly designed for uplift modeling.
  arXiv  Detail & Related papers  (2020-05-05T00:28:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.