Interpret-able feedback for AutoML systems

• URL: http://arxiv.org/abs/2102.11267v1
• Date: Mon, 22 Feb 2021 18:54:26 GMT
• Title: Interpret-able feedback for AutoML systems
• Authors: Behnaz Arzani, Kevin Hsieh, Haoxian Chen
• Abstract summary: Automated machine learning (AutoML) systems aim to enable training machine learning (ML) models for non-ML experts. A shortcoming of these systems is that when they fail to produce a model with high accuracy, the user has no path to improve the model. We introduce an interpretable data feedback solution for AutoML.
• Score: 5.5524559605452595
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Automated machine learning (AutoML) systems aim to enable training machine learning (ML) models for non-ML experts. A shortcoming of these systems is that when they fail to produce a model with high accuracy, the user has no path to improve the model other than hiring a data scientist or learning ML -- this defeats the purpose of AutoML and limits its adoption. We introduce an interpretable data feedback solution for AutoML. Our solution suggests new data points for the user to label (without requiring a pool of unlabeled data) to improve the model's accuracy. Our solution analyzes how features influence the prediction among all ML models in an AutoML ensemble, and we suggest more data samples from feature ranges that have high variance in such analysis. Our evaluation shows that our solution can improve the accuracy of AutoML by 7-8% and significantly outperforms popular active learning solutions in data efficiency, all the while providing the added benefit of being interpretable.

Related papers

• AutoML-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoML [56.565200973244146]
  Automated machine learning (AutoML) accelerates AI development by automating tasks in the development pipeline. Recent works have started exploiting large language models (LLM) to lessen such burden. This paper proposes AutoML-Agent, a novel multi-agent framework tailored for full-pipeline AutoML.
  arXiv  Detail & Related papers  (2024-10-03T20:01:09Z)
• LML-DAP: Language Model Learning a Dataset for Data-Augmented Prediction [0.0]
  This paper introduces a new approach for classification tasks using Large Language Models (LLMs) in an explainable method. The classification is performed by LLMs using a method similar to that used by humans who manually explore and understand the data to decide classifications. The system scored an accuracy above 90% in some test cases, confirming the effectiveness and potential of the system to outperform Machine Learning models in various scenarios.
  arXiv  Detail & Related papers  (2024-09-27T17:58:50Z)
• Verbalized Machine Learning: Revisiting Machine Learning with Language Models [63.10391314749408]
  We introduce the framework of verbalized machine learning (VML) VML constrains the parameter space to be human-interpretable natural language. We empirically verify the effectiveness of VML, and hope that VML can serve as a stepping stone to stronger interpretability.
  arXiv  Detail & Related papers  (2024-06-06T17:59:56Z)
• Position: A Call to Action for a Human-Centered AutoML Paradigm [83.78883610871867]
  Automated machine learning (AutoML) was formed around the fundamental objectives of automatically and efficiently configuring machine learning (ML) We argue that a key to unlocking AutoML's full potential lies in addressing the currently underexplored aspect of user interaction with AutoML systems.
  arXiv  Detail & Related papers  (2024-06-05T15:05:24Z)
• MLLM-DataEngine: An Iterative Refinement Approach for MLLM [62.30753425449056]
  We propose a novel closed-loop system that bridges data generation, model training, and evaluation. Within each loop, the MLLM-DataEngine first analyze the weakness of the model based on the evaluation results. For targeting, we propose an Adaptive Bad-case Sampling module, which adjusts the ratio of different types of data. For quality, we resort to GPT-4 to generate high-quality data with each given data type.
  arXiv  Detail & Related papers  (2023-08-25T01:41:04Z)
• From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning [52.257422715393574]
  We introduce a self-guided methodology for Large Language Models (LLMs) to autonomously discern and select cherry samples from open-source datasets. Our key innovation, the Instruction-Following Difficulty (IFD) metric, emerges as a pivotal metric to identify discrepancies between a model's expected responses and its intrinsic generation capability.
  arXiv  Detail & Related papers  (2023-08-23T09:45:29Z)
• The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluation [93.01964988474755]
  AutoMQM is a prompting technique which asks large language models to identify and categorize errors in translations. We study the impact of labeled data through in-context learning and finetuning. We then evaluate AutoMQM with PaLM-2 models, and we find that it improves performance compared to just prompting for scores.
  arXiv  Detail & Related papers  (2023-08-14T17:17:21Z)
• XAutoML: A Visual Analytics Tool for Understanding and Validating Automated Machine Learning [5.633209323925663]
  XAutoML is an interactive visual analytics tool for explaining arbitrary AutoML optimization procedures and ML pipelines constructed by AutoML. XAutoML combines interactive visualizations with established techniques from explainable artificial intelligence (XAI) to make the complete AutoML procedure transparent and explainable.
  arXiv  Detail & Related papers  (2022-02-24T08:18:25Z)
• Automatic Componentwise Boosting: An Interpretable AutoML System [1.1709030738577393]
  We propose an AutoML system that constructs an interpretable additive model that can be fitted using a highly scalable componentwise boosting algorithm. Our system provides tools for easy model interpretation such as visualizing partial effects and pairwise interactions. Despite its restriction to an interpretable model space, our system is competitive in terms of predictive performance on most data sets.
  arXiv  Detail & Related papers  (2021-09-12T18:34:33Z)
• Man versus Machine: AutoML and Human Experts' Role in Phishing Detection [4.124446337711138]
  This paper compares the performances of six well-known, state-of-the-art AutoML frameworks on ten different phishing datasets. Our results indicate that AutoML-based models are able to outperform manually developed machine learning models in complex classification tasks.
  arXiv  Detail & Related papers  (2021-08-27T09:26:20Z)
• Robusta: Robust AutoML for Feature Selection via Reinforcement Learning [24.24652530951966]
  We propose the first robust AutoML framework, Robusta--based on reinforcement learning (RL) We show that the framework is able to improve the model robustness by up to 22% while maintaining competitive accuracy on benign samples.
  arXiv  Detail & Related papers  (2021-01-15T03:12:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.