Model LineUpper: Supporting Interactive Model Comparison at Multiple Levels for AutoML

• URL: http://arxiv.org/abs/2104.04375v1
• Date: Fri, 9 Apr 2021 14:06:13 GMT
• Title: Model LineUpper: Supporting Interactive Model Comparison at Multiple Levels for AutoML
• Authors: Shweta Narkar, Yunfeng Zhang, Q. Vera Liao, Dakuo Wang, Justin D Weisz
• Abstract summary: In current AutoML systems, selection is supported only by performance metrics. We develop tool to support interactive model comparison for AutoML by integrating multiple Explainable AI (XAI) and visualization techniques.
• Score: 29.04776652873194
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Automated Machine Learning (AutoML) is a rapidly growing set of technologies that automate the model development pipeline by searching model space and generating candidate models. A critical, final step of AutoML is human selection of a final model from dozens of candidates. In current AutoML systems, selection is supported only by performance metrics. Prior work has shown that in practice, people evaluate ML models based on additional criteria, such as the way a model makes predictions. Comparison may happen at multiple levels, from types of errors, to feature importance, to how the model makes predictions of specific instances. We developed \tool{} to support interactive model comparison for AutoML by integrating multiple Explainable AI (XAI) and visualization techniques. We conducted a user study in which we both evaluated the system and used it as a technology probe to understand how users perform model comparison in an AutoML system. We discuss design implications for utilizing XAI techniques for model comparison and supporting the unique needs of data scientists in comparing AutoML models.

Related papers

• Matchmaker: Self-Improving Large Language Model Programs for Schema Matching [60.23571456538149]
  We propose a compositional language model program for schema matching, comprised of candidate generation, refinement and confidence scoring. Matchmaker self-improves in a zero-shot manner without the need for labeled demonstrations. Empirically, we demonstrate on real-world medical schema matching benchmarks that Matchmaker outperforms previous ML-based approaches.
  arXiv  Detail & Related papers  (2024-10-31T16:34:03Z)
• UICoder: Finetuning Large Language Models to Generate User Interface Code through Automated Feedback [21.858896845159208]
  Large language models (LLMs) struggle to consistently generate UI code that compiles and produces visually relevant designs. Existing approaches to improve generation rely on expensive human feedback or distilling a proprietary model. Our method starts with an existing LLM and iteratively produces improved models by self-generating a large synthetic dataset.
  arXiv  Detail & Related papers  (2024-06-11T21:53:46Z)
• Long-Span Question-Answering: Automatic Question Generation and QA-System Ranking via Side-by-Side Evaluation [65.16137964758612]
  We explore the use of long-context capabilities in large language models to create synthetic reading comprehension data from entire books. Our objective is to test the capabilities of LLMs to analyze, understand, and reason over problems that require a detailed comprehension of long spans of text.
  arXiv  Detail & Related papers  (2024-05-31T20:15:10Z)
• AIDE: An Automatic Data Engine for Object Detection in Autonomous Driving [68.73885845181242]
  We propose an Automatic Data Engine (AIDE) that automatically identifies issues, efficiently curates data, improves the model through auto-labeling, and verifies the model through generation of diverse scenarios. We further establish a benchmark for open-world detection on AV datasets to comprehensively evaluate various learning paradigms, demonstrating our method's superior performance at a reduced cost.
  arXiv  Detail & Related papers  (2024-03-26T04:27:56Z)
• Deciphering AutoML Ensembles: cattleia's Assistance in Decision-Making [0.0]
  Cattleia is an application that deciphers the ensembles for regression, multiclass, and binary classification tasks. It works with models built by three AutoML packages: auto-sklearn, AutoGluon, and FLAML.
  arXiv  Detail & Related papers  (2024-03-19T11:56:21Z)
• AutoXPCR: Automated Multi-Objective Model Selection for Time Series Forecasting [1.0515439489916734]
  We propose AutoXPCR - a novel method for automated and explainable multi-objective model selection. Our approach leverages meta-learning to estimate any model's performance along PCR criteria, which encompass (P)redictive error, (C)omplexity, and (R)esource demand. Our method clearly outperforms other model selection approaches - on average, it only requires 20% of computation costs for recommending models with 90% of the best-possible quality.
  arXiv  Detail & Related papers  (2023-12-20T14:04:57Z)
• The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluation [93.01964988474755]
  AutoMQM is a prompting technique which asks large language models to identify and categorize errors in translations. We study the impact of labeled data through in-context learning and finetuning. We then evaluate AutoMQM with PaLM-2 models, and we find that it improves performance compared to just prompting for scores.
  arXiv  Detail & Related papers  (2023-08-14T17:17:21Z)
• Dataless Knowledge Fusion by Merging Weights of Language Models [51.8162883997512]
  Fine-tuning pre-trained language models has become the prevalent paradigm for building downstream NLP models. This creates a barrier to fusing knowledge across individual models to yield a better single model. We propose a dataless knowledge fusion method that merges models in their parameter space.
  arXiv  Detail & Related papers  (2022-12-19T20:46:43Z)
• Automatic Componentwise Boosting: An Interpretable AutoML System [1.1709030738577393]
  We propose an AutoML system that constructs an interpretable additive model that can be fitted using a highly scalable componentwise boosting algorithm. Our system provides tools for easy model interpretation such as visualizing partial effects and pairwise interactions. Despite its restriction to an interpretable model space, our system is competitive in terms of predictive performance on most data sets.
  arXiv  Detail & Related papers  (2021-09-12T18:34:33Z)
• Interpret-able feedback for AutoML systems [5.5524559605452595]
  Automated machine learning (AutoML) systems aim to enable training machine learning (ML) models for non-ML experts. A shortcoming of these systems is that when they fail to produce a model with high accuracy, the user has no path to improve the model. We introduce an interpretable data feedback solution for AutoML.
  arXiv  Detail & Related papers  (2021-02-22T18:54:26Z)
• AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data [120.2298620652828]
  We introduce AutoGluon-Tabular, an open-source AutoML framework that requires only a single line of Python to train highly accurate machine learning models. Tests on a suite of 50 classification and regression tasks from Kaggle and the OpenML AutoML Benchmark reveal that AutoGluon is faster, more robust, and much more accurate.
  arXiv  Detail & Related papers  (2020-03-13T23:10:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.