Monitoring and explainability of models in production

• URL: http://arxiv.org/abs/2007.06299v1
• Date: Mon, 13 Jul 2020 10:37:05 GMT
• Title: Monitoring and explainability of models in production
• Authors: Janis Klaise, Arnaud Van Looveren, Clive Cox, Giovanni Vacanti, Alexandru Coca
• Abstract summary: Monitoring deployed models is crucial for continued provision of high quality machine learning enabled services. We discuss the challenges to successful implementation of solutions in each of these areas with some recent examples of production ready solutions using open source tools.
• Score: 58.720142291102135
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The machine learning lifecycle extends beyond the deployment stage. Monitoring deployed models is crucial for continued provision of high quality machine learning enabled services. Key areas include model performance and data monitoring, detecting outliers and data drift using statistical techniques, and providing explanations of historic predictions. We discuss the challenges to successful implementation of solutions in each of these areas with some recent examples of production ready solutions using open source tools.

Related papers

• Meta-Statistical Learning: Supervised Learning of Statistical Inference [59.463430294611626]
  This work demonstrates that the tools and principles driving the success of large language models (LLMs) can be repurposed to tackle distribution-level tasks. We propose meta-statistical learning, a framework inspired by multi-instance learning that reformulates statistical inference tasks as supervised learning problems.
  arXiv  Detail & Related papers  (2025-02-17T18:04:39Z)
• Data-driven tool wear prediction in milling, based on a process-integrated single-sensor approach [1.6574413179773764]
  This study explores data-driven methods, in particular deep learning, for tool wear prediction. The study evaluates several machine learning models, including convolutional neural networks (CNN), long short-term memory networks (LSTM), support vector machines (SVM) and decision trees. The ConvNeXt model has an exceptional performance, achieving a 99.1% accuracy in identifying tool wear using data from only four milling tools operated until they are worn.
  arXiv  Detail & Related papers  (2024-12-27T23:10:32Z)
• Explanatory Model Monitoring to Understand the Effects of Feature Shifts on Performance [61.06245197347139]
  We propose a novel approach to explain the behavior of a black-box model under feature shifts. We refer to our method that combines concepts from Optimal Transport and Shapley Values as Explanatory Performance Estimation.
  arXiv  Detail & Related papers  (2024-08-24T18:28:19Z)
• Safe AI for health and beyond -- Monitoring to transform a health service [51.8524501805308]
  We will assess the infrastructure required to monitor the outputs of a machine learning algorithm. We will present two scenarios with examples of monitoring and updates of models.
  arXiv  Detail & Related papers  (2023-03-02T17:27:45Z)
• A monitoring framework for deployed machine learning models with supply chain examples [2.904613270228912]
  We describe a framework for monitoring machine learning models; and, (2) its implementation for a big data supply chain application. We use our implementation to study drift in model features, predictions, and performance on three real data sets.
  arXiv  Detail & Related papers  (2022-11-11T14:31:38Z)
• SOLIS -- The MLOps journey from data acquisition to actionable insights [62.997667081978825]
  In this paper we present a unified deployment pipeline and freedom-to-operate approach that supports all requirements while using basic cross-platform tensor framework and script language engines. This approach however does not supply the needed procedures and pipelines for the actual deployment of machine learning capabilities in real production grade systems.
  arXiv  Detail & Related papers  (2021-12-22T14:45:37Z)
• Self-Supervised Representation Learning: Introduction, Advances and Challenges [125.38214493654534]
  Self-supervised representation learning methods aim to provide powerful deep feature learning without the requirement of large annotated datasets. This article introduces this vibrant area including key concepts, the four main families of approach and associated state of the art, and how self-supervised methods are applied to diverse modalities of data.
  arXiv  Detail & Related papers  (2021-10-18T13:51:22Z)
• Explainable Machine Learning for Fraud Detection [0.47574189356217006]
  The application of machine learning to support the processing of large datasets holds promise in many industries, including financial services. In this paper, we explore explainability methods in the domain of real-time fraud detection by investigating the selection of appropriate background datasets and runtime trade-offs on both supervised and unsupervised models.
  arXiv  Detail & Related papers  (2021-05-13T14:12:02Z)
• Analytics of Longitudinal System Monitoring Data for Performance Prediction [0.7004662712880577]
  We create data-driven models that can predict the performance of jobs waiting in the scheduler queue. We analyze these prediction models in great detail to identify the features that are dominant predictors of performance. We demonstrate that such models can be application-agnostic and can be used for predicting performance of applications that are not included in training.
  arXiv  Detail & Related papers  (2020-07-07T13:57:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.