Expert-Driven Monitoring of Operational ML Models

• URL: http://arxiv.org/abs/2401.11993v1
• Date: Mon, 22 Jan 2024 14:46:41 GMT
• Title: Expert-Driven Monitoring of Operational ML Models
• Authors: Joran Leest, Claudia Raibulet, Ilias Gerostathopoulos, Patricia Lago
• Abstract summary: We propose Expert Monitoring to enhance the detection and mitigation of concept drift in machine learning (ML) models. Our approach supports practitioners by consolidating domain expertise related to concept drift-inducing events, making this expertise accessible to on-call personnel.
• Score: 11.118653703503599
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We propose Expert Monitoring, an approach that leverages domain expertise to enhance the detection and mitigation of concept drift in machine learning (ML) models. Our approach supports practitioners by consolidating domain expertise related to concept drift-inducing events, making this expertise accessible to on-call personnel, and enabling automatic adaptability with expert oversight.

Related papers

• Offline Imitation Learning with Model-based Reverse Augmentation [48.64791438847236]
  We propose a novel model-based framework, called offline Imitation Learning with Self-paced Reverse Augmentation. Specifically, we build a reverse dynamic model from the offline demonstrations, which can efficiently generate trajectories leading to the expert-observed states. We use the subsequent reinforcement learning method to learn from the augmented trajectories and transit from expert-unobserved states to expert-observed states.
  arXiv  Detail & Related papers  (2024-06-18T12:27:02Z)
• An Explanatory Model Steering System for Collaboration between Domain Experts and AI [4.398998634035531]
  We introduce an Explanatory Model Steering system that allows domain experts to steer prediction models using their domain knowledge. The system includes an explanation dashboard that combines different types of data-centric and model-centric explanations. Our findings highlight the importance of involving domain experts during model steering, ultimately leading to improved human-AI collaboration.
  arXiv  Detail & Related papers  (2024-05-17T07:27:48Z)
• DEEM: Dynamic Experienced Expert Modeling for Stance Detection [22.826544082557316]
  We propose a Dynamic Experienced Expert Modeling (DEEM) method which can leverage the generated experts and let LLMs reason in a semi-parametric way. Experimental results demonstrate that DEEM consistently achieves the best results on three standard benchmarks.
  arXiv  Detail & Related papers  (2024-02-23T11:24:00Z)
• Visualizing Transferred Knowledge: An Interpretive Model of Unsupervised Domain Adaptation [70.85686267987744]
  Unsupervised domain adaptation problems can transfer knowledge from a labeled source domain to an unlabeled target domain. We propose an interpretive model of unsupervised domain adaptation, as the first attempt to visually unveil the mystery of transferred knowledge. Our method provides an intuitive explanation for the base model's predictions and unveils transfer knowledge by matching the image patches with the same semantics across both source and target domains.
  arXiv  Detail & Related papers  (2023-03-04T03:02:12Z)
• Measuring Competency of Machine Learning Systems and Enforcing Reliability [0.0]
  We explore the impact of environmental conditions on the competency of machine learning agents. We learn a representation of conditions which impact the strategies and performance of the ML agent.
  arXiv  Detail & Related papers  (2022-12-02T19:37:37Z)
• The Ability of Image-Language Explainable Models to Resemble Domain Expertise [0.0]
  We study the use of the local surrogate explainability technique to overcome the problem of black-box deep learning models. We demonstrate that such explanations can serve as helpful feedback in guiding model training for data scientists and machine learning engineers in the field.
  arXiv  Detail & Related papers  (2022-09-19T19:09:51Z)
• Human-in-the-Loop Large-Scale Predictive Maintenance of Workstations [89.51621054382878]
  Predictive maintenance (PdM) is the task of scheduling maintenance operations based on a statistical analysis of the system's condition. We propose a human-in-the-loop PdM approach in which a machine learning system predicts future problems in sets of workstations.
  arXiv  Detail & Related papers  (2022-06-23T09:40:46Z)
• Perspectives on Incorporating Expert Feedback into Model Updates [46.99664744930785]
  We devise a taxonomy to match expert feedback types with practitioner updates. A practitioner may receive feedback from an expert at the observation- or domain-level. We review existing work from ML and human-computer interaction to describe this feedback-update taxonomy.
  arXiv  Detail & Related papers  (2022-05-13T21:46:55Z)
• Soft Expert Reward Learning for Vision-and-Language Navigation [94.86954695912125]
  Vision-and-Language Navigation (VLN) requires an agent to find a specified spot in an unseen environment by following natural language instructions. We introduce a Soft Expert Reward Learning (SERL) model to overcome the reward engineering designing and generalisation problems of the VLN task.
  arXiv  Detail & Related papers  (2020-07-21T14:17:36Z)
• Multiple Expert Brainstorming for Domain Adaptive Person Re-identification [140.3998019639158]
  We propose a multiple expert brainstorming network (MEB-Net) for domain adaptive person re-ID. MEB-Net adopts a mutual learning strategy, where multiple networks with different architectures are pre-trained within a source domain. Experiments on large-scale datasets demonstrate the superior performance of MEB-Net over the state-of-the-arts.
  arXiv  Detail & Related papers  (2020-07-03T08:16:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.