Self-Updating Models with Error Remediation

• URL: http://arxiv.org/abs/2005.09787v1
• Date: Tue, 19 May 2020 23:09:38 GMT
• Title: Self-Updating Models with Error Remediation
• Authors: Justin E. Doak, Michael R. Smith, Joey B. Ingram
• Abstract summary: We propose a framework, Self-Updating Models with Error Remediation (SUMER), in which a deployed model updates itself as new data becomes available. A key component of SUMER is the notion of error remediation as self-labeled data can be susceptible to the propagation of errors. We find that self-updating models (SUMs) generally perform better than models that do not attempt to self-update when presented with additional previously-unseen data.
• Score: 0.5156484100374059
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Many environments currently employ machine learning models for data processing and analytics that were built using a limited number of training data points. Once deployed, the models are exposed to significant amounts of previously-unseen data, not all of which is representative of the original, limited training data. However, updating these deployed models can be difficult due to logistical, bandwidth, time, hardware, and/or data sensitivity constraints. We propose a framework, Self-Updating Models with Error Remediation (SUMER), in which a deployed model updates itself as new data becomes available. SUMER uses techniques from semi-supervised learning and noise remediation to iteratively retrain a deployed model using intelligently-chosen predictions from the model as the labels for new training iterations. A key component of SUMER is the notion of error remediation as self-labeled data can be susceptible to the propagation of errors. We investigate the use of SUMER across various data sets and iterations. We find that self-updating models (SUMs) generally perform better than models that do not attempt to self-update when presented with additional previously-unseen data. This performance gap is accentuated in cases where there is only limited amounts of initial training data. We also find that the performance of SUMER is generally better than the performance of SUMs, demonstrating a benefit in applying error remediation. Consequently, SUMER can autonomously enhance the operational capabilities of existing data processing systems by intelligently updating models in dynamic environments.

Related papers

• MUSCLE: A Model Update Strategy for Compatible LLM Evolution [29.032461144831053]
  Large Language Models (LLMs) are regularly updated to enhance performance. Instance-level degradation (instance regression) of performance from one model version to the next can interfere with a user's mental model of the capabilities of a particular language model. We propose a training strategy to minimize the extent of instance regression in model updates.
  arXiv  Detail & Related papers  (2024-07-12T17:12:48Z)
• Data Shapley in One Training Run [88.59484417202454]
  Data Shapley provides a principled framework for attributing data's contribution within machine learning contexts. Existing approaches require re-training models on different data subsets, which is computationally intensive. This paper introduces In-Run Data Shapley, which addresses these limitations by offering scalable data attribution for a target model of interest.
  arXiv  Detail & Related papers  (2024-06-16T17:09:24Z)
• Dataless Knowledge Fusion by Merging Weights of Language Models [51.8162883997512]
  Fine-tuning pre-trained language models has become the prevalent paradigm for building downstream NLP models. This creates a barrier to fusing knowledge across individual models to yield a better single model. We propose a dataless knowledge fusion method that merges models in their parameter space.
  arXiv  Detail & Related papers  (2022-12-19T20:46:43Z)
• Synthetic Model Combination: An Instance-wise Approach to Unsupervised Ensemble Learning [92.89846887298852]
  Consider making a prediction over new test data without any opportunity to learn from a training set of labelled data. Give access to a set of expert models and their predictions alongside some limited information about the dataset used to train them.
  arXiv  Detail & Related papers  (2022-10-11T10:20:31Z)
• Learning from aggregated data with a maximum entropy model [73.63512438583375]
  We show how a new model, similar to a logistic regression, may be learned from aggregated data only by approximating the unobserved feature distribution with a maximum entropy hypothesis. We present empirical evidence on several public datasets that the model learned this way can achieve performances comparable to those of a logistic model trained with the full unaggregated data.
  arXiv  Detail & Related papers  (2022-10-05T09:17:27Z)
• AI Total: Analyzing Security ML Models with Imperfect Data in Production [2.629585075202626]
  Development of new machine learning models is typically done on manually curated data sets. We develop a web-based visualization system that allows the users to quickly gather headline performance numbers. It also enables the users to immediately observe the root cause of an issue when something goes wrong.
  arXiv  Detail & Related papers  (2021-10-13T20:56:05Z)
• Exposing Shallow Heuristics of Relation Extraction Models with Challenge Data [49.378860065474875]
  We identify failure modes of SOTA relation extraction (RE) models trained on TACRED. By adding some of the challenge data as training examples, the performance of the model improves.
  arXiv  Detail & Related papers  (2020-10-07T21:17:25Z)
• A Simple and Interpretable Predictive Model for Healthcare [0.0]
  Deep learning models are currently dominating most state-of-the-art solutions for disease prediction. These deep learning models, with trainable parameters running into millions, require huge amounts of compute and data to train and deploy. We develop a simpler yet interpretable non-deep learning based model for application to EHR data.
  arXiv  Detail & Related papers  (2020-07-27T08:13:37Z)
• Model Reuse with Reduced Kernel Mean Embedding Specification [70.044322798187]
  We present a two-phase framework for finding helpful models for a current application. In the upload phase, when a model is uploading into the pool, we construct a reduced kernel mean embedding (RKME) as a specification for the model. Then in the deployment phase, the relatedness of the current task and pre-trained models will be measured based on the value of the RKME specification.
  arXiv  Detail & Related papers  (2020-01-20T15:15:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.