Quantifying Local Model Validity using Active Learning

• URL: http://arxiv.org/abs/2406.07474v2
• Date: Mon, 17 Jun 2024 17:19:01 GMT
• Title: Quantifying Local Model Validity using Active Learning
• Authors: Sven Lämmle, Can Bogoclu, Robert Voßhall, Anselm Haselhoff, Dirk Roos,
• Abstract summary: Real-world applications of machine learning models are often subject to legal or policy-based regulations. Some of these regulations require ensuring the validity of the model, i.e., the approximation error being smaller than a threshold. We propose learning the model error to acquire a local validity estimate while reducing the amount of required data through active learning.
• Score: 2.8078480738404
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Real-world applications of machine learning models are often subject to legal or policy-based regulations. Some of these regulations require ensuring the validity of the model, i.e., the approximation error being smaller than a threshold. A global metric is generally too insensitive to determine the validity of a specific prediction, whereas evaluating local validity is costly since it requires gathering additional data.We propose learning the model error to acquire a local validity estimate while reducing the amount of required data through active learning. Using model validation benchmarks, we provide empirical evidence that the proposed method can lead to an error model with sufficient discriminative properties using a relatively small amount of data. Furthermore, an increased sensitivity to local changes of the validity bounds compared to alternative approaches is demonstrated.

Related papers

• Application and Optimization of Large Models Based on Prompt Tuning for Fact-Check-Worthiness Estimation [9.577859600179083]
  We construct a model for fact-check-worthiness estimation at the methodological level using prompt tuning. We establish in-context learning and leverage prompt tuning technology to improve the accuracy of determining whether claims have fact-check-worthiness.
  arXiv  Detail & Related papers  (2025-04-25T06:16:41Z)
• Distribution Learning with Valid Outputs Beyond the Worst-Case [25.788559173418363]
  Validity-constrained distribution learning attempts to address this problem by requiring that the learned distribution have a provably small fraction of its mass in invalid parts of space. We show that when the data distribution lies in the model class and the log-loss is minimized, the number of samples required to ensure validity has a weak dependence on the validity requirement.
  arXiv  Detail & Related papers  (2024-10-21T17:56:09Z)
• Certain and Approximately Certain Models for Statistical Learning [4.318959672085627]
  We show that it is possible to learn accurate models directly from data with missing values for certain training data and target models. We build efficient algorithms with theoretical guarantees to check this necessity and return accurate models in cases where imputation is unnecessary.
  arXiv  Detail & Related papers  (2024-02-27T22:49:33Z)
• Robustness, Evaluation and Adaptation of Machine Learning Models in the Wild [4.304803366354879]
  We study causes of impaired robustness to domain shifts and present algorithms for training domain robust models. A key source of model brittleness is due to domain overfitting, which our new training algorithms suppress and instead encourage domain-general hypotheses.
  arXiv  Detail & Related papers  (2023-03-05T21:41:16Z)
• Conditional expectation with regularization for missing data imputation [19.254291863337347]
  Missing data frequently occurs in datasets across various domains, such as medicine, sports, and finance. We propose a new algorithm named "conditional Distribution-based Imputation of Missing Values with Regularization" (DIMV) DIMV operates by determining the conditional distribution of a feature that has missing entries, using the information from the fully observed features as a basis.
  arXiv  Detail & Related papers  (2023-02-02T06:59:15Z)
• Exploring validation metrics for offline model-based optimisation with diffusion models [50.404829846182764]
  In model-based optimisation (MBO) we are interested in using machine learning to design candidates that maximise some measure of reward with respect to a black box function called the (ground truth) oracle. While an approximation to the ground oracle can be trained and used in place of it during model validation to measure the mean reward over generated candidates, the evaluation is approximate and vulnerable to adversarial examples. This is encapsulated under our proposed evaluation framework which is also designed to measure extrapolation.
  arXiv  Detail & Related papers  (2022-11-19T16:57:37Z)
• Canary in a Coalmine: Better Membership Inference with Ensembled Adversarial Queries [53.222218035435006]
  We use adversarial tools to optimize for queries that are discriminative and diverse. Our improvements achieve significantly more accurate membership inference than existing methods.
  arXiv  Detail & Related papers  (2022-10-19T17:46:50Z)
• MACE: An Efficient Model-Agnostic Framework for Counterfactual Explanation [132.77005365032468]
  We propose a novel framework of Model-Agnostic Counterfactual Explanation (MACE) In our MACE approach, we propose a novel RL-based method for finding good counterfactual examples and a gradient-less descent method for improving proximity. Experiments on public datasets validate the effectiveness with better validity, sparsity and proximity.
  arXiv  Detail & Related papers  (2022-05-31T04:57:06Z)
• Knowledge-driven Active Learning [70.37119719069499]
  Active learning strategies aim at minimizing the amount of labelled data required to train a Deep Learning model. Most active strategies are based on uncertain sample selection, and even often restricted to samples lying close to the decision boundary. Here we propose to take into consideration common domain-knowledge and enable non-expert users to train a model with fewer samples.
  arXiv  Detail & Related papers  (2021-10-15T06:11:53Z)
• ALT-MAS: A Data-Efficient Framework for Active Testing of Machine Learning Algorithms [58.684954492439424]
  We propose a novel framework to efficiently test a machine learning model using only a small amount of labeled test data. The idea is to estimate the metrics of interest for a model-under-test using Bayesian neural network (BNN)
  arXiv  Detail & Related papers  (2021-04-11T12:14:04Z)
• Scalable Marginal Likelihood Estimation for Model Selection in Deep Learning [78.83598532168256]
  Marginal-likelihood based model-selection is rarely used in deep learning due to estimation difficulties. Our work shows that marginal likelihoods can improve generalization and be useful when validation data is unavailable.
  arXiv  Detail & Related papers  (2021-04-11T09:50:24Z)
• Identifying Wrongly Predicted Samples: A Method for Active Learning [6.976600214375139]
  We propose a simple sample selection criterion that moves beyond uncertainty. We show state-of-the-art results and better rates at identifying wrongly predicted samples.
  arXiv  Detail & Related papers  (2020-10-14T09:00:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.