Unsupervised Difficulty Estimation with Action Scores

• URL: http://arxiv.org/abs/2011.11461v1
• Date: Mon, 23 Nov 2020 15:18:44 GMT
• Title: Unsupervised Difficulty Estimation with Action Scores
• Authors: Octavio Arriaga and Matias Valdenegro-Toro
• Abstract summary: We present a simple method for calculating a difficulty score based on the accumulation of losses for each sample during training. Our proposed method does not require any modification of the model neither any external supervision, as it can be implemented as callback.
• Score: 7.6146285961466
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Evaluating difficulty and biases in machine learning models has become of extreme importance as current models are now being applied in real-world situations. In this paper we present a simple method for calculating a difficulty score based on the accumulation of losses for each sample during training. We call this the action score. Our proposed method does not require any modification of the model neither any external supervision, as it can be implemented as callback that gathers information from the training process. We test and analyze our approach in two different settings: image classification, and object detection, and we show that in both settings the action score can provide insights about model and dataset biases.

Related papers

• Towards Understanding the Feasibility of Machine Unlearning [14.177012256360635]
  We present a set of novel metrics for quantifying the difficulty of unlearning. Specifically, we propose several metrics to assess the conditions necessary for a successful unlearning operation. We also present a ranking mechanism to identify the most challenging samples to unlearn.
  arXiv  Detail & Related papers  (2024-10-03T23:41:42Z)
• Reinforcing Pre-trained Models Using Counterfactual Images [54.26310919385808]
  This paper proposes a novel framework to reinforce classification models using language-guided generated counterfactual images. We identify model weaknesses by testing the model using the counterfactual image dataset. We employ the counterfactual images as an augmented dataset to fine-tune and reinforce the classification model.
  arXiv  Detail & Related papers  (2024-06-19T08:07:14Z)
• Distilled Datamodel with Reverse Gradient Matching [74.75248610868685]
  We introduce an efficient framework for assessing data impact, comprising offline training and online evaluation stages. Our proposed method achieves comparable model behavior evaluation while significantly speeding up the process compared to the direct retraining method.
  arXiv  Detail & Related papers  (2024-04-22T09:16:14Z)
• An Information Theoretic Approach to Machine Unlearning [45.600917449314444]
  Key challenge in unlearning is forgetting the necessary data in a timely manner, while preserving model performance. In this work, we address the zero-shot unlearning scenario, whereby an unlearning algorithm must be able to remove data given only a trained model and the data to be forgotten. We derive a simple but principled zero-shot unlearning method based on the geometry of the model.
  arXiv  Detail & Related papers  (2024-02-02T13:33:30Z)
• Generalization Properties of Retrieval-based Models [50.35325326050263]
  Retrieval-based machine learning methods have enjoyed success on a wide range of problems. Despite growing literature showcasing the promise of these models, the theoretical underpinning for such models remains underexplored. We present a formal treatment of retrieval-based models to characterize their generalization ability.
  arXiv  Detail & Related papers  (2022-10-06T00:33:01Z)
• Probing Representation Forgetting in Supervised and Unsupervised Continual Learning [14.462797749666992]
  Catastrophic forgetting is associated with an abrupt loss of knowledge previously learned by a model. We show that representation forgetting can lead to new insights on the effect of model capacity and loss function used in continual learning.
  arXiv  Detail & Related papers  (2022-03-24T23:06:08Z)
• On Modality Bias Recognition and Reduction [70.69194431713825]
  We study the modality bias problem in the context of multi-modal classification. We propose a plug-and-play loss function method, whereby the feature space for each label is adaptively learned. Our method yields remarkable performance improvements compared with the baselines.
  arXiv  Detail & Related papers  (2022-02-25T13:47:09Z)
• CMW-Net: Learning a Class-Aware Sample Weighting Mapping for Robust Deep Learning [55.733193075728096]
  Modern deep neural networks can easily overfit to biased training data containing corrupted labels or class imbalance. Sample re-weighting methods are popularly used to alleviate this data bias issue. We propose a meta-model capable of adaptively learning an explicit weighting scheme directly from data.
  arXiv  Detail & Related papers  (2022-02-11T13:49:51Z)
• Few-shot Action Recognition with Prototype-centered Attentive Learning [88.10852114988829]
  Prototype-centered Attentive Learning (PAL) model composed of two novel components. First, a prototype-centered contrastive learning loss is introduced to complement the conventional query-centered learning objective. Second, PAL integrates a attentive hybrid learning mechanism that can minimize the negative impacts of outliers.
  arXiv  Detail & Related papers  (2021-01-20T11:48:12Z)
• Identifying Wrongly Predicted Samples: A Method for Active Learning [6.976600214375139]
  We propose a simple sample selection criterion that moves beyond uncertainty. We show state-of-the-art results and better rates at identifying wrongly predicted samples.
  arXiv  Detail & Related papers  (2020-10-14T09:00:42Z)
• Visually Analyzing and Steering Zero Shot Learning [2.802183323381949]
  We propose a visual analytics system to help a user analyze and steer zero-shot learning models. Through usage scenarios, we highlight how our system can help a user improve performance in zero-shot learning.
  arXiv  Detail & Related papers  (2020-09-11T06:58:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.