Active Learning++: Incorporating Annotator's Rationale using Local Model Explanation

• URL: http://arxiv.org/abs/2009.04568v1
• Date: Sun, 6 Sep 2020 08:07:33 GMT
• Title: Active Learning++: Incorporating Annotator's Rationale using Local Model Explanation
• Authors: Bhavya Ghai, Q. Vera Liao, Yunfeng Zhang, Klaus Mueller
• Abstract summary: Annotators can provide their rationale for choosing a label by ranking input features based on their importance for a given query. Instead of weighing all committee models equally to select the next instance, we assign higher weight to the committee model with higher agreement with the annotator's ranking. This approach is applicable to any kind of ML model using model-agnostic techniques to generate local explanation such as LIME.
• Score: 84.10721065676913
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We propose a new active learning (AL) framework, Active Learning++, which can utilize an annotator's labels as well as its rationale. Annotators can provide their rationale for choosing a label by ranking input features based on their importance for a given query. To incorporate this additional input, we modified the disagreement measure for a bagging-based Query by Committee (QBC) sampling strategy. Instead of weighing all committee models equally to select the next instance, we assign higher weight to the committee model with higher agreement with the annotator's ranking. Specifically, we generated a feature importance-based local explanation for each committee model. The similarity score between feature rankings provided by the annotator and the local model explanation is used to assign a weight to each corresponding committee model. This approach is applicable to any kind of ML model using model-agnostic techniques to generate local explanation such as LIME. With a simulation study, we show that our framework significantly outperforms a QBC based vanilla AL framework.

Related papers

• The OCON model: an old but gold solution for distributable supervised classification [0.28675177318965045]
  This paper introduces a structured application of the One-Class approach and the One-Class-One-Network model for supervised classification tasks. We achieve classification accuracy comparable to nowadays complex architectures (90.0 - 93.7%)
  arXiv  Detail & Related papers  (2024-10-05T09:15:01Z)
• Prompt Algebra for Task Composition [131.97623832435812]
  We consider Visual Language Models with prompt tuning as our base classifier. We propose constrained prompt tuning to improve performance of the composite classifier. On UTZappos it improves classification accuracy over the best base model by 8.45% on average.
  arXiv  Detail & Related papers  (2023-06-01T03:20:54Z)
• KGxBoard: Explainable and Interactive Leaderboard for Evaluation of Knowledge Graph Completion Models [76.01814380927507]
  KGxBoard is an interactive framework for performing fine-grained evaluation on meaningful subsets of the data. In our experiments, we highlight the findings with the use of KGxBoard, which would have been impossible to detect with standard averaged single-score metrics.
  arXiv  Detail & Related papers  (2022-08-23T15:11:45Z)
• Knowledge Base Question Answering by Case-based Reasoning over Subgraphs [81.22050011503933]
  We show that our model answers queries requiring complex reasoning patterns more effectively than existing KG completion algorithms. The proposed model outperforms or performs competitively with state-of-the-art models on several KBQA benchmarks.
  arXiv  Detail & Related papers  (2022-02-22T01:34:35Z)
• Active metric learning and classification using similarity queries [21.589707834542338]
  We show that a novel unified query framework can be applied to any problem in which a key component is learning a representation of the data that reflects similarity. We demonstrate the effectiveness of the proposed strategy on two tasks -- active metric learning and active classification.
  arXiv  Detail & Related papers  (2022-02-04T03:34:29Z)
• SLADE: A Self-Training Framework For Distance Metric Learning [75.54078592084217]
  We present a self-training framework, SLADE, to improve retrieval performance by leveraging additional unlabeled data. We first train a teacher model on the labeled data and use it to generate pseudo labels for the unlabeled data. We then train a student model on both labels and pseudo labels to generate final feature embeddings.
  arXiv  Detail & Related papers  (2020-11-20T08:26:10Z)
• Probabilistic Case-based Reasoning for Open-World Knowledge Graph Completion [59.549664231655726]
  A case-based reasoning (CBR) system solves a new problem by retrieving cases' that are similar to the given problem. In this paper, we demonstrate that such a system is achievable for reasoning in knowledge-bases (KBs) Our approach predicts attributes for an entity by gathering reasoning paths from similar entities in the KB.
  arXiv  Detail & Related papers  (2020-10-07T17:48:12Z)
• Multi-label learning for dynamic model type recommendation [13.304462985219237]
  We propose a problem-independent dynamic base-classifier model recommendation for the online local pool (OLP) technique. Our proposed framework builds a multi-label meta-classifier responsible for recommending a set of relevant model types. Experimental results show that different data distributions favored different model types on a local scope.
  arXiv  Detail & Related papers  (2020-04-01T16:42:12Z)
• Customized Video QoE Estimation with Algorithm-Agnostic Transfer Learning [1.452875650827562]
  Small datasets, lack of diversity in user profiles in source domain, and too much diversity in target domains of QoE models are challenges for QoE models. We present a transfer learning-based ML model training approach, which allows decentralized local models to share generic indicators on Mean Opinion Scores (MOS) We show that the proposed approach is agnostic to specific ML algorithms, stacked upon each other, as it does not necessitate the collaborating localized nodes to run the same ML algorithm.
  arXiv  Detail & Related papers  (2020-03-12T15:28:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.