Beyond Labels: Empowering Human Annotators with Natural Language Explanations through a Novel Active-Learning Architecture

• URL: http://arxiv.org/abs/2305.12710v2
• Date: Mon, 23 Oct 2023 16:44:59 GMT
• Title: Beyond Labels: Empowering Human Annotators with Natural Language Explanations through a Novel Active-Learning Architecture
• Authors: Bingsheng Yao, Ishan Jindal, Lucian Popa, Yannis Katsis, Sayan Ghosh, Lihong He, Yuxuan Lu, Shashank Srivastava, Yunyao Li, James Hendler, Dakuo Wang
• Abstract summary: Real-world domain experts (e.g., doctors) rarely annotate only a decision label in their day-to-day workflow without providing explanations. This work proposes a novel Active Learning architecture to support experts' real-world need for label and explanation annotations.
• Score: 43.85335847262138
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Real-world domain experts (e.g., doctors) rarely annotate only a decision label in their day-to-day workflow without providing explanations. Yet, existing low-resource learning techniques, such as Active Learning (AL), that aim to support human annotators mostly focus on the label while neglecting the natural language explanation of a data point. This work proposes a novel AL architecture to support experts' real-world need for label and explanation annotations in low-resource scenarios. Our AL architecture leverages an explanation-generation model to produce explanations guided by human explanations, a prediction model that utilizes generated explanations toward prediction faithfully, and a novel data diversity-based AL sampling strategy that benefits from the explanation annotations. Automated and human evaluations demonstrate the effectiveness of incorporating explanations into AL sampling and the improved human annotation efficiency and trustworthiness with our AL architecture. Additional ablation studies illustrate the potential of our AL architecture for transfer learning, generalizability, and integration with large language models (LLMs). While LLMs exhibit exceptional explanation-generation capabilities for relatively simple tasks, their effectiveness in complex real-world tasks warrants further in-depth study.

Related papers

• Evolving Knowledge Distillation with Large Language Models and Active Learning [46.85430680828938]
  Large language models (LLMs) have demonstrated remarkable capabilities across various NLP tasks. Previous research has attempted to distill the knowledge of LLMs into smaller models by generating annotated data. We propose EvoKD: Evolving Knowledge Distillation, which leverages the concept of active learning to interactively enhance the process of data generation using large language models.
  arXiv  Detail & Related papers  (2024-03-11T03:55:24Z)
• Large Language Models for Scientific Information Extraction: An Empirical Study for Virology [0.0]
  We champion the use of structured and semantic content representation of discourse-based scholarly communication. Inspired by tools like Wikipedia infoboxes or structured Amazon product descriptions, we develop an automated approach to produce structured scholarly contribution summaries. Our results show that finetuned FLAN-T5 with 1000x fewer parameters than the state-of-the-art GPT-davinci is competitive for the task.
  arXiv  Detail & Related papers  (2024-01-18T15:04:55Z)
• FIND: A Function Description Benchmark for Evaluating Interpretability Methods [86.80718559904854]
  This paper introduces FIND (Function INterpretation and Description), a benchmark suite for evaluating automated interpretability methods. FIND contains functions that resemble components of trained neural networks, and accompanying descriptions of the kind we seek to generate. We evaluate methods that use pretrained language models to produce descriptions of function behavior in natural language and code.
  arXiv  Detail & Related papers  (2023-09-07T17:47:26Z)
• Exploring Large Language Model for Graph Data Understanding in Online Job Recommendations [63.19448893196642]
  We present a novel framework that harnesses the rich contextual information and semantic representations provided by large language models to analyze behavior graphs. By leveraging this capability, our framework enables personalized and accurate job recommendations for individual users.
  arXiv  Detail & Related papers  (2023-07-10T11:29:41Z)
• Post Hoc Explanations of Language Models Can Improve Language Models [43.2109029463221]
  We present a novel framework, Amplifying Model Performance by Leveraging In-Context Learning with Post Hoc Explanations (AMPLIFY) We leverage post hoc explanation methods which output attribution scores (explanations) capturing the influence of each of the input features on model predictions. Our framework, AMPLIFY, leads to prediction accuracy improvements of about 10-25% over a wide range of tasks.
  arXiv  Detail & Related papers  (2023-05-19T04:46:04Z)
• Active Learning for Abstractive Text Summarization [50.79416783266641]
  We propose the first effective query strategy for Active Learning in abstractive text summarization. We show that using our strategy in AL annotation helps to improve the model performance in terms of ROUGE and consistency scores.
  arXiv  Detail & Related papers  (2023-01-09T10:33:14Z)
• Schema-aware Reference as Prompt Improves Data-Efficient Knowledge Graph Construction [57.854498238624366]
  We propose a retrieval-augmented approach, which retrieves schema-aware Reference As Prompt (RAP) for data-efficient knowledge graph construction. RAP can dynamically leverage schema and knowledge inherited from human-annotated and weak-supervised data as a prompt for each sample.
  arXiv  Detail & Related papers  (2022-10-19T16:40:28Z)
• A Dependency Syntactic Knowledge Augmented Interactive Architecture for End-to-End Aspect-based Sentiment Analysis [73.74885246830611]
  We propose a novel dependency syntactic knowledge augmented interactive architecture with multi-task learning for end-to-end ABSA. This model is capable of fully exploiting the syntactic knowledge (dependency relations and types) by leveraging a well-designed Dependency Relation Embedded Graph Convolutional Network (DreGcn) Extensive experimental results on three benchmark datasets demonstrate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2020-04-04T14:59:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.