Active Learning for Abstractive Text Summarization
- URL: http://arxiv.org/abs/2301.03252v1
- Date: Mon, 9 Jan 2023 10:33:14 GMT
- Title: Active Learning for Abstractive Text Summarization
- Authors: Akim Tsvigun, Ivan Lysenko, Danila Sedashov, Ivan Lazichny, Eldar
Damirov, Vladimir Karlov, Artemy Belousov, Leonid Sanochkin, Maxim Panov,
Alexander Panchenko, Mikhail Burtsev, Artem Shelmanov
- Abstract summary: We propose the first effective query strategy for Active Learning in abstractive text summarization.
We show that using our strategy in AL annotation helps to improve the model performance in terms of ROUGE and consistency scores.
- Score: 50.79416783266641
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Construction of human-curated annotated datasets for abstractive text
summarization (ATS) is very time-consuming and expensive because creating each
instance requires a human annotator to read a long document and compose a
shorter summary that would preserve the key information relayed by the original
document. Active Learning (AL) is a technique developed to reduce the amount of
annotation required to achieve a certain level of machine learning model
performance. In information extraction and text classification, AL can reduce
the amount of labor up to multiple times. Despite its potential for aiding
expensive annotation, as far as we know, there were no effective AL query
strategies for ATS. This stems from the fact that many AL strategies rely on
uncertainty estimation, while as we show in our work, uncertain instances are
usually noisy, and selecting them can degrade the model performance compared to
passive annotation. We address this problem by proposing the first effective
query strategy for AL in ATS based on diversity principles. We show that given
a certain annotation budget, using our strategy in AL annotation helps to
improve the model performance in terms of ROUGE and consistency scores.
Additionally, we analyze the effect of self-learning and show that it can
further increase the performance of the model.
Related papers
- Enhancing Text Classification through LLM-Driven Active Learning and Human Annotation [2.0411082897313984]
This study introduces a novel methodology that integrates human annotators and Large Language Models.
The proposed framework integrates human annotation with the output of LLMs, depending on the model uncertainty levels.
The empirical results show a substantial decrease in the costs associated with data annotation while either maintaining or improving model accuracy.
arXiv Detail & Related papers (2024-06-17T21:45:48Z) - CoAnnotating: Uncertainty-Guided Work Allocation between Human and Large
Language Models for Data Annotation [94.59630161324013]
We propose CoAnnotating, a novel paradigm for Human-LLM co-annotation of unstructured texts at scale.
Our empirical study shows CoAnnotating to be an effective means to allocate work from results on different datasets, with up to 21% performance improvement over random baseline.
arXiv Detail & Related papers (2023-10-24T08:56:49Z) - ALE: A Simulation-Based Active Learning Evaluation Framework for the
Parameter-Driven Comparison of Query Strategies for NLP [3.024761040393842]
Active Learning (AL) proposes promising data points to annotators they annotate next instead of a subsequent or random sample.
This method is supposed to save annotation effort while maintaining model performance.
We introduce a reproducible active learning evaluation framework for the comparative evaluation of AL strategies in NLP.
arXiv Detail & Related papers (2023-08-01T10:42:11Z) - Language models are weak learners [71.33837923104808]
We show that prompt-based large language models can operate effectively as weak learners.
We incorporate these models into a boosting approach, which can leverage the knowledge within the model to outperform traditional tree-based boosting.
Results illustrate the potential for prompt-based LLMs to function not just as few-shot learners themselves, but as components of larger machine learning pipelines.
arXiv Detail & Related papers (2023-06-25T02:39:19Z) - An Efficient Active Learning Pipeline for Legal Text Classification [2.462514989381979]
We propose a pipeline for effectively using active learning with pre-trained language models in the legal domain.
We use knowledge distillation to guide the model's embeddings to a semantically meaningful space.
Our experiments on Contract-NLI, adapted to the classification task, and LEDGAR benchmarks show that our approach outperforms standard AL strategies.
arXiv Detail & Related papers (2022-11-15T13:07:02Z) - Assisted Text Annotation Using Active Learning to Achieve High Quality
with Little Effort [9.379650501033465]
We propose a tool that enables researchers to create large, high-quality, annotated datasets with only a few manual annotations.
We combine an active learning (AL) approach with a pre-trained language model to semi-automatically identify annotation categories.
Our preliminary results show that employing AL strongly reduces the number of annotations for correct classification of even complex and subtle frames.
arXiv Detail & Related papers (2021-12-15T13:14:58Z) - Revisiting Self-Training for Few-Shot Learning of Language Model [61.173976954360334]
Unlabeled data carry rich task-relevant information, they are proven useful for few-shot learning of language model.
In this work, we revisit the self-training technique for language model fine-tuning and present a state-of-the-art prompt-based few-shot learner, SFLM.
arXiv Detail & Related papers (2021-10-04T08:51:36Z) - Active$^2$ Learning: Actively reducing redundancies in Active Learning
methods for Sequence Tagging and Machine Translation [14.030275887949147]
Active Learning (AL) strategies reduce the need for huge volumes of labeled data by iteratively selecting a small number of examples for manual annotation.
In this paper, we argue that since AL strategies choose examples independently, they may potentially select similar examples, all of which may not contribute significantly to the learning process.
Our proposed approach, Active$mathbf2$ Learning (A$mathbf2$L), actively adapts to the deep learning model being trained to eliminate further such redundant examples.
arXiv Detail & Related papers (2021-03-11T06:27:31Z) - Reducing Confusion in Active Learning for Part-Of-Speech Tagging [100.08742107682264]
Active learning (AL) uses a data selection algorithm to select useful training samples to minimize annotation cost.
We study the problem of selecting instances which maximally reduce the confusion between particular pairs of output tags.
Our proposed AL strategy outperforms other AL strategies by a significant margin.
arXiv Detail & Related papers (2020-11-02T06:24:58Z) - Few-Shot Learning for Opinion Summarization [117.70510762845338]
Opinion summarization is the automatic creation of text reflecting subjective information expressed in multiple documents.
In this work, we show that even a handful of summaries is sufficient to bootstrap generation of the summary text.
Our approach substantially outperforms previous extractive and abstractive methods in automatic and human evaluation.
arXiv Detail & Related papers (2020-04-30T15:37:38Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.