Noised Consistency Training for Text Summarization
- URL: http://arxiv.org/abs/2105.13635v1
- Date: Fri, 28 May 2021 07:21:39 GMT
- Title: Noised Consistency Training for Text Summarization
- Authors: Junnan Liu, Qianren Mao, Bang Liu, Hao Peng, Hongdong Zhu, Jianxin Li
- Abstract summary: We argue that consistency training can be overcome by a semi-supervised approach.
We have verified that leveraging large amounts of unlabeled data decently improves the performance of supervised learning over an insufficient labeled dataset.
- Score: 23.16890559954038
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Neural abstractive summarization methods often require large quantities of
labeled training data. However, labeling large amounts of summarization data is
often prohibitive due to time, financial, and expertise constraints, which has
limited the usefulness of summarization systems to practical applications. In
this paper, we argue that this limitation can be overcome by a semi-supervised
approach: consistency training which is to leverage large amounts of unlabeled
data to improve the performance of supervised learning over a small corpus. The
consistency regularization semi-supervised learning can regularize model
predictions to be invariant to small noise applied to input articles. By adding
noised unlabeled corpus to help regularize consistency training, this framework
obtains comparative performance without using the full dataset. In particular,
we have verified that leveraging large amounts of unlabeled data decently
improves the performance of supervised learning over an insufficient labeled
dataset.
Related papers
- Late Stopping: Avoiding Confidently Learning from Mislabeled Examples [61.00103151680946]
We propose a new framework, Late Stopping, which leverages the intrinsic robust learning ability of DNNs through a prolonged training process.
We empirically observe that mislabeled and clean examples exhibit differences in the number of epochs required for them to be consistently and correctly classified.
Experimental results on benchmark-simulated and real-world noisy datasets demonstrate that the proposed method outperforms state-of-the-art counterparts.
arXiv Detail & Related papers (2023-08-26T12:43:25Z) - Soft Curriculum for Learning Conditional GANs with Noisy-Labeled and
Uncurated Unlabeled Data [70.25049762295193]
We introduce a novel conditional image generation framework that accepts noisy-labeled and uncurated data during training.
We propose soft curriculum learning, which assigns instance-wise weights for adversarial training while assigning new labels for unlabeled data.
Our experiments show that our approach outperforms existing semi-supervised and label-noise robust methods in terms of both quantitative and qualitative performance.
arXiv Detail & Related papers (2023-07-17T08:31:59Z) - SoftMatch: Addressing the Quantity-Quality Trade-off in Semi-supervised
Learning [101.86916775218403]
This paper revisits the popular pseudo-labeling methods via a unified sample weighting formulation.
We propose SoftMatch to overcome the trade-off by maintaining both high quantity and high quality of pseudo-labels during training.
In experiments, SoftMatch shows substantial improvements across a wide variety of benchmarks, including image, text, and imbalanced classification.
arXiv Detail & Related papers (2023-01-26T03:53:25Z) - Controller-Guided Partial Label Consistency Regularization with
Unlabeled Data [49.24911720809604]
We propose a controller-guided consistency regularization at both the label-level and representation-level.
We dynamically adjust the confidence thresholds so that the number of samples of each class participating in consistency regularization remains roughly equal to alleviate the problem of class-imbalance.
arXiv Detail & Related papers (2022-10-20T12:15:13Z) - An Embarrassingly Simple Approach to Semi-Supervised Few-Shot Learning [58.59343434538218]
We propose a simple but quite effective approach to predict accurate negative pseudo-labels of unlabeled data from an indirect learning perspective.
Our approach can be implemented in just few lines of code by only using off-the-shelf operations.
arXiv Detail & Related papers (2022-09-28T02:11:34Z) - Contrastive Regularization for Semi-Supervised Learning [46.020125061295886]
We propose contrastive regularization to improve both efficiency and accuracy of the consistency regularization by well-clustered features of unlabeled data.
Our method also shows robust performance on open-set semi-supervised learning where unlabeled data includes out-of-distribution samples.
arXiv Detail & Related papers (2022-01-17T07:20:11Z) - Self-Training with Weak Supervision [32.68342091430266]
State-of-the-art deep neural networks require large-scale labeled training data that is often expensive to obtain or not available for many tasks.
weak supervision in the form of domain-specific rules has been shown to be useful in such settings.
We develop a weak supervision framework (ASTRA) that leverages all the available data for a given task.
arXiv Detail & Related papers (2021-04-12T14:45:04Z) - Learning to Count in the Crowd from Limited Labeled Data [109.2954525909007]
We focus on reducing the annotation efforts by learning to count in the crowd from limited number of labeled samples.
Specifically, we propose a Gaussian Process-based iterative learning mechanism that involves estimation of pseudo-ground truth for the unlabeled data.
arXiv Detail & Related papers (2020-07-07T04:17:01Z) - Learning from Imperfect Annotations [15.306536555936692]
Many machine learning systems today are trained on large amounts of human-annotated data.
We propose a new end-to-end framework that enables us to merge the aggregation step with model training.
We show accuracy gains of up to 25% over the current state-of-the-art approaches for aggregating annotations.
arXiv Detail & Related papers (2020-04-07T15:21:08Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.