Learning with Different Amounts of Annotation: From Zero to Many Labels

• URL: http://arxiv.org/abs/2109.04408v2
• Date: Fri, 10 Sep 2021 18:01:57 GMT
• Title: Learning with Different Amounts of Annotation: From Zero to Many Labels
• Authors: Shujian Zhang, Chengyue Gong, Eunsol Choi
• Abstract summary: Training NLP systems typically assume access to annotated data that has a single human label per example. We explore new annotation distribution schemes, assigning multiple labels per example for a small subset of training examples. Introducing such multi label examples at the cost of annotating fewer examples brings clear gains on natural language inference task and entity typing task.
• Score: 19.869498599986006
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Training NLP systems typically assumes access to annotated data that has a single human label per example. Given imperfect labeling from annotators and inherent ambiguity of language, we hypothesize that single label is not sufficient to learn the spectrum of language interpretation. We explore new annotation distribution schemes, assigning multiple labels per example for a small subset of training examples. Introducing such multi label examples at the cost of annotating fewer examples brings clear gains on natural language inference task and entity typing task, even when we simply first train with a single label data and then fine tune with multi label examples. Extending a MixUp data augmentation framework, we propose a learning algorithm that can learn from training examples with different amount of annotation (with zero, one, or multiple labels). This algorithm efficiently combines signals from uneven training data and brings additional gains in low annotation budget and cross domain settings. Together, our method achieves consistent gains in two tasks, suggesting distributing labels unevenly among training examples can be beneficial for many NLP tasks.

Related papers

• Determined Multi-Label Learning via Similarity-Based Prompt [12.428779617221366]
  In multi-label classification, each training instance is associated with multiple class labels simultaneously. To alleviate this problem, a novel labeling setting termed textitDetermined Multi-Label Learning (DMLL) is proposed.
  arXiv  Detail & Related papers  (2024-03-25T07:08:01Z)
• Substituting Data Annotation with Balanced Updates and Collective Loss in Multi-label Text Classification [19.592985329023733]
  Multi-label text classification (MLTC) is the task of assigning multiple labels to a given text. We study the MLTC problem in annotation-free and scarce-annotation settings in which the magnitude of available supervision signals is linear to the number of labels. Our method follows three steps, (1) mapping input text into a set of preliminary label likelihoods by natural language inference using a pre-trained language model, (2) calculating a signed label dependency graph by label descriptions, and (3) updating the preliminary label likelihoods with message passing along the label dependency graph.
  arXiv  Detail & Related papers  (2023-09-24T04:12:52Z)
• Towards Imbalanced Large Scale Multi-label Classification with Partially Annotated Labels [8.977819892091]
  Multi-label classification is a widely encountered problem in daily life, where an instance can be associated with multiple classes. In this work, we address the issue of label imbalance and investigate how to train neural networks using partial labels.
  arXiv  Detail & Related papers  (2023-07-31T21:50:48Z)
• Robust Assignment of Labels for Active Learning with Sparse and Noisy Annotations [0.17188280334580192]
  Supervised classification algorithms are used to solve a growing number of real-life problems around the globe. Unfortunately, acquiring good-quality annotations for many tasks is infeasible or too expensive to be done in practice. We propose two novel annotation unification algorithms that utilize unlabeled parts of the sample space.
  arXiv  Detail & Related papers  (2023-07-25T19:40:41Z)
• Multi-Instance Partial-Label Learning: Towards Exploiting Dual Inexact Supervision [53.530957567507365]
  In some real-world tasks, each training sample is associated with a candidate label set that contains one ground-truth label and some false positive labels. In this paper, we formalize such problems as multi-instance partial-label learning (MIPL) Existing multi-instance learning algorithms and partial-label learning algorithms are suboptimal for solving MIPL problems.
  arXiv  Detail & Related papers  (2022-12-18T03:28:51Z)
• Learning to Imagine: Diversify Memory for Incremental Learning using Unlabeled Data [69.30452751012568]
  We develop a learnable feature generator to diversify exemplars by adaptively generating diverse counterparts of exemplars. We introduce semantic contrastive learning to enforce the generated samples to be semantic consistent with exemplars. Our method does not bring any extra inference cost and outperforms state-of-the-art methods on two benchmarks.
  arXiv  Detail & Related papers  (2022-04-19T15:15:18Z)
• Trustable Co-label Learning from Multiple Noisy Annotators [68.59187658490804]
  Supervised deep learning depends on massive accurately annotated examples. A typical alternative is learning from multiple noisy annotators. This paper proposes a data-efficient approach, called emphTrustable Co-label Learning (TCL)
  arXiv  Detail & Related papers  (2022-03-08T16:57:00Z)
• Learning with Noisy Labels by Targeted Relabeling [52.0329205268734]
  Crowdsourcing platforms are often used to collect datasets for training deep neural networks. We propose an approach which reserves a fraction of annotations to explicitly relabel highly probable labeling errors.
  arXiv  Detail & Related papers  (2021-10-15T20:37:29Z)
• Dash: Semi-Supervised Learning with Dynamic Thresholding [72.74339790209531]
  We propose a semi-supervised learning (SSL) approach that uses unlabeled examples to train models. Our proposed approach, Dash, enjoys its adaptivity in terms of unlabeled data selection.
  arXiv  Detail & Related papers  (2021-09-01T23:52:29Z)
• Adaptive Self-training for Few-shot Neural Sequence Labeling [55.43109437200101]
  We develop techniques to address the label scarcity challenge for neural sequence labeling models. Self-training serves as an effective mechanism to learn from large amounts of unlabeled data. meta-learning helps in adaptive sample re-weighting to mitigate error propagation from noisy pseudo-labels.
  arXiv  Detail & Related papers  (2020-10-07T22:29:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.