Towards Zero-Label Language Learning

• URL: http://arxiv.org/abs/2109.09193v1
• Date: Sun, 19 Sep 2021 19:00:07 GMT
• Title: Towards Zero-Label Language Learning
• Authors: Zirui Wang, Adams Wei Yu, Orhan Firat, Yuan Cao
• Abstract summary: This paper explores zero-label learning in Natural Language Processing (NLP) No human-annotated data is used anywhere during training and models are trained purely on synthetic data. Inspired by the recent success of few-shot inference on GPT-3, we present a training data creation procedure named Unsupervised Data Generation.
• Score: 20.28186484098947
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper explores zero-label learning in Natural Language Processing (NLP), whereby no human-annotated data is used anywhere during training and models are trained purely on synthetic data. At the core of our framework is a novel approach for better leveraging the powerful pretrained language models. Specifically, inspired by the recent success of few-shot inference on GPT-3, we present a training data creation procedure named Unsupervised Data Generation (UDG), which leverages few-shot prompts to synthesize high-quality training data without real human annotations. Our method enables zero-label learning as we train task-specific models solely on the synthetic data, yet we achieve better or comparable results from strong baseline models trained on human-labeled data. Furthermore, when mixed with labeled data, our approach serves as a highly effective data augmentation procedure, achieving new state-of-the-art results on the SuperGLUE benchmark.

Related papers

• Forewarned is Forearmed: Leveraging LLMs for Data Synthesis through Failure-Inducing Exploration [90.41908331897639]
  Large language models (LLMs) have significantly benefited from training on diverse, high-quality task-specific data. We present a novel approach, ReverseGen, designed to automatically generate effective training samples.
  arXiv  Detail & Related papers  (2024-10-22T06:43:28Z)
• Co-training for Low Resource Scientific Natural Language Inference [65.37685198688538]
  We propose a novel co-training method that assigns weights based on the training dynamics of the classifiers to the distantly supervised labels. By assigning importance weights instead of filtering out examples based on an arbitrary threshold on the predicted confidence, we maximize the usage of automatically labeled data. The proposed method obtains an improvement of 1.5% in Macro F1 over the distant supervision baseline, and substantial improvements over several other strong SSL baselines.
  arXiv  Detail & Related papers  (2024-06-20T18:35:47Z)
• Harnessing large-language models to generate private synthetic text [18.863579044812703]
  Differentially private training algorithms like DP-SGD protect sensitive training data by ensuring that trained models do not reveal private information. This paper studies an alternative approach to generate synthetic data that is differentially private with respect to the original data, and then non-privately training a model on the synthetic data. generating private synthetic data is much harder than training a private model.
  arXiv  Detail & Related papers  (2023-06-02T16:59:36Z)
• ReGen: Zero-Shot Text Classification via Training Data Generation with Progressive Dense Retrieval [22.882301169283323]
  We propose a retrieval-enhanced framework to create training data from a general-domain unlabeled corpus. Experiments on nine datasets demonstrate that REGEN achieves 4.3% gain over the strongest baselines and saves around 70% of the time compared to baselines using large NLG models.
  arXiv  Detail & Related papers  (2023-05-18T04:30:09Z)
• Curriculum-Based Self-Training Makes Better Few-Shot Learners for Data-to-Text Generation [56.98033565736974]
  We propose Curriculum-Based Self-Training (CBST) to leverage unlabeled data in a rearranged order determined by the difficulty of text generation. Our method can outperform fine-tuning and task-adaptive pre-training methods, and achieve state-of-the-art performance in the few-shot setting of data-to-text generation.
  arXiv  Detail & Related papers  (2022-06-06T16:11:58Z)
• ZeroGen$^+$: Self-Guided High-Quality Data Generation in Efficient Zero-Shot Learning [97.2907428983142]
  ZeroGen attempts to purely use PLM to generate data and train a tiny model without relying on task-specific annotation. We propose a noise-robust bi-level re-weighting framework which is able to learn the per-sample weights measuring the data quality without requiring any gold data.
  arXiv  Detail & Related papers  (2022-05-25T11:38:48Z)
• DAGA: Data Augmentation with a Generation Approach for Low-resource Tagging Tasks [88.62288327934499]
  We propose a novel augmentation method with language models trained on the linearized labeled sentences. Our method is applicable to both supervised and semi-supervised settings.
  arXiv  Detail & Related papers  (2020-11-03T07:49:15Z)
• Omni-supervised Facial Expression Recognition via Distilled Data [120.11782405714234]
  We propose omni-supervised learning to exploit reliable samples in a large amount of unlabeled data for network training. We experimentally verify that the new dataset can significantly improve the ability of the learned FER model. To tackle this, we propose to apply a dataset distillation strategy to compress the created dataset into several informative class-wise images.
  arXiv  Detail & Related papers  (2020-05-18T09:36:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.