ConVEx: Data-Efficient and Few-Shot Slot Labeling

• URL: http://arxiv.org/abs/2010.11791v2
• Date: Mon, 7 Jun 2021 16:28:03 GMT
• Title: ConVEx: Data-Efficient and Few-Shot Slot Labeling
• Abstract summary: ConVEx is an efficient pretraining and fine-tuning neural approach for slot-labeling dialog tasks. We report state-of-the-art performance of ConVEx across a range of diverse domains and data sets for dialog slot-labeling.
• Score: 72.00513667636801
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: We propose ConVEx (Conversational Value Extractor), an efficient pretraining and fine-tuning neural approach for slot-labeling dialog tasks. Instead of relying on more general pretraining objectives from prior work (e.g., language modeling, response selection), ConVEx's pretraining objective, a novel pairwise cloze task using Reddit data, is well aligned with its intended usage on sequence labeling tasks. This enables learning domain-specific slot labelers by simply fine-tuning decoding layers of the pretrained general-purpose sequence labeling model, while the majority of the pretrained model's parameters are kept frozen. We report state-of-the-art performance of ConVEx across a range of diverse domains and data sets for dialog slot-labeling, with the largest gains in the most challenging, few-shot setups. We believe that ConVEx's reduced pretraining times (i.e., only 18 hours on 12 GPUs) and cost, along with its efficient fine-tuning and strong performance, promise wider portability and scalability for data-efficient sequence-labeling tasks in general.

Related papers

• Generalization v.s. Memorization: Tracing Language Models' Capabilities Back to Pretraining Data [76.90128359866462]
  We introduce an extended concept of memorization, distributional memorization, which measures the correlation between the output probabilities and the pretraining data frequency. This study demonstrates that memorization plays a larger role in simpler, knowledge-intensive tasks, while generalization is the key for harder, reasoning-based tasks.
  arXiv  Detail & Related papers  (2024-07-20T21:24:40Z)
• Unified Pretraining for Recommendation via Task Hypergraphs [55.98773629788986]
  We propose a novel multitask pretraining framework named Unified Pretraining for Recommendation via Task Hypergraphs. For a unified learning pattern to handle diverse requirements and nuances of various pretext tasks, we design task hypergraphs to generalize pretext tasks to hyperedge prediction. A novel transitional attention layer is devised to discriminatively learn the relevance between each pretext task and recommendation.
  arXiv  Detail & Related papers  (2023-10-20T05:33:21Z)
• Never Train from Scratch: Fair Comparison of Long-Sequence Models Requires Data-Driven Priors [44.5740422079]
  We show that pretraining with standard denoising objectives leads to dramatic gains across multiple architectures. In stark contrast to prior works, we find vanilla Transformers to match the performance of S4 on Long Range Arena when properly pretrained.
  arXiv  Detail & Related papers  (2023-10-04T17:17:06Z)
• LESS: Label-Efficient Semantic Segmentation for LiDAR Point Clouds [62.49198183539889]
  We propose a label-efficient semantic segmentation pipeline for outdoor scenes with LiDAR point clouds. Our method co-designs an efficient labeling process with semi/weakly supervised learning. Our proposed method is even highly competitive compared to the fully supervised counterpart with 100% labels.
  arXiv  Detail & Related papers  (2022-10-14T19:13:36Z)
• Few-Shot Learning with Siamese Networks and Label Tuning [5.006086647446482]
  We show that with proper pre-training, Siamese Networks that embed texts and labels offer a competitive alternative. We introduce label tuning, a simple and computationally efficient approach that allows to adapt the models in a few-shot setup by only changing the label embeddings.
  arXiv  Detail & Related papers  (2022-03-28T11:16:46Z)
• ST++: Make Self-training Work Better for Semi-supervised Semantic Segmentation [23.207191521477654]
  We investigate if we could make the self-training -- a simple but popular framework -- work better for semi-supervised segmentation. We propose an advanced self-training framework (namely ST++) that performs selective re-training via selecting and prioritizing the more reliable unlabeled images. As a result, the proposed ST++ boosts the performance of semi-supervised model significantly and surpasses existing methods by a large margin on the Pascal VOC 2012 and Cityscapes benchmark.
  arXiv  Detail & Related papers  (2021-06-09T14:18:32Z)
• Pretraining Representations for Data-Efficient Reinforcement Learning [12.43475487724972]
  We use unlabeled data to pretrain an encoder which is then finetuned on a small amount of task-specific data. When limited to 100k steps of interaction on Atari games, our approach significantly surpasses prior work. Our approach shows particular promise when combined with larger models as well as more diverse, task-aligned observational data.
  arXiv  Detail & Related papers  (2021-06-09T04:14:27Z)
• Adaptive Self-training for Few-shot Neural Sequence Labeling [55.43109437200101]
  We develop techniques to address the label scarcity challenge for neural sequence labeling models. Self-training serves as an effective mechanism to learn from large amounts of unlabeled data. meta-learning helps in adaptive sample re-weighting to mitigate error propagation from noisy pseudo-labels.
  arXiv  Detail & Related papers  (2020-10-07T22:29:05Z)
• Improving Semantic Segmentation via Self-Training [75.07114899941095]
  We show that we can obtain state-of-the-art results using a semi-supervised approach, specifically a self-training paradigm. We first train a teacher model on labeled data, and then generate pseudo labels on a large set of unlabeled data. Our robust training framework can digest human-annotated and pseudo labels jointly and achieve top performances on Cityscapes, CamVid and KITTI datasets.
  arXiv  Detail & Related papers  (2020-04-30T17:09:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.