PSSAT: A Perturbed Semantic Structure Awareness Transferring Method for Perturbation-Robust Slot Filling

• URL: http://arxiv.org/abs/2208.11508v1
• Date: Wed, 24 Aug 2022 13:01:00 GMT
• Title: PSSAT: A Perturbed Semantic Structure Awareness Transferring Method for Perturbation-Robust Slot Filling
• Authors: Guanting Dong, Daichi Guo, Liwen Wang, Xuefeng Li, Zechen Wang, Chen Zeng, Keqing He, Jinzheng Zhao, Hao Lei, Xinyue Cui, Yi Huang, Junlan Feng, Weiran Xu
• Abstract summary: Most existing slot filling models tend to memorize inherent patterns of entities and corresponding contexts from training data. We propose a semantic awareness structure transferring method for training perturbation-robust slot filling models.
• Score: 27.602336774468
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Most existing slot filling models tend to memorize inherent patterns of entities and corresponding contexts from training data. However, these models can lead to system failure or undesirable outputs when being exposed to spoken language perturbation or variation in practice. We propose a perturbed semantic structure awareness transferring method for training perturbation-robust slot filling models. Specifically, we introduce two MLM-based training strategies to respectively learn contextual semantic structure and word distribution from unsupervised language perturbation corpus. Then, we transfer semantic knowledge learned from upstream training procedure into the original samples and filter generated data by consistency processing. These procedures aim to enhance the robustness of slot filling models. Experimental results show that our method consistently outperforms the previous basic methods and gains strong generalization while preventing the model from memorizing inherent patterns of entities and contexts.

Related papers

• Contextual Biasing to Improve Domain-specific Custom Vocabulary Audio Transcription without Explicit Fine-Tuning of Whisper Model [0.0]
  OpenAI's Whisper Automated Speech Recognition model excels in generalizing across diverse datasets and domains. We propose a method to enhance transcription accuracy without explicit fine-tuning or altering model parameters.
  arXiv  Detail & Related papers  (2024-10-24T01:58:11Z)
• Towards Robust Pruning: An Adaptive Knowledge-Retention Pruning Strategy for Language Models [35.58379464827462]
  We introduce a post-training pruning strategy designed to faithfully replicate the embedding space and feature space of dense language models. Compared to other state-of-art baselines, our approach demonstrates a superior balance between accuracy, sparsity, robustness, and pruning cost with BERT on datasets SST2, IMDB, and AGNews.
  arXiv  Detail & Related papers  (2023-10-19T23:02:29Z)
• On Conditional and Compositional Language Model Differentiable Prompting [75.76546041094436]
  Prompts have been shown to be an effective method to adapt a frozen Pretrained Language Model (PLM) to perform well on downstream tasks. We propose a new model, Prompt Production System (PRopS), which learns to transform task instructions or input metadata, into continuous prompts.
  arXiv  Detail & Related papers  (2023-07-04T02:47:42Z)
• Syntactically Robust Training on Partially-Observed Data for Open Information Extraction [25.59133746149343]
  Open Information Extraction models have shown promising results with sufficient supervision. We propose a syntactically robust training framework that enables models to be trained on a syntactic-abundant distribution.
  arXiv  Detail & Related papers  (2023-01-17T12:39:13Z)
• Autoregressive Structured Prediction with Language Models [73.11519625765301]
  We describe an approach to model structures as sequences of actions in an autoregressive manner with PLMs. Our approach achieves the new state-of-the-art on all the structured prediction tasks we looked at.
  arXiv  Detail & Related papers  (2022-10-26T13:27:26Z)
• Improving Meta-learning for Low-resource Text Classification and Generation via Memory Imitation [87.98063273826702]
  We propose a memory imitation meta-learning (MemIML) method that enhances the model's reliance on support sets for task adaptation. A theoretical analysis is provided to prove the effectiveness of our method.
  arXiv  Detail & Related papers  (2022-03-22T12:41:55Z)
• Does Pre-training Induce Systematic Inference? How Masked Language Models Acquire Commonsense Knowledge [91.15301779076187]
  We introduce verbalized knowledge into the minibatches of a BERT model during pre-training and evaluate how well the model generalizes to supported inferences. We find generalization does not improve over the course of pre-training, suggesting that commonsense knowledge is acquired from surface-level, co-occurrence patterns rather than induced, systematic reasoning.
  arXiv  Detail & Related papers  (2021-12-16T03:13:04Z)
• SLM: Learning a Discourse Language Representation with Sentence Unshuffling [53.42814722621715]
  We introduce Sentence-level Language Modeling, a new pre-training objective for learning a discourse language representation. We show that this feature of our model improves the performance of the original BERT by large margins.
  arXiv  Detail & Related papers  (2020-10-30T13:33:41Z)
• Unsupervised Paraphrasing with Pretrained Language Models [85.03373221588707]
  We propose a training pipeline that enables pre-trained language models to generate high-quality paraphrases in an unsupervised setting. Our recipe consists of task-adaptation, self-supervision, and a novel decoding algorithm named Dynamic Blocking. We show with automatic and human evaluations that our approach achieves state-of-the-art performance on both the Quora Question Pair and the ParaNMT datasets.
  arXiv  Detail & Related papers  (2020-10-24T11:55:28Z)
• Systematic Generalization on gSCAN with Language Conditioned Embedding [19.39687991647301]
  Systematic Generalization refers to a learning algorithm's ability to extrapolate learned behavior to unseen situations. We propose a novel method that learns objects' contextualized embeddings with dynamic message passing conditioned on the input natural language.
  arXiv  Detail & Related papers  (2020-09-11T17:35:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.