Efficient slot labelling

• URL: http://arxiv.org/abs/2401.09343v2
• Date: Fri, 19 Jan 2024 13:33:22 GMT
• Title: Efficient slot labelling
• Authors: Vladimir Vlasov
• Abstract summary: Slot labelling is an essential component of any dialogue system, aiming to find important arguments in every user turn. Common approaches involve large pre-trained language models (PLMs) like BERT or RoBERTa, but they face challenges such as high computational requirements and dependence on pre-training data. We propose a lightweight method which performs on par or better than the state-of-the-art PLM-based methods, while having almost 10x less trainable parameters.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Slot labelling is an essential component of any dialogue system, aiming to find important arguments in every user turn. Common approaches involve large pre-trained language models (PLMs) like BERT or RoBERTa, but they face challenges such as high computational requirements and dependence on pre-training data. In this work, we propose a lightweight method which performs on par or better than the state-of-the-art PLM-based methods, while having almost 10x less trainable parameters. This makes it especially applicable for real-life industry scenarios.

Related papers

• MoLEx: Mixture of Layer Experts for Finetuning with Sparse Upcycling [2.1605931466490795]
  Large-scale pre-training of deep models, followed by fine-tuning them, has become the cornerstone of natural language processing (NLP) In this paper, we study layers as extractors of different types of linguistic information that are valuable when used in conjunction. We propose the Mixture of Layer Experts (MoLEx), a novel sparse mixture of experts whose experts are layers in the pre-trained model.
  arXiv  Detail & Related papers  (2025-03-14T07:22:07Z)
• Generative Context Distillation [48.91617280112579]
  Generative Context Distillation (GCD) is a lightweight prompt internalization method that employs a joint training approach. We demonstrate that our approach effectively internalizes complex prompts across various agent-based application scenarios.
  arXiv  Detail & Related papers  (2024-11-24T17:32:20Z)
• Q-SFT: Q-Learning for Language Models via Supervised Fine-Tuning [62.984693936073974]
  Value-based reinforcement learning can learn effective policies for a wide range of multi-turn problems. Current value-based RL methods have proven particularly challenging to scale to the setting of large language models. We propose a novel offline RL algorithm that addresses these drawbacks, casting Q-learning as a modified supervised fine-tuning problem.
  arXiv  Detail & Related papers  (2024-11-07T21:36:52Z)
• Denoising Pre-Training and Customized Prompt Learning for Efficient Multi-Behavior Sequential Recommendation [69.60321475454843]
  We propose DPCPL, the first pre-training and prompt-tuning paradigm tailored for Multi-Behavior Sequential Recommendation. In the pre-training stage, we propose a novel Efficient Behavior Miner (EBM) to filter out the noise at multiple time scales. Subsequently, we propose to tune the pre-trained model in a highly efficient manner with the proposed Customized Prompt Learning (CPL) module.
  arXiv  Detail & Related papers  (2024-08-21T06:48:38Z)
• Personalized LoRA for Human-Centered Text Understanding [15.704545145736676]
  We introduce personalized LoRA (PLoRA) with a plug-and-play framework for the HCTU task. PLoRA is effective, parameter-efficient and dynamically deploying in PLMs. Experiments conducted on four benchmark datasets show that the proposed method outperforms existing methods in full/few/zero-shot learning scenarios.
  arXiv  Detail & Related papers  (2024-03-10T13:04:54Z)
• Making Large Language Models Better Data Creators [22.0882632635255]
  Large language models (LLMs) have advanced the state-of-the-art in NLP significantly. deploying them for downstream applications is still challenging due to cost, responsiveness, control, or concerns around privacy and security. We propose a unified data creation pipeline that requires only a single format example.
  arXiv  Detail & Related papers  (2023-10-31T01:08:34Z)
• eP-ALM: Efficient Perceptual Augmentation of Language Models [70.47962271121389]
  We propose to direct effort to efficient adaptations of existing models, and propose to augment Language Models with perception. Existing approaches for adapting pretrained models for vision-language tasks still rely on several key components that hinder their efficiency. We show that by freezing more than 99% of total parameters, training only one linear projection layer, and prepending only one trainable token, our approach (dubbed eP-ALM) significantly outperforms other baselines on VQA and Captioning.
  arXiv  Detail & Related papers  (2023-03-20T19:20:34Z)
• Instance-wise Prompt Tuning for Pretrained Language Models [72.74916121511662]
  Instance-wise Prompt Tuning (IPT) is the first prompt learning paradigm that injects knowledge from the input data instances to the prompts. IPT significantly outperforms task-based prompt learning methods, and achieves comparable performance to conventional finetuning with only 0.5% - 1.5% of tuned parameters.
  arXiv  Detail & Related papers  (2022-06-04T10:08:50Z)
• Parameter-Efficient Sparsity for Large Language Models Fine-Tuning [63.321205487234074]
  We propose a. sparse-efficient Sparse Training (PST) method to reduce the number of trainable parameters during sparse-aware training. Experiments with diverse networks (i.e., BERT, RoBERTa and GPT-2) demonstrate PST performs on par or better than previous sparsity methods.
  arXiv  Detail & Related papers  (2022-05-23T02:43:45Z)
• Adaptive Self-training for Few-shot Neural Sequence Labeling [55.43109437200101]
  We develop techniques to address the label scarcity challenge for neural sequence labeling models. Self-training serves as an effective mechanism to learn from large amounts of unlabeled data. meta-learning helps in adaptive sample re-weighting to mitigate error propagation from noisy pseudo-labels.
  arXiv  Detail & Related papers  (2020-10-07T22:29:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.