A Comparative Study on Language Models for Task-Oriented Dialogue Systems

• URL: http://arxiv.org/abs/2201.08687v1
• Date: Fri, 21 Jan 2022 13:24:25 GMT
• Title: A Comparative Study on Language Models for Task-Oriented Dialogue Systems
• Authors: Vinsen Marselino Andreas, Genta Indra Winata, Ayu Purwarianti
• Abstract summary: In task-oriented dialogue (ToD) systems, language models can be used for end-to-end training. BART and T5 outperform GPT-based models in BLEU and F1 scores and achieve state-of-the-art performance in a ToD system.
• Score: 14.634286037008017
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The recent development of language models has shown promising results by achieving state-of-the-art performance on various natural language tasks by fine-tuning pretrained models. In task-oriented dialogue (ToD) systems, language models can be used for end-to-end training without relying on dialogue state tracking to track the dialogue history but allowing the language models to generate responses according to the context given as input. This paper conducts a comparative study to show the effectiveness and strength of using recent pretrained models for fine-tuning, such as BART and T5, on endto-end ToD systems. The experimental results show substantial performance improvements after language model fine-tuning. The models produce more fluent responses after adding knowledge to the context that guides the model to avoid hallucination and generate accurate entities in the generated responses. Furthermore, we found that BART and T5 outperform GPT-based models in BLEU and F1 scores and achieve state-of-the-art performance in a ToD system.

Related papers

• Phonetic Enhanced Language Modeling for Text-to-Speech Synthesis [33.909582975045545]
  We propose a phonetic enhanced language modeling method to improve the performance of TTS models. We leverage self-supervised representations that are phonetically rich as the training target for the autoregressive language model.
  arXiv  Detail & Related papers  (2024-06-04T06:43:34Z)
• Stabilized In-Context Learning with Pre-trained Language Models for Few Shot Dialogue State Tracking [57.92608483099916]
  Large pre-trained language models (PLMs) have shown impressive unaided performance across many NLP tasks. For more complex tasks such as dialogue state tracking (DST), designing prompts that reliably convey the desired intent is nontrivial. We introduce a saliency model to limit dialogue text length, allowing us to include more exemplars per query.
  arXiv  Detail & Related papers  (2023-02-12T15:05:10Z)
• Few-shot Subgoal Planning with Language Models [58.11102061150875]
  We show that language priors encoded in pre-trained language models allow us to infer fine-grained subgoal sequences. In contrast to recent methods which make strong assumptions about subgoal supervision, our experiments show that language models can infer detailed subgoal sequences without any fine-tuning.
  arXiv  Detail & Related papers  (2022-05-28T01:03:30Z)
• PaLM: Scaling Language Modeling with Pathways [180.69584031908113]
  We trained a 540-billion parameter, densely activated, Transformer language model, which we call Pathways Language Model PaLM. We trained PaLM on 6144 TPU v4 chips using Pathways, a new ML system which enables highly efficient training across multiple TPU Pods. We demonstrate continued benefits of scaling by achieving state-of-the-art few-shot learning results on hundreds of language understanding and generation benchmarks.
  arXiv  Detail & Related papers  (2022-04-05T16:11:45Z)
• ANNA: Enhanced Language Representation for Question Answering [5.713808202873983]
  We show how approaches affect performance individually and that the approaches are jointly considered in pre-training models. We propose an extended pre-training task, and a new neighbor-aware mechanism that attends neighboring tokens more to capture the richness of context for pre-training language modeling. Our best model achieves new state-of-the-art results of 95.7% F1 and 90.6% EM on SQuAD 1.1 and also outperforms existing pre-trained language models such as RoBERTa, ALBERT, ELECTRA, and XLNet.
  arXiv  Detail & Related papers  (2022-03-28T05:26:52Z)
• Interpreting Language Models Through Knowledge Graph Extraction [42.97929497661778]
  We compare BERT-based language models through snapshots of acquired knowledge at sequential stages of the training process. We present a methodology to unveil a knowledge acquisition timeline by generating knowledge graph extracts from cloze "fill-in-the-blank" statements. We extend this analysis to a comparison of pretrained variations of BERT models (DistilBERT, BERT-base, RoBERTa)
  arXiv  Detail & Related papers  (2021-11-16T15:18:01Z)
• Language Models are Few-shot Multilingual Learners [66.11011385895195]
  We evaluate the multilingual skills of the GPT and T5 models in conducting multi-class classification on non-English languages. We show that, given a few English examples as context, pre-trained language models can predict not only English test samples but also non-English ones.
  arXiv  Detail & Related papers  (2021-09-16T03:08:22Z)
• Unsupervised Paraphrasing with Pretrained Language Models [85.03373221588707]
  We propose a training pipeline that enables pre-trained language models to generate high-quality paraphrases in an unsupervised setting. Our recipe consists of task-adaptation, self-supervision, and a novel decoding algorithm named Dynamic Blocking. We show with automatic and human evaluations that our approach achieves state-of-the-art performance on both the Quora Question Pair and the ParaNMT datasets.
  arXiv  Detail & Related papers  (2020-10-24T11:55:28Z)
• Comparative Study of Language Models on Cross-Domain Data with Model Agnostic Explainability [0.0]
  The study compares the state-of-the-art language models - BERT, ELECTRA and its derivatives which include RoBERTa, ALBERT and DistilBERT. The experimental results establish new state-of-the-art for 2013 rating classification task and Financial Phrasebank sentiment detection task with 69% accuracy and 88.2% accuracy respectively.
  arXiv  Detail & Related papers  (2020-09-09T04:31:44Z)
• Labeling Explicit Discourse Relations using Pre-trained Language Models [0.0]
  State-of-the-art models achieve slightly above 45% of F-score by using hand-crafted features. We find that the pre-trained language models, when finetuned, are powerful enough to replace the linguistic features. This is the first time when a model outperforms the knowledge intensive models without employing any linguistic features.
  arXiv  Detail & Related papers  (2020-06-21T17:18:01Z)
• A Tailored Pre-Training Model for Task-Oriented Dialog Generation [60.05269529832447]
  We propose a Pre-trained Role Alternating Language model (PRAL) for task-oriented conversational systems. We introduce a task-oriented dialog pretraining dataset by cleaning 13 existing data sets. The results show that PRAL performs better or on par with state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-04-24T09:25:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.