Improving Multi-Domain Task-Oriented Dialogue System with Offline Reinforcement Learning

• URL: http://arxiv.org/abs/2411.05340v1
• Date: Fri, 08 Nov 2024 05:43:40 GMT
• Title: Improving Multi-Domain Task-Oriented Dialogue System with Offline Reinforcement Learning
• Authors: Dharmendra Prajapat, Durga Toshniwal,
• Abstract summary: Task-oriented dialogue (TOD) system is designed to accomplish user-defined tasks through dialogues. We propose a TOD system that leverages a unified pre-trained language model, GPT2, as a base model. We show that our model increases the inform rate by 1.60% and the success rate by 3.17% compared to the baseline.
• Score: 5.18578230548982
• License:
• Abstract: Task-oriented dialogue (TOD) system is designed to accomplish user-defined tasks through dialogues. The TOD system has progressed towards end-to-end modeling by leveraging pre-trained large language models. Fine-tuning the pre-trained language models using only supervised learning leads to the exposure bias and token loss problem and it deviates the models from completing the user's task. To address these issues, we propose a TOD system that leverages a unified pre-trained language model, GPT2, as a base model. It is optimized using supervised learning and reinforcement learning (RL). The issues in the TOD system are mitigated using a non-differentiable reward function. The reward is calculated using the weighted sum of the success rate and BLEU evaluation metrics. The success rate and BLEU metrics in reward calculation guide the language model for user task completion while ensuring a coherent and fluent response. Our model is acquired by fine-tuning a pre-trained model on the dialogue-session level which comprises user utterance, belief state, system act, and system response. Experimental results on MultiWOZ2.1 demonstrate that our model increases the inform rate by 1.60% and the success rate by 3.17% compared to the baseline.

Related papers

• Learning from Relevant Subgoals in Successful Dialogs using Iterative Training for Task-oriented Dialog Systems [6.226609932118126]
  Task-oriented Dialog (ToD) systems have to solve multiple subgoals to accomplish user goals, whereas feedback is often obtained only at the end of the dialog. We propose SUIT (SUbgoal-aware ITerative Training), an iterative training approach for improving ToD systems.
  arXiv  Detail & Related papers  (2024-11-25T11:47:31Z)
• Enhancing Large Language Model Induced Task-Oriented Dialogue Systems Through Look-Forward Motivated Goals [76.69419538047813]
  ProToD approach anticipates the future dialogue actions and incorporates the goal-oriented reward signal to enhance ToD systems. We present a novel evaluation method that assesses ToD systems based on goal-driven dialogue simulations. Empirical experiments conducted on the MultiWoZ 2.1 dataset demonstrate that our model can achieve superior performance using only 10% of the data.
  arXiv  Detail & Related papers  (2023-09-16T10:56:00Z)
• Discovering Customer-Service Dialog System with Semi-Supervised Learning and Coarse-to-Fine Intent Detection [6.869753194843482]
  Task-oriented dialog aims to assist users in achieving specific goals through multi-turn conversation. We constructed a weakly supervised dataset based on a teacher/student paradigm. We also built a modular dialogue system and integrated coarse-to-fine grained classification for user intent detection.
  arXiv  Detail & Related papers  (2022-12-23T14:36:43Z)
• Few-shot Prompting Towards Controllable Response Generation [49.479958672988566]
  We first explored the combination of prompting and reinforcement learning (RL) to steer models' generation without accessing any of the models' parameters. We apply multi-task learning to make the model learn to generalize to new tasks better. Experiment results show that our proposed method can successfully control several state-of-the-art (SOTA) dialogue models without accessing their parameters.
  arXiv  Detail & Related papers  (2022-06-08T14:48:06Z)
• A Comparative Study on Language Models for Task-Oriented Dialogue Systems [14.634286037008017]
  In task-oriented dialogue (ToD) systems, language models can be used for end-to-end training. BART and T5 outperform GPT-based models in BLEU and F1 scores and achieve state-of-the-art performance in a ToD system.
  arXiv  Detail & Related papers  (2022-01-21T13:24:25Z)
• Self-training Improves Pre-training for Few-shot Learning in Task-oriented Dialog Systems [47.937191088981436]
  Large-scale pre-trained language models, have shown promising results for few-shot learning in ToD. We propose a self-training approach that iteratively labels the most confident unlabeled data to train a stronger Student model. We conduct experiments and present analyses on four downstream tasks in ToD, including intent classification, dialog state tracking, dialog act prediction, and response selection.
  arXiv  Detail & Related papers  (2021-08-28T07:22:06Z)
• Modelling Hierarchical Structure between Dialogue Policy and Natural Language Generator with Option Framework for Task-oriented Dialogue System [49.39150449455407]
  HDNO is an option framework for designing latent dialogue acts to avoid designing specific dialogue act representations. We test HDNO on MultiWoz 2.0 and MultiWoz 2.1, the datasets on multi-domain dialogues, in comparison with word-level E2E model trained with RL, LaRL and HDSA.
  arXiv  Detail & Related papers  (2020-06-11T20:55:28Z)
• SOLOIST: Building Task Bots at Scale with Transfer Learning and Machine Teaching [81.45928589522032]
  We parameterize modular task-oriented dialog systems using a Transformer-based auto-regressive language model. We pre-train, on heterogeneous dialog corpora, a task-grounded response generation model. Experiments show that SOLOIST creates new state-of-the-art on well-studied task-oriented dialog benchmarks.
  arXiv  Detail & Related papers  (2020-05-11T17:58:34Z)
• A Tailored Pre-Training Model for Task-Oriented Dialog Generation [60.05269529832447]
  We propose a Pre-trained Role Alternating Language model (PRAL) for task-oriented conversational systems. We introduce a task-oriented dialog pretraining dataset by cleaning 13 existing data sets. The results show that PRAL performs better or on par with state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-04-24T09:25:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.