Evaluate On-the-job Learning Dialogue Systems and a Case Study for Natural Language Understanding

• URL: http://arxiv.org/abs/2102.13589v1
• Date: Fri, 26 Feb 2021 16:54:16 GMT
• Title: Evaluate On-the-job Learning Dialogue Systems and a Case Study for Natural Language Understanding
• Authors: Mathilde Veron, Sophie Rosset, Olivier Galibert, Guillaume Bernard
• Abstract summary: We propose a first general methodology for evaluating on-the-job learning dialogue systems. We describe a task-oriented dialogue system which improves on-the-job its natural language component through its user interactions.
• Score: 3.557633666039596
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: On-the-job learning consists in continuously learning while being used in production, in an open environment, meaning that the system has to deal on its own with situations and elements never seen before. The kind of systems that seem to be especially adapted to on-the-job learning are dialogue systems, since they can take advantage of their interactions with users to collect feedback to adapt and improve their components over time. Some dialogue systems performing on-the-job learning have been built and evaluated but no general methodology has yet been defined. Thus in this paper, we propose a first general methodology for evaluating on-the-job learning dialogue systems. We also describe a task-oriented dialogue system which improves on-the-job its natural language component through its user interactions. We finally evaluate our system with the described methodology.

Related papers

• An Efficient Self-Learning Framework For Interactive Spoken Dialog Systems [18.829793635104608]
  We introduce a general framework for ASR in dialog systems. We show that leveraging our new framework compared to traditional training leads to relative WER reductions of close to 10% in real-world dialog systems.
  arXiv  Detail & Related papers  (2024-09-16T17:59:50Z)
• Evaluating Task-oriented Dialogue Systems: A Systematic Review of Measures, Constructs and their Operationalisations [2.6122764214161363]
  This review provides an overview of the used constructs and metrics in previous work. It also discusses challenges in the context of dialogue system evaluation. It develops a research agenda for the future of dialogue system evaluation.
  arXiv  Detail & Related papers  (2023-12-21T14:15:46Z)
• A Survey of the Evolution of Language Model-Based Dialogue Systems: Data, Task and Models [24.120097746860928]
  We take a deep look at the history of the dialogue system, especially its special relationship with the advancements of language models.<n>This survey delves into the dynamic interplay between language models and dialogue systems, unraveling the evolutionary path of this essential relationship.
  arXiv  Detail & Related papers  (2023-11-28T13:51:32Z)
• LEEETs-Dial: Linguistic Entrainment in End-to-End Task-oriented Dialogue systems [0.0]
  We introduce methods for achieving dialogue entrainment in a GPT-2-based end-to-end task-oriented dialogue system. We experiment with training instance weighting, entrainment-specific loss, and additional conditioning to generate responses that align with the user.
  arXiv  Detail & Related papers  (2023-11-15T21:35:25Z)
• User Adaptive Language Learning Chatbots with a Curriculum [55.63893493019025]
  We adapt lexically constrained decoding to a dialog system, which urges the dialog system to include curriculum-aligned words and phrases in its generated utterances. The evaluation result demonstrates that the dialog system with curriculum infusion improves students' understanding of target words and increases their interest in practicing English.
  arXiv  Detail & Related papers  (2023-04-11T20:41:41Z)
• PK-Chat: Pointer Network Guided Knowledge Driven Generative Dialogue Model [79.64376762489164]
  PK-Chat is a Pointer network guided generative dialogue model, incorporating a unified pretrained language model and a pointer network over knowledge graphs. The words generated by PK-Chat in the dialogue are derived from the prediction of word lists and the direct prediction of the external knowledge graph knowledge. Based on the PK-Chat, a dialogue system is built for academic scenarios in the case of geosciences.
  arXiv  Detail & Related papers  (2023-04-02T18:23:13Z)
• Helpfulness and Fairness of Task-Oriented Dialogue Systems [35.135740285082356]
  We study computational measurements of helpfulness of goal-oriented dialogue systems. We propose to use the helpfulness level of a dialogue system towards different user queries to measure the fairness of a dialogue system.
  arXiv  Detail & Related papers  (2022-05-25T07:58:38Z)
• A Review of Dialogue Systems: From Trained Monkeys to Stochastic Parrots [0.0]
  We aim to deploy artificial intelligence to build automated dialogue agents that can converse with humans. We present a broad overview of methods developed to build dialogue systems over the years.
  arXiv  Detail & Related papers  (2021-11-02T08:07:55Z)
• Advances in Multi-turn Dialogue Comprehension: A Survey [51.215629336320305]
  Training machines to understand natural language and interact with humans is an elusive and essential task of artificial intelligence. This paper reviews the previous methods from the technical perspective of dialogue modeling for the dialogue comprehension task. In addition, we categorize dialogue-related pre-training techniques which are employed to enhance PrLMs in dialogue scenarios.
  arXiv  Detail & Related papers  (2021-10-11T03:52:37Z)
• Advances in Multi-turn Dialogue Comprehension: A Survey [51.215629336320305]
  We review the previous methods from the perspective of dialogue modeling. We discuss three typical patterns of dialogue modeling that are widely-used in dialogue comprehension tasks.
  arXiv  Detail & Related papers  (2021-03-04T15:50:17Z)
• Continual Learning in Task-Oriented Dialogue Systems [49.35627673523519]
  Continual learning in task-oriented dialogue systems can allow us to add new domains and functionalities through time without incurring the high cost of a whole system retraining. We propose a continual learning benchmark for task-oriented dialogue systems with 37 domains to be learned continuously in four settings.
  arXiv  Detail & Related papers  (2020-12-31T08:44:25Z)
• Is Your Goal-Oriented Dialog Model Performing Really Well? Empirical Analysis of System-wise Evaluation [114.48767388174218]
  This paper presents an empirical analysis on different types of dialog systems composed of different modules in different settings. Our results show that a pipeline dialog system trained using fine-grained supervision signals at different component levels often obtains better performance than the systems that use joint or end-to-end models trained on coarse-grained labels.
  arXiv  Detail & Related papers  (2020-05-15T05:20:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.