Towards Automatic Evaluation of Task-Oriented Dialogue Flows

• URL: http://arxiv.org/abs/2411.10416v1
• Date: Fri, 15 Nov 2024 18:35:00 GMT
• Title: Towards Automatic Evaluation of Task-Oriented Dialogue Flows
• Authors: Mehrnoosh Mirtaheri, Nikhil Varghese, Chandra Khatri, Amol Kelkar,
• Abstract summary: We introduce FuDGE (Fuzzy Dialogue-Graph Edit Distance), a novel metric for evaluating the quality of dialogue flows. FuDGE measures how well individual conversations align with a flow and, consequently, how well a set of conversations is represented by the flow overall. By standardizing and optimizing dialogue flows, FuDGE enables conversational designers and automated techniques to achieve higher levels of efficiency and automation.
• Score: 5.146847146797646
• License:
• Abstract: Task-oriented dialogue systems rely on predefined conversation schemes (dialogue flows) often represented as directed acyclic graphs. These flows can be manually designed or automatically generated from previously recorded conversations. Due to variations in domain expertise or reliance on different sets of prior conversations, these dialogue flows can manifest in significantly different graph structures. Despite their importance, there is no standard method for evaluating the quality of dialogue flows. We introduce FuDGE (Fuzzy Dialogue-Graph Edit Distance), a novel metric that evaluates dialogue flows by assessing their structural complexity and representational coverage of the conversation data. FuDGE measures how well individual conversations align with a flow and, consequently, how well a set of conversations is represented by the flow overall. Through extensive experiments on manually configured flows and flows generated by automated techniques, we demonstrate the effectiveness of FuDGE and its evaluation framework. By standardizing and optimizing dialogue flows, FuDGE enables conversational designers and automated techniques to achieve higher levels of efficiency and automation.

Related papers

• Dialog2Flow: Pre-training Soft-Contrastive Action-Driven Sentence Embeddings for Automatic Dialog Flow Extraction [0.0]
  We introduce Dialog2Flow embeddings, which group dialogs according to their communicative and informative functions. By clustering D2F embeddings, the latent space is quantized, and dialogs can be converted into sequences of region/action IDs. We show that D2F yields superior qualitative and quantitative results across diverse domains.
  arXiv  Detail & Related papers  (2024-10-24T07:10:18Z)
• Unsupervised Extraction of Dialogue Policies from Conversations [3.102576158218633]
  We show how Large Language Models can be instrumental in extracting dialogue policies from datasets. We then propose a novel method for generating dialogue policies utilizing a controllable and interpretable graph-based methodology.
  arXiv  Detail & Related papers  (2024-06-21T14:57:25Z)
• Unsupervised Flow Discovery from Task-oriented Dialogues [0.988655456942026]
  We propose an approach for the unsupervised discovery of flows from dialogue history. We present concrete examples of flows, discovered from MultiWOZ, a public TOD dataset.
  arXiv  Detail & Related papers  (2024-05-02T15:54:36Z)
• TOD-Flow: Modeling the Structure of Task-Oriented Dialogues [77.15457469745364]
  We propose a novel approach focusing on inferring the TOD-Flow graph from dialogue data annotated with dialog acts. The inferred TOD-Flow graph can be easily integrated with any dialogue model to improve its prediction performance, transparency, and controllability.
  arXiv  Detail & Related papers  (2023-12-07T20:06:23Z)
• CTRLStruct: Dialogue Structure Learning for Open-Domain Response Generation [38.60073402817218]
  Well-structured topic flow can leverage background information and predict future topics to help generate controllable and explainable responses. We present a new framework for dialogue structure learning to effectively explore topic-level dialogue clusters as well as their transitions with unlabelled information. Experiments on two popular open-domain dialogue datasets show our model can generate more coherent responses compared to some excellent dialogue models.
  arXiv  Detail & Related papers  (2023-03-02T09:27:11Z)
• Manual-Guided Dialogue for Flexible Conversational Agents [84.46598430403886]
  How to build and use dialogue data efficiently, and how to deploy models in different domains at scale can be critical issues in building a task-oriented dialogue system. We propose a novel manual-guided dialogue scheme, where the agent learns the tasks from both dialogue and manuals. Our proposed scheme reduces the dependence of dialogue models on fine-grained domain ontology, and makes them more flexible to adapt to various domains.
  arXiv  Detail & Related papers  (2022-08-16T08:21:12Z)
• GODEL: Large-Scale Pre-Training for Goal-Directed Dialog [119.1397031992088]
  We introduce GODEL, a large pre-trained language model for dialog. We show that GODEL outperforms state-of-the-art pre-trained dialog models in few-shot fine-tuning setups. A novel feature of our evaluation methodology is the introduction of a notion of utility that assesses the usefulness of responses.
  arXiv  Detail & Related papers  (2022-06-22T18:19:32Z)
• Structure Extraction in Task-Oriented Dialogues with Slot Clustering [94.27806592467537]
  In task-oriented dialogues, dialogue structure has often been considered as transition graphs among dialogue states. We propose a simple yet effective approach for structure extraction in task-oriented dialogues.
  arXiv  Detail & Related papers  (2022-02-28T20:18:12Z)
• FlowEval: A Consensus-Based Dialogue Evaluation Framework Using Segment Act Flows [63.116280145770006]
  We propose segment act, an extension of dialog act from utterance level to segment level, and crowdsource a large-scale dataset for it. To utilize segment act flows, sequences of segment acts, for evaluation, we develop the first consensus-based dialogue evaluation framework, FlowEval.
  arXiv  Detail & Related papers  (2022-02-14T11:37:20Z)
• Learning an Unreferenced Metric for Online Dialogue Evaluation [53.38078951628143]
  We propose an unreferenced automated evaluation metric that uses large pre-trained language models to extract latent representations of utterances. We show that our model achieves higher correlation with human annotations in an online setting, while not requiring true responses for comparison during inference.
  arXiv  Detail & Related papers  (2020-05-01T20:01:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.