DFEE: Interactive DataFlow Execution and Evaluation Kit

• URL: http://arxiv.org/abs/2212.08099v1
• Date: Sun, 4 Dec 2022 23:44:34 GMT
• Title: DFEE: Interactive DataFlow Execution and Evaluation Kit
• Authors: Han He, Song Feng, Daniele Bonadiman, Yi Zhang, Saab Mansour
• Abstract summary: We present DFEE, an interactive DataFlow Execution and Evaluation toolkit. We demonstrate a complex dialog task: event scheduling that involves temporal reasoning. To illustrate how to benchmark SoTA models, we propose a novel benchmark that covers more sophisticated event scheduling scenarios.
• Score: 15.437150666291457
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: DataFlow has been emerging as a new paradigm for building task-oriented chatbots due to its expressive semantic representations of the dialogue tasks. Despite the availability of a large dataset SMCalFlow and a simplified syntax, the development and evaluation of DataFlow-based chatbots remain challenging due to the system complexity and the lack of downstream toolchains. In this demonstration, we present DFEE, an interactive DataFlow Execution and Evaluation toolkit that supports execution, visualization and benchmarking of semantic parsers given dialogue input and backend database. We demonstrate the system via a complex dialog task: event scheduling that involves temporal reasoning. It also supports diagnosing the parsing results via a friendly interface that allows developers to examine dynamic DataFlow and the corresponding execution results. To illustrate how to benchmark SoTA models, we propose a novel benchmark that covers more sophisticated event scheduling scenarios and a new metric on task success evaluation. The codes of DFEE have been released on https://github.com/amazonscience/dataflow-evaluation-toolkit.

Related papers

• Unsupervised Flow Discovery from Task-oriented Dialogues [0.988655456942026]
  We propose an approach for the unsupervised discovery of flows from dialogue history. We present concrete examples of flows, discovered from MultiWOZ, a public TOD dataset.
  arXiv  Detail & Related papers  (2024-05-02T15:54:36Z)
• TOD-Flow: Modeling the Structure of Task-Oriented Dialogues [77.15457469745364]
  We propose a novel approach focusing on inferring the TOD-Flow graph from dialogue data annotated with dialog acts. The inferred TOD-Flow graph can be easily integrated with any dialogue model to improve its prediction performance, transparency, and controllability.
  arXiv  Detail & Related papers  (2023-12-07T20:06:23Z)
• Conversational Semantic Parsing using Dynamic Context Graphs [68.72121830563906]
  We consider the task of conversational semantic parsing over general purpose knowledge graphs (KGs) with millions of entities, and thousands of relation-types. We focus on models which are capable of interactively mapping user utterances into executable logical forms.
  arXiv  Detail & Related papers  (2023-05-04T16:04:41Z)
• Turning Flowchart into Dialog: Augmenting Flowchart-grounded Troubleshooting Dialogs via Synthetic Data Generation [50.06143883455979]
  Flowchart-grounded troubleshooting dialogue (FTD) systems follow the instructions of a flowchart to diagnose users' problems in specific domains. We propose a plan-based synthetic data generation approach that generates diverse synthetic dialog data at scale.
  arXiv  Detail & Related papers  (2023-05-02T11:08:27Z)
• A Closer Look at Debiased Temporal Sentence Grounding in Videos: Dataset, Metric, and Approach [53.727460222955266]
  Temporal Sentence Grounding in Videos (TSGV) aims to ground a natural language sentence in an untrimmed video. Recent studies have found that current benchmark datasets may have obvious moment annotation biases. We introduce a new evaluation metric "dR@n,IoU@m" that discounts the basic recall scores to alleviate the inflating evaluation caused by biased datasets.
  arXiv  Detail & Related papers  (2022-03-10T08:58:18Z)
• Enel: Context-Aware Dynamic Scaling of Distributed Dataflow Jobs using Graph Propagation [52.9168275057997]
  This paper presents Enel, a novel dynamic scaling approach that uses message propagation on an attributed graph to model dataflow jobs. We show that Enel is able to identify effective rescaling actions, reacting for instance to node failures, and can be reused across different execution contexts.
  arXiv  Detail & Related papers  (2021-08-27T10:21:08Z)
• Conversations Are Not Flat: Modeling the Dynamic Information Flow across Dialogue Utterances [28.255324166852535]
  Open-domain dialogue models can generate acceptable responses according to the historical context. We propose the DialoFlow model, in which we introduce a dynamic flow mechanism to model the context flow. Code and pre-trained models will be public.
  arXiv  Detail & Related papers  (2021-06-04T03:04:06Z)
• Task-Oriented Dialogue as Dataflow Synthesis [158.77123205487334]
  We describe an approach to task-oriented dialogue in which dialogue state is represented as a dataflow graph. A dialogue agent maps each user utterance to a program that extends this graph. We introduce a new dataset, SMCalFlow, featuring complex dialogues about events, weather, places, and people.
  arXiv  Detail & Related papers  (2020-09-24T00:35:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.