Helpfulness and Fairness of Task-Oriented Dialogue Systems
- URL: http://arxiv.org/abs/2205.12554v3
- Date: Sun, 7 May 2023 08:39:00 GMT
- Title: Helpfulness and Fairness of Task-Oriented Dialogue Systems
- Authors: Jiao Sun, Yu Hou, Jiin Kim and Nanyun Peng
- Abstract summary: We study computational measurements of helpfulness of goal-oriented dialogue systems.
We propose to use the helpfulness level of a dialogue system towards different user queries to measure the fairness of a dialogue system.
- Score: 35.135740285082356
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Goal-oriented dialogue systems aim to help users achieve certain goals.
Therefore, how humans perceive their helpfulness is important. However, neither
the human-perceived helpfulness of goal-oriented dialogue systems nor its
fairness implication has been well studied. In this paper, we study
computational measurements of helpfulness. We first formally define a dialogue
response as helpful if it is relevant & coherent, useful, and informative to a
query. Then, we collect human annotations for the helpfulness of dialogue
responses based on our definition and build a classifier to automatically
determine the helpfulness of a response. We further propose to use the
helpfulness level of a dialogue system towards different user queries to
measure the fairness of a dialogue system. Experiments with state-of-the-art
dialogue systems under three information-seeking scenarios reveal that existing
systems tend to be more helpful for questions regarding concepts from
highly-developed countries than less-developed countries, uncovering potential
fairness concerns underlying the current goal-oriented dialogue systems.
Related papers
- PK-Chat: Pointer Network Guided Knowledge Driven Generative Dialogue
Model [79.64376762489164]
PK-Chat is a Pointer network guided generative dialogue model, incorporating a unified pretrained language model and a pointer network over knowledge graphs.
The words generated by PK-Chat in the dialogue are derived from the prediction of word lists and the direct prediction of the external knowledge graph knowledge.
Based on the PK-Chat, a dialogue system is built for academic scenarios in the case of geosciences.
arXiv Detail & Related papers (2023-04-02T18:23:13Z) - PK-ICR: Persona-Knowledge Interactive Context Retrieval for Grounded Dialogue [21.266410719325208]
Persona and Knowledge Dual Context Identification is a task to identify persona and knowledge jointly for a given dialogue.
We develop a novel grounding retrieval method that utilizes all contexts of dialogue simultaneously.
arXiv Detail & Related papers (2023-02-13T20:27:26Z) - Target-Guided Dialogue Response Generation Using Commonsense and Data
Augmentation [32.764356638437214]
We introduce a new technique for target-guided response generation.
We also propose techniques to re-purpose existing dialogue datasets for target-guided generation.
Our work generally enables dialogue system designers to exercise more control over the conversations that their systems produce.
arXiv Detail & Related papers (2022-05-19T04:01:40Z) - Towards Large-Scale Interpretable Knowledge Graph Reasoning for Dialogue
Systems [109.16553492049441]
We propose a novel method to incorporate the knowledge reasoning capability into dialogue systems in a more scalable and generalizable manner.
To the best of our knowledge, this is the first work to have transformer models generate responses by reasoning over differentiable knowledge graphs.
arXiv Detail & Related papers (2022-03-20T17:51:49Z) - Task-oriented Dialogue Systems: performance vs. quality-optima, a review [0.0]
State-of-the-art task-oriented dialogue systems are not yet reaching their full potential.
Other conversational quality attributes that may point to the success, or otherwise, of the dialogue, may be ignored.
This paper explores the literature on evaluative frameworks of dialogue systems and the role of conversational quality attributes in dialogue systems.
arXiv Detail & Related papers (2021-12-21T13:16:24Z) - A Review of Dialogue Systems: From Trained Monkeys to Stochastic Parrots [0.0]
We aim to deploy artificial intelligence to build automated dialogue agents that can converse with humans.
We present a broad overview of methods developed to build dialogue systems over the years.
arXiv Detail & Related papers (2021-11-02T08:07:55Z) - Natural Language Understanding for Argumentative Dialogue Systems in the
Opinion Building Domain [6.951113351928047]
This paper introduces a framework for argumentative dialogue systems in the information-seeking domain.
Our approach distinguishes multiple user intents and identifies system arguments the user refers to in his or her natural language utterances.
arXiv Detail & Related papers (2021-03-03T21:17:24Z) - Rethinking Dialogue State Tracking with Reasoning [76.0991910623001]
This paper proposes to track dialogue states gradually with reasoning over dialogue turns with the help of the back-end data.
Empirical results demonstrate that our method significantly outperforms the state-of-the-art methods by 38.6% in terms of joint belief accuracy for MultiWOZ 2.1.
arXiv Detail & Related papers (2020-05-27T02:05:33Z) - You Impress Me: Dialogue Generation via Mutual Persona Perception [62.89449096369027]
The research in cognitive science suggests that understanding is an essential signal for a high-quality chit-chat conversation.
Motivated by this, we propose P2 Bot, a transmitter-receiver based framework with the aim of explicitly modeling understanding.
arXiv Detail & Related papers (2020-04-11T12:51:07Z) - Recent Advances and Challenges in Task-oriented Dialog System [63.82055978899631]
Task-oriented dialog systems are attracting more and more attention in academic and industrial communities.
We discuss three critical topics for task-oriented dialog systems: (1) improving data efficiency to facilitate dialog modeling in low-resource settings, (2) modeling multi-turn dynamics for dialog policy learning, and (3) integrating domain knowledge into the dialog model.
arXiv Detail & Related papers (2020-03-17T01:34:56Z) - Attention over Parameters for Dialogue Systems [69.48852519856331]
We learn a dialogue system that independently parameterizes different dialogue skills, and learns to select and combine each of them through Attention over Parameters (AoP)
The experimental results show that this approach achieves competitive performance on a combined dataset of MultiWOZ, In-Car Assistant, and Persona-Chat.
arXiv Detail & Related papers (2020-01-07T03:10:42Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.