Inconsistent dialogue responses and how to recover from them
- URL: http://arxiv.org/abs/2401.10353v1
- Date: Thu, 18 Jan 2024 19:46:04 GMT
- Title: Inconsistent dialogue responses and how to recover from them
- Authors: Mian Zhang, Lifeng Jin, Linfeng Song, Haitao Mi and Dong Yu
- Abstract summary: One critical issue for chat systems is to stay consistent about preferences, opinions, beliefs and facts of itself.
In this work, we study methods to assess and bolster utterance consistency of chat systems.
- Score: 45.933921383946576
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: One critical issue for chat systems is to stay consistent about preferences,
opinions, beliefs and facts of itself, which has been shown a difficult
problem. In this work, we study methods to assess and bolster utterance
consistency of chat systems. A dataset is first developed for studying the
inconsistencies, where inconsistent dialogue responses, explanations of the
inconsistencies, and recovery utterances are authored by annotators. This
covers the life span of inconsistencies, namely introduction, understanding,
and resolution. Building on this, we introduce a set of tasks centered on
dialogue consistency, specifically focused on its detection and resolution. Our
experimental findings indicate that our dataset significantly helps the
progress in identifying and resolving conversational inconsistencies, and
current popular large language models like ChatGPT which are good at resolving
inconsistencies however still struggle with detection.
Related papers
- Evaluating Task-Oriented Dialogue Consistency through Constraint Satisfaction [1.4272411349249625]
We propose to conceptualize dialogue consistency as a Constraint Satisfaction Problem (CSP)
We utilize a CSP solver to detect inconsistencies in dialogues re-lexicalized by an LLM.
We argue that CSP captures core properties of dialogue consistency that have been poorly considered by approaches based on component pipelines.
arXiv Detail & Related papers (2024-07-16T15:38:41Z) - Red Teaming Language Models for Processing Contradictory Dialogues [30.522961490270532]
Most language models currently available are prone to self-contradiction during dialogues.
This study explores a novel contradictory dialogue processing task that aims to detect and modify contradictory statements in a conversation.
arXiv Detail & Related papers (2024-05-16T14:27:32Z) - Context Does Matter: Implications for Crowdsourced Evaluation Labels in Task-Oriented Dialogue Systems [57.16442740983528]
Crowdsourced labels play a crucial role in evaluating task-oriented dialogue systems.
Previous studies suggest using only a portion of the dialogue context in the annotation process.
This study investigates the influence of dialogue context on annotation quality.
arXiv Detail & Related papers (2024-04-15T17:56:39Z) - Evaluating Robustness of Dialogue Summarization Models in the Presence
of Naturally Occurring Variations [13.749495524988774]
We systematically investigate the impact of real-life variations on state-of-the-art dialogue summarization models.
We introduce two types of perturbations: utterance-level perturbations that modify individual utterances with errors and language variations, and dialogue-level perturbations that add non-informative exchanges.
We find that both fine-tuned and instruction-tuned models are affected by input variations, with the latter being more susceptible.
arXiv Detail & Related papers (2023-11-15T05:11:43Z) - Solving NLP Problems through Human-System Collaboration: A
Discussion-based Approach [98.13835740351932]
This research aims to create a dataset and computational framework for systems that discuss and refine their predictions through dialogue.
We show that the proposed system can have beneficial discussions with humans improving the accuracy by up to 25 points in the natural language inference task.
arXiv Detail & Related papers (2023-05-19T16:24:50Z) - Dialogue Inspectional Summarization with Factual Inconsistency Awareness [34.97845384948336]
We investigate the factual inconsistency problem for Dialogue Inspectional Summarization (DIS) under non-pretraining and pretraining settings.
An innovative end-to-end dialogue summary generation framework is proposed with two auxiliary tasks.
Comprehensive experiments demonstrate that the proposed model can generate a more readable summary with accurate coverage of factual aspects.
arXiv Detail & Related papers (2021-11-05T06:26:22Z) - "How Robust r u?": Evaluating Task-Oriented Dialogue Systems on Spoken
Conversations [87.95711406978157]
This work presents a new benchmark on spoken task-oriented conversations.
We study multi-domain dialogue state tracking and knowledge-grounded dialogue modeling.
Our data set enables speech-based benchmarking of task-oriented dialogue systems.
arXiv Detail & Related papers (2021-09-28T04:51:04Z) - Self-supervised Dialogue Learning for Spoken Conversational Question
Answering [29.545937716796082]
In spoken conversational question answering (SCQA), the answer to the corresponding question is generated by retrieving and then analyzing a fixed spoken document, including multi-part conversations.
We introduce a self-supervised learning approach, including incoherence discrimination, insertion detection, and question prediction, to explicitly capture the coreference resolution and dialogue coherence.
Our proposed method provides more coherent, meaningful, and appropriate responses, yielding superior performance gains compared to the original pre-trained language models.
arXiv Detail & Related papers (2021-06-04T00:09:38Z) - I like fish, especially dolphins: Addressing Contradictions in Dialogue
Modeling [104.09033240889106]
We introduce the DialoguE COntradiction DEtection task (DECODE) and a new conversational dataset containing both human-human and human-bot contradictory dialogues.
We then compare a structured utterance-based approach of using pre-trained Transformer models for contradiction detection with the typical unstructured approach.
arXiv Detail & Related papers (2020-12-24T18:47:49Z) - Rethinking Dialogue State Tracking with Reasoning [76.0991910623001]
This paper proposes to track dialogue states gradually with reasoning over dialogue turns with the help of the back-end data.
Empirical results demonstrate that our method significantly outperforms the state-of-the-art methods by 38.6% in terms of joint belief accuracy for MultiWOZ 2.1.
arXiv Detail & Related papers (2020-05-27T02:05:33Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.