Improving Bot Response Contradiction Detection via Utterance Rewriting
- URL: http://arxiv.org/abs/2207.11862v1
- Date: Mon, 25 Jul 2022 00:54:30 GMT
- Title: Improving Bot Response Contradiction Detection via Utterance Rewriting
- Authors: Di Jin, Sijia Liu, Yang Liu, Dilek Hakkani-Tur
- Abstract summary: This work aims to improve the contradiction detection via rewriting all bot utterances to restore antecedents and ellipsis.
We empirically demonstrate that this model can produce satisfactory rewrites to make bot utterances more complete.
Using rewritten utterances improves contradiction detection performance significantly, e.g., the AUPR and joint accuracy scores (detecting contradiction along with evidence) increase by 6.5% and 4.5%, respectively.
- Score: 45.55560596440624
- License: http://creativecommons.org/licenses/by-nc-nd/4.0/
- Abstract: Though chatbots based on large neural models can often produce fluent
responses in open domain conversations, one salient error type is contradiction
or inconsistency with the preceding conversation turns. Previous work has
treated contradiction detection in bot responses as a task similar to natural
language inference, e.g., detect the contradiction between a pair of bot
utterances. However, utterances in conversations may contain co-references or
ellipsis, and using these utterances as is may not always be sufficient for
identifying contradictions. This work aims to improve the contradiction
detection via rewriting all bot utterances to restore antecedents and ellipsis.
We curated a new dataset for utterance rewriting and built a rewriting model on
it. We empirically demonstrate that this model can produce satisfactory
rewrites to make bot utterances more complete. Furthermore, using rewritten
utterances improves contradiction detection performance significantly, e.g.,
the AUPR and joint accuracy scores (detecting contradiction along with
evidence) increase by 6.5% and 4.5% (absolute increase), respectively.
Related papers
- SparseCL: Sparse Contrastive Learning for Contradiction Retrieval [87.02936971689817]
Contradiction retrieval refers to identifying and extracting documents that explicitly disagree with or refute the content of a query.
Existing methods such as similarity search and crossencoder models exhibit significant limitations.
We introduce SparseCL that leverages specially trained sentence embeddings designed to preserve subtle, contradictory nuances between sentences.
arXiv Detail & Related papers (2024-06-15T21:57:03Z) - Self-contradictory Hallucinations of Large Language Models: Evaluation, Detection and Mitigation [5.043563227694139]
Large language models (large LMs) are susceptible to producing text that contains hallucinated content.
We present a comprehensive investigation into self-contradiction for various instruction-tuned LMs.
We propose a novel prompting-based framework designed to effectively detect and mitigate self-contradictions.
arXiv Detail & Related papers (2023-05-25T08:43:46Z) - AutoReply: Detecting Nonsense in Dialogue Introspectively with
Discriminative Replies [71.62832112141913]
We show that dialogue models can detect errors in their own messages introspectively, by calculating the likelihood of replies that are indicative of poor messages.
We first show that hand-crafted replies can be effective for the task of detecting nonsense in applications as complex as Diplomacy.
We find that AutoReply-generated replies outperform handcrafted replies and perform on par with carefully fine-tuned large supervised models.
arXiv Detail & Related papers (2022-11-22T22:31:34Z) - CDConv: A Benchmark for Contradiction Detection in Chinese Conversations [74.78715797366395]
We propose a benchmark for Contradiction Detection in Chinese Conversations, namely CDConv.
It contains 12K multi-turn conversations annotated with three typical contradiction categories: Intra-sentence Contradiction, Role Confusion, and History Contradiction.
arXiv Detail & Related papers (2022-10-16T11:37:09Z) - Towards Robust Online Dialogue Response Generation [62.99904593650087]
We argue that this can be caused by a discrepancy between training and real-world testing.
We propose a hierarchical sampling-based method consisting of both utterance-level sampling and semi-utterance-level sampling.
arXiv Detail & Related papers (2022-03-07T06:51:41Z) - WikiContradiction: Detecting Self-Contradiction Articles on Wikipedia [8.755487474723994]
We propose a task of detecting self-contradiction articles in Wikipedia.
Based on the "self-contradictory" template, we create a novel dataset for the self-contradiction detection task.
We present the first model, Pairwise Contradiction Neural Network (PCNN), to not only effectively identify self-contradiction articles, but also highlight the most contradiction pairs of contradiction sentences.
arXiv Detail & Related papers (2021-11-16T15:12:37Z) - I Beg to Differ: A study of constructive disagreement in online
conversations [15.581515781839656]
We construct a corpus of 7 425 Wikipedia Talk page conversations that contain content disputes.
We define the task of predicting whether disagreements will be escalated to mediation by a moderator.
We develop a variety of neural models and show that taking into account the structure of the conversation improves predictive accuracy.
arXiv Detail & Related papers (2021-01-26T16:36:43Z) - I like fish, especially dolphins: Addressing Contradictions in Dialogue
Modeling [104.09033240889106]
We introduce the DialoguE COntradiction DEtection task (DECODE) and a new conversational dataset containing both human-human and human-bot contradictory dialogues.
We then compare a structured utterance-based approach of using pre-trained Transformer models for contradiction detection with the typical unstructured approach.
arXiv Detail & Related papers (2020-12-24T18:47:49Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.