Don't be Contradicted with Anything! CI-ToD: Towards Benchmarking Consistency for Task-oriented Dialogue System

• URL: http://arxiv.org/abs/2109.11292v1
• Date: Thu, 23 Sep 2021 10:56:31 GMT
• Title: Don't be Contradicted with Anything! CI-ToD: Towards Benchmarking Consistency for Task-oriented Dialogue System
• Abstract summary: In this paper, we argue that consistency problem is more urgent in task-oriented domain. To facilitate the research, we introduce CI-ToD, a novel dataset for Consistency Identification in Task-oriented Dialog system. Empirical results show that state-of-the-art methods only achieve 51.3%, which is far behind the human performance of 93.2%.
• Score: 24.227068548489385
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Consistency Identification has obtained remarkable success on open-domain dialogue, which can be used for preventing inconsistent response generation. However, in contrast to the rapid development in open-domain dialogue, few efforts have been made to the task-oriented dialogue direction. In this paper, we argue that consistency problem is more urgent in task-oriented domain. To facilitate the research, we introduce CI-ToD, a novel dataset for Consistency Identification in Task-oriented Dialog system. In addition, we not only annotate the single label to enable the model to judge whether the system response is contradictory, but also provide more fine-grained labels (i.e., Dialogue History Inconsistency, User Query Inconsistency and Knowledge Base Inconsistency) to encourage model to know what inconsistent sources lead to it. Empirical results show that state-of-the-art methods only achieve 51.3%, which is far behind the human performance of 93.2%, indicating that there is ample room for improving consistency identification ability. Finally, we conduct exhaustive experiments and qualitative analysis to comprehend key challenges and provide guidance for future directions. All datasets and models are publicly available at \url{https://github.com/yizhen20133868/CI-ToD}.

Related papers

• Towards Detecting Inconsistencies in End-to-end Generated TODs [2.5343479690037896]
  Large Language Models (LLMs) may still generate inconsistencies in Task-Oriented Dialogues (TODs)<n>We investigate a method for automatically detecting inconsistencies by conceptualizing TODs as a Constraint Satisfaction Problem (CSP)<n>We propose a pipeline that first identifies variables in a target dialogue and then applies a CSP solver to identify valid solutions.
  arXiv  Detail & Related papers  (2026-07-10T12:19:59Z)
• BeyondSWE: Can Current Code Agent Survive Beyond Single-Repo Bug Fixing? [61.247730037229815]
  We introduce BeyondSWE, a comprehensive benchmark that broadens existing evaluations along two axes - resolution scope and knowledge scope.<n>To investigate the role of external knowledge, we develop SearchSWE, a framework that integrates deep search with coding abilities.<n>This work offers both a realistic, challenging evaluation benchmark and a flexible framework to advance research toward more capable code agents.
  arXiv  Detail & Related papers  (2026-03-03T17:52:01Z)
• D-SMART: Enhancing LLM Dialogue Consistency via Dynamic Structured Memory And Reasoning Tree [22.420810089099614]
  Large Language Models (LLMs) often exhibit factual inconsistencies and logical decay in extended, multi-turn dialogues.<n>We propose D--101, a model-agnostic framework designed to maintain multi-turn dialogue consistency.<n>We introduce new NLI-based metrics to better measure multiturn dialogue consistency.
  arXiv  Detail & Related papers  (2025-10-15T09:53:11Z)
• Demystifying deep search: a holistic evaluation with hint-free multi-hop questions and factorised metrics [89.1999907891494]
  We present WebDetective, a benchmark of hint-free multi-hop questions paired with a controlled Wikipedia sandbox.<n>Our evaluation of 25 state-of-the-art models reveals systematic weaknesses across all architectures.<n>We develop an agentic workflow, EvidenceLoop, that explicitly targets the challenges our benchmark identifies.
  arXiv  Detail & Related papers  (2025-10-01T07:59:03Z)
• A Knowledge Graph and a Tripartite Evaluation Framework Make Retrieval-Augmented Generation Scalable and Transparent [0.0]
  This study presents a Retrieval Augmented Generation (RAG) that harnesses a knowledge graph and vector search retrieval to deliver context-rich responses.<n>A central innovation of this work is the introduction of RAG Evaluation (RAG-Eval), a novel chain-of-thought tripartite evaluation framework.<n>RAG-Eval reliably detects factual gaps and query mismatches, thereby fostering trust in high demand, data centric environments.
  arXiv  Detail & Related papers  (2025-09-23T16:29:22Z)
• Benchmarking Deep Search over Heterogeneous Enterprise Data [73.55304268238474]
  We present a new benchmark for evaluating a form of retrieval-augmented generation (RAG)<n>RAG requires source-aware, multi-hop reasoning over diverse, sparsed, but related sources.<n>We build it using a synthetic data pipeline that simulates business across product planning, development, and support stages.
  arXiv  Detail & Related papers  (2025-06-29T08:34:59Z)
• AI-Facilitated Analysis of Abstracts and Conclusions: Flagging Unsubstantiated Claims and Ambiguous Pronouns [0.0]
  We present and evaluate a suite of proof-of-concept prompts designed to elicit human-like hierarchical reasoning.<n>The prompts target two non-trivial analytical tasks within academic summaries (abstracts and conclusions)<n>We conducted a systematic, multi-run evaluation on two models (Gemini Pro 2.5 Pro and ChatGPT Plus o3) under varied context conditions.
  arXiv  Detail & Related papers  (2025-06-16T07:34:31Z)
• Collective Reasoning Among LLMs: A Framework for Answer Validation Without Ground Truth [0.0]
  We introduce a new approach in which several advanced large language models produce and answer intricate, doctoral-level probability problems.<n>Our investigation focuses on how agreement among diverse models can signal the reliability of their outputs.
  arXiv  Detail & Related papers  (2025-02-28T06:20:52Z)
• Explainable Procedural Mistake Detection [27.40806437649092]
  Procedural mistake detection is a challenging sub-problem of classifying whether a human user has successfully executed the task at hand. We recast PMD to an explanatory self-dialog of questions and answers, which serve as evidence for a decision. Our results show that while open-source VLMs struggle with this task off-the-shelf, their accuracy, coherence, and dialog efficiency can be vastly improved.
  arXiv  Detail & Related papers  (2024-12-16T16:13:55Z)
• Localizing Factual Inconsistencies in Attributable Text Generation [91.981439746404]
  We introduce QASemConsistency, a new formalism for localizing factual inconsistencies in attributable text generation. We first demonstrate the effectiveness of the QASemConsistency methodology for human annotation. We then implement several methods for automatically detecting localized factual inconsistencies.
  arXiv  Detail & Related papers  (2024-10-09T22:53:48Z)
• Self-Taught Recognizer: Toward Unsupervised Adaptation for Speech Foundation Models [84.8919069953397]
  Self-TAught Recognizer (STAR) is an unsupervised adaptation framework for speech recognition systems. We show that STAR achieves an average of 13.5% relative reduction in word error rate across 14 target domains. STAR exhibits high data efficiency that only requires less than one-hour unlabeled data.
  arXiv  Detail & Related papers  (2024-05-23T04:27:11Z)
• Inconsistent dialogue responses and how to recover from them [45.933921383946576]
  One critical issue for chat systems is to stay consistent about preferences, opinions, beliefs and facts of itself. In this work, we study methods to assess and bolster utterance consistency of chat systems.
  arXiv  Detail & Related papers  (2024-01-18T19:46:04Z)
• Improving the Robustness of Knowledge-Grounded Dialogue via Contrastive Learning [71.8876256714229]
  We propose an entity-based contrastive learning framework for improving the robustness of knowledge-grounded dialogue systems. Our method achieves new state-of-the-art performance in terms of automatic evaluation scores.
  arXiv  Detail & Related papers  (2024-01-09T05:16:52Z)
• PICK: Polished & Informed Candidate Scoring for Knowledge-Grounded Dialogue Systems [59.1250765143521]
  Current knowledge-grounded dialogue systems often fail to align the generated responses with human-preferred qualities. We propose Polished & Informed Candidate Scoring (PICK), a generation re-scoring framework. We demonstrate the effectiveness of PICK in generating responses that are more faithful while keeping them relevant to the dialogue history.
  arXiv  Detail & Related papers  (2023-09-19T08:27:09Z)
• DiactTOD: Learning Generalizable Latent Dialogue Acts for Controllable Task-Oriented Dialogue Systems [15.087619144902776]
  We present a novel end-to-end latent dialogue act model (DiactTOD) that represents dialogue acts in a latent space. When pre-trained on a large corpus, DiactTOD is able to predict and control dialogue acts to generate controllable responses.
  arXiv  Detail & Related papers  (2023-08-01T23:29:16Z)
• Q-TOD: A Query-driven Task-oriented Dialogue System [33.18698942938547]
  We introduce a novel query-driven task-oriented dialogue system, namely Q-TOD. The essential information from the dialogue context is extracted into a query, which is further employed to retrieve relevant knowledge records for response generation. To evaluate the effectiveness of the proposed Q-TOD, we collect query annotations for three publicly available task-oriented dialogue datasets.
  arXiv  Detail & Related papers  (2022-10-14T06:38:19Z)
• A Universality-Individuality Integration Model for Dialog Act Classification [0.0]
  Dialog Act (DA) reveals the general intent of the speaker utterance in a conversation. This paper suggests that word cues, part-of-speech cues and statistical cues can complement each other to improve the basis for recognition. We propose a novel model based on universality and individuality strategies, called Universality-Individuality Integration Model (UIIM)
  arXiv  Detail & Related papers  (2022-04-13T06:05:34Z)
• Utterance Rewriting with Contrastive Learning in Multi-turn Dialogue [22.103162555263143]
  We introduce contrastive learning and multi-task learning to jointly model the problem. Our proposed model achieves state-of-the-art performance on several public datasets.
  arXiv  Detail & Related papers  (2022-03-22T10:13:27Z)
• Improving Limited Labeled Dialogue State Tracking with Self-Supervision [91.68515201803986]
  Existing dialogue state tracking (DST) models require plenty of labeled data. We present and investigate two self-supervised objectives: preserving latent consistency and modeling conversational behavior. Our proposed self-supervised signals can improve joint goal accuracy by 8.95% when only 1% labeled data is used.
  arXiv  Detail & Related papers  (2020-10-26T21:57:42Z)
• CoCo: Controllable Counterfactuals for Evaluating Dialogue State Trackers [92.5628632009802]
  We propose controllable counterfactuals (CoCo) to bridge the gap and evaluate dialogue state tracking (DST) models on novel scenarios. CoCo generates novel conversation scenarios in two steps: (i) counterfactual goal generation at turn-level by dropping and adding slots followed by replacing slot values, and (ii) counterfactual conversation generation that is conditioned on (i) and consistent with the dialogue flow. Human evaluations show that COCO-generated conversations perfectly reflect the underlying user goal with more than 95% accuracy and are as human-like as the original conversations.
  arXiv  Detail & Related papers  (2020-10-24T09:39:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.