Towards Detecting Inconsistencies in End-to-end Generated TODs

• URL: http://arxiv.org/abs/2607.09338v1
• Date: Fri, 10 Jul 2026 12:19:59 GMT
• Title: Towards Detecting Inconsistencies in End-to-end Generated TODs
• Abstract summary: Large Language Models (LLMs) may still generate inconsistencies in Task-Oriented Dialogues (TODs)<n>We investigate a method for automatically detecting inconsistencies by conceptualizing TODs as a Constraint Satisfaction Problem (CSP)<n>We propose a pipeline that first identifies variables in a target dialogue and then applies a CSP solver to identify valid solutions.
• Score: 2.5343479690037896
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Generative AI is profoundly transforming the core technologies behind conversational systems, shifting from component-based to end-to-end approaches. However, Large Language Models (LLMs) may still generate inconsistencies, a critical issue particularly in Task-Oriented Dialogues (TODs), where system responses must strictly adhere to information from a domain knowledge base (e.g., restaurants in a city). A single hallucination (e.g., suggesting a non-existent restaurant) can lead to severe task failures. We investigate a method for automatically detecting inconsistencies by conceptualizing TODs as a Constraint Satisfaction Problem (CSP), where variables represent dialogue segments referencing the conversational domain, and constraints among variables capture dialogue properties such as turn coherence and adherence to domain knowledge. We propose a pipeline that first identifies variables in a target dialogue and then applies a CSP solver to identify valid solutions. By comparing the target dialogue with valid variable assignments, we can detect inconsistencies and suggest minimal changes to ensure dialogue consistency. We demonstrate the high accuracy of the CSP-based approach in detecting inconsistencies, and provide a detailed analysis of our findings.

Related papers

• When Disagreements Elicit Robustness: Investigating Self-Repair Capabilities under LLM Multi-Agent Disagreements [56.29265568399648]
  We argue that disagreements prevent premature consensus and expand the explored solution space.<n>Disagreements on task-critical steps can derail collaboration depending on the topology of solution paths.
  arXiv  Detail & Related papers  (2025-02-21T02:24:43Z)
• Evaluating Task-Oriented Dialogue Consistency through Constraint Satisfaction [1.4272411349249625]
  We propose to conceptualize dialogue consistency as a Constraint Satisfaction Problem (CSP) We utilize a CSP solver to detect inconsistencies in dialogues re-lexicalized by an LLM. We argue that CSP captures core properties of dialogue consistency that have been poorly considered by approaches based on component pipelines.
  arXiv  Detail & Related papers  (2024-07-16T15:38:41Z)
• Inconsistent dialogue responses and how to recover from them [45.933921383946576]
  One critical issue for chat systems is to stay consistent about preferences, opinions, beliefs and facts of itself. In this work, we study methods to assess and bolster utterance consistency of chat systems.
  arXiv  Detail & Related papers  (2024-01-18T19:46:04Z)
• Are cascade dialogue state tracking models speaking out of turn in spoken dialogues? [1.786898113631979]
  This paper proposes a comprehensive analysis of the errors of state of the art systems in complex settings such as Dialogue State Tracking. Based on spoken MultiWoz, we identify that errors on non-categorical slots' values are essential to address in order to bridge the gap between spoken and chat-based dialogue systems.
  arXiv  Detail & Related papers  (2023-11-03T08:45:22Z)
• PICK: Polished & Informed Candidate Scoring for Knowledge-Grounded Dialogue Systems [59.1250765143521]
  Current knowledge-grounded dialogue systems often fail to align the generated responses with human-preferred qualities. We propose Polished & Informed Candidate Scoring (PICK), a generation re-scoring framework. We demonstrate the effectiveness of PICK in generating responses that are more faithful while keeping them relevant to the dialogue history.
  arXiv  Detail & Related papers  (2023-09-19T08:27:09Z)
• DiactTOD: Learning Generalizable Latent Dialogue Acts for Controllable Task-Oriented Dialogue Systems [15.087619144902776]
  We present a novel end-to-end latent dialogue act model (DiactTOD) that represents dialogue acts in a latent space. When pre-trained on a large corpus, DiactTOD is able to predict and control dialogue acts to generate controllable responses.
  arXiv  Detail & Related papers  (2023-08-01T23:29:16Z)
• Unsupervised Dialogue Topic Segmentation with Topic-aware Utterance Representation [51.22712675266523]
  Dialogue Topic (DTS) plays an essential role in a variety of dialogue modeling tasks. We propose a novel unsupervised DTS framework, which learns topic-aware utterance representations from unlabeled dialogue data.
  arXiv  Detail & Related papers  (2023-05-04T11:35:23Z)
• Don't be Contradicted with Anything! CI-ToD: Towards Benchmarking Consistency for Task-oriented Dialogue System [24.227068548489385]
  In this paper, we argue that consistency problem is more urgent in task-oriented domain. To facilitate the research, we introduce CI-ToD, a novel dataset for Consistency Identification in Task-oriented Dialog system. Empirical results show that state-of-the-art methods only achieve 51.3%, which is far behind the human performance of 93.2%.
  arXiv  Detail & Related papers  (2021-09-23T10:56:31Z)
• I like fish, especially dolphins: Addressing Contradictions in Dialogue Modeling [104.09033240889106]
  We introduce the DialoguE COntradiction DEtection task (DECODE) and a new conversational dataset containing both human-human and human-bot contradictory dialogues. We then compare a structured utterance-based approach of using pre-trained Transformer models for contradiction detection with the typical unstructured approach.
  arXiv  Detail & Related papers  (2020-12-24T18:47:49Z)
• Rethinking Dialogue State Tracking with Reasoning [76.0991910623001]
  This paper proposes to track dialogue states gradually with reasoning over dialogue turns with the help of the back-end data. Empirical results demonstrate that our method significantly outperforms the state-of-the-art methods by 38.6% in terms of joint belief accuracy for MultiWOZ 2.1.
  arXiv  Detail & Related papers  (2020-05-27T02:05:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.