Enhancing Visual Dialog State Tracking through Iterative Object-Entity Alignment in Multi-Round Conversations

• URL: http://arxiv.org/abs/2408.06725v1
• Date: Tue, 13 Aug 2024 08:36:15 GMT
• Title: Enhancing Visual Dialog State Tracking through Iterative Object-Entity Alignment in Multi-Round Conversations
• Authors: Wei Pang, Ruixue Duan, Jinfu Yang, Ning Li,
• Abstract summary: We introduce Multi-round Dialogue State Tracking model (MDST) MDST captures each round of dialog history, constructing internal dialogue state representations defined as 2-tuples of vision-language representations. Experimental results on the VisDial v1.0 dataset demonstrate that MDST achieves a new state-of-the-art performance in generative setting.
• Score: 3.784841749866846
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Visual Dialog (VD) is a task where an agent answers a series of image-related questions based on a multi-round dialog history. However, previous VD methods often treat the entire dialog history as a simple text input, disregarding the inherent conversational information flows at the round level. In this paper, we introduce Multi-round Dialogue State Tracking model (MDST), a framework that addresses this limitation by leveraging the dialogue state learned from dialog history to answer questions. MDST captures each round of dialog history, constructing internal dialogue state representations defined as 2-tuples of vision-language representations. These representations effectively ground the current question, enabling the generation of accurate answers. Experimental results on the VisDial v1.0 dataset demonstrate that MDST achieves a new state-of-the-art performance in generative setting. Furthermore, through a series of human studies, we validate the effectiveness of MDST in generating long, consistent, and human-like answers while consistently answering a series of questions correctly.

Related papers

• BI-MDRG: Bridging Image History in Multimodal Dialogue Response Generation [21.052101309555464]
  Multimodal Dialogue Response Generation (MDRG) is a recently proposed task where the model needs to generate responses in texts, images, or a blend of both. Previous work relies on the text modality as an intermediary step for both the image input and output of the model rather than adopting an end-to-end approach. We propose BI-MDRG that bridges the response generation path such that the image history information is utilized for enhanced relevance of text responses to the image content.
  arXiv  Detail & Related papers  (2024-08-12T05:22:42Z)
• SPACE-2: Tree-Structured Semi-Supervised Contrastive Pre-training for Task-Oriented Dialog Understanding [68.94808536012371]
  We propose a tree-structured pre-trained conversation model, which learns dialog representations from limited labeled dialogs and large-scale unlabeled dialog corpora. Our method can achieve new state-of-the-art results on the DialoGLUE benchmark consisting of seven datasets and four popular dialog understanding tasks.
  arXiv  Detail & Related papers  (2022-09-14T13:42:50Z)
• Multimodal Dialogue State Tracking [97.25466640240619]
  Video-Dialogue Transformer Network (VDTN) learns contextual dependencies between videos and dialogues to generate multimodal dialogue states. VDTN combines both object-level features and segment-level features and learns contextual dependencies between videos and dialogues to generate multimodal dialogue states.
  arXiv  Detail & Related papers  (2022-06-16T03:18:42Z)
• Beyond the Granularity: Multi-Perspective Dialogue Collaborative Selection for Dialogue State Tracking [18.172993687706708]
  In dialogue state tracking, dialogue history is a crucial material, and its utilization varies between different models. We propose DiCoS-DST to dynamically select the relevant dialogue contents corresponding to each slot for state updating. Our approach achieves new state-of-the-art performance on MultiWOZ 2.1 and MultiWOZ 2.2, and achieves superior performance on multiple mainstream benchmark datasets.
  arXiv  Detail & Related papers  (2022-05-20T10:08:45Z)
• HybriDialogue: An Information-Seeking Dialogue Dataset Grounded on Tabular and Textual Data [87.67278915655712]
  We present a new dialogue dataset, HybriDialogue, which consists of crowdsourced natural conversations grounded on both Wikipedia text and tables. The conversations are created through the decomposition of complex multihop questions into simple, realistic multiturn dialogue interactions.
  arXiv  Detail & Related papers  (2022-04-28T00:52:16Z)
• Back to the Future: Bidirectional Information Decoupling Network for Multi-turn Dialogue Modeling [80.51094098799736]
  We propose Bidirectional Information Decoupling Network (BiDeN) as a universal dialogue encoder. BiDeN explicitly incorporates both the past and future contexts and can be generalized to a wide range of dialogue-related tasks. Experimental results on datasets of different downstream tasks demonstrate the universality and effectiveness of our BiDeN.
  arXiv  Detail & Related papers  (2022-04-18T03:51:46Z)
• Dialogue Summaries as Dialogue States (DS2), Template-Guided Summarization for Few-shot Dialogue State Tracking [16.07100713414678]
  Few-shot dialogue state tracking (DST) is a realistic solution to this problem. We propose to reformulate dialogue state tracking as a dialogue summarization problem.
  arXiv  Detail & Related papers  (2022-03-03T07:54:09Z)
• Dialogue State Tracking with Multi-Level Fusion of Predicted Dialogue States and Conversations [2.6529642559155944]
  We propose the Dialogue State Tracking with Multi-Level Fusion of Predicted Dialogue States and Conversations network. This model extracts information of each dialogue turn by modeling interactions among each turn utterance, the corresponding last dialogue states, and dialogue slots.
  arXiv  Detail & Related papers  (2021-07-12T02:30:30Z)
• Reasoning in Dialog: Improving Response Generation by Context Reading Comprehension [49.92173751203827]
  In multi-turn dialog, utterances do not always take the full form of sentences. We propose to improve the response generation performance by examining the model's ability to answer a reading comprehension question.
  arXiv  Detail & Related papers  (2020-12-14T10:58:01Z)
• UniConv: A Unified Conversational Neural Architecture for Multi-domain Task-oriented Dialogues [101.96097419995556]
  "UniConv" is a novel unified neural architecture for end-to-end conversational systems in task-oriented dialogues. We conduct comprehensive experiments in dialogue state tracking, context-to-text, and end-to-end settings on the MultiWOZ2.1 benchmark.
  arXiv  Detail & Related papers  (2020-04-29T16:28:22Z)
• Multi-View Attention Network for Visual Dialog [5.731758300670842]
  It is necessary for an agent to 1) determine the semantic intent of question and 2) align question-relevant textual and visual contents. We propose Multi-View Attention Network (MVAN), which leverages multiple views about heterogeneous inputs. MVAN effectively captures the question-relevant information from the dialog history with two complementary modules.
  arXiv  Detail & Related papers  (2020-04-29T08:46:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.