On the Robustness of Dialogue History Representation in Conversational Question Answering: A Comprehensive Study and a New Prompt-based Method

• URL: http://arxiv.org/abs/2206.14796v1
• Date: Wed, 29 Jun 2022 17:55:43 GMT
• Title: On the Robustness of Dialogue History Representation in Conversational Question Answering: A Comprehensive Study and a New Prompt-based Method
• Authors: Zorik Gekhman, Nadav Oved, Orgad Keller, Idan Szpektor, Roi Reichart
• Abstract summary: We conduct the first large-scale robustness study of history modeling approaches for CQA. We design a novel prompt-based history modeling approach, and demonstrate its strong robustness across various settings. Our approach is simple, easy-to-plug into practically any model, and highly effective.
• Score: 22.53491549115403
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Most works on modeling the conversation history in Conversational Question Answering (CQA) report a single main result on a common CQA benchmark. While existing models show impressive results on CQA leaderboards, it remains unclear whether they are robust to shifts in setting (sometimes to more realistic ones), training data size (e.g. from large to small sets) and domain. In this work, we design and conduct the first large-scale robustness study of history modeling approaches for CQA. We find that high benchmark scores do not necessarily translate to strong robustness, and that various methods can perform extremely differently under different settings. Equipped with the insights from our study, we design a novel prompt-based history modeling approach, and demonstrate its strong robustness across various settings. Our approach is inspired by existing methods that highlight historic answers in the passage. However, instead of highlighting by modifying the passage token embeddings, we add textual prompts directly in the passage text. Our approach is simple, easy-to-plug into practically any model, and highly effective, thus we recommend it as a starting point for future model developers. We also hope that our study and insights will raise awareness to the importance of robustness-focused evaluation, in addition to obtaining high leaderboard scores, leading to better CQA systems.

Related papers

• Long-Span Question-Answering: Automatic Question Generation and QA-System Ranking via Side-by-Side Evaluation [65.16137964758612]
  We explore the use of long-context capabilities in large language models to create synthetic reading comprehension data from entire books. Our objective is to test the capabilities of LLMs to analyze, understand, and reason over problems that require a detailed comprehension of long spans of text.
  arXiv  Detail & Related papers  (2024-05-31T20:15:10Z)
• Multi-Modal Prompt Learning on Blind Image Quality Assessment [65.0676908930946]
  Image Quality Assessment (IQA) models benefit significantly from semantic information, which allows them to treat different types of objects distinctly. Traditional methods, hindered by a lack of sufficiently annotated data, have employed the CLIP image-text pretraining model as their backbone to gain semantic awareness. Recent approaches have attempted to address this mismatch using prompt technology, but these solutions have shortcomings. This paper introduces an innovative multi-modal prompt-based methodology for IQA.
  arXiv  Detail & Related papers  (2024-04-23T11:45:32Z)
• History Semantic Graph Enhanced Conversational KBQA with Temporal Information Modeling [28.27368343751272]
  We propose a History Semantic Graph Enhanced KBQA model (HSGE) that is able to effectively model long-range semantic dependencies in conversation history. We evaluate HSGE on a widely used benchmark dataset for complex sequential question answering.
  arXiv  Detail & Related papers  (2023-06-12T05:10:58Z)
• Recitation-Augmented Language Models [85.30591349383849]
  We show that RECITE is a powerful paradigm for knowledge-intensive NLP tasks. Specifically, we show that by utilizing recitation as the intermediate step, a recite-and-answer scheme can achieve new state-of-the-art performance.
  arXiv  Detail & Related papers  (2022-10-04T00:49:20Z)
• Augmenting Pre-trained Language Models with QA-Memory for Open-Domain Question Answering [38.071375112873675]
  We propose a question-answer augmented encoder-decoder model and accompanying pretraining strategy. This yields an end-to-end system that outperforms prior QA retrieval methods on single-hop QA tasks.
  arXiv  Detail & Related papers  (2022-04-10T02:33:00Z)
• KGI: An Integrated Framework for Knowledge Intensive Language Tasks [16.511913995069097]
  In this paper, we propose a system based on an enhanced version of this approach for other knowledge intensive language tasks. Our system achieves results comparable to the best models in the KILT leaderboards.
  arXiv  Detail & Related papers  (2022-04-08T10:36:21Z)
• elBERto: Self-supervised Commonsense Learning for Question Answering [131.51059870970616]
  We propose a Self-supervised Bidirectional Representation Learning of Commonsense framework, which is compatible with off-the-shelf QA model architectures. The framework comprises five self-supervised tasks to force the model to fully exploit the additional training signals from contexts containing rich commonsense. elBERto achieves substantial improvements on out-of-paragraph and no-effect questions where simple lexical similarity comparison does not help.
  arXiv  Detail & Related papers  (2022-03-17T16:23:45Z)
• Conversational Query Rewriting with Self-supervised Learning [36.392717968127016]
  Conversational Query Rewriting (CQR) aims to simplify the multi-turn dialogue modeling into a single-turn problem by explicitly rewriting the conversational query into a self-contained utterance. Existing approaches rely on massive supervised training data, which is labor-intensive to annotate. We propose to construct a large-scale CQR dataset automatically via self-supervised learning, which does not need human annotation.
  arXiv  Detail & Related papers  (2021-02-09T08:57:53Z)
• Counterfactual Variable Control for Robust and Interpretable Question Answering [57.25261576239862]
  Deep neural network based question answering (QA) models are neither robust nor explainable in many cases. In this paper, we inspect such spurious "capability" of QA models using causal inference. We propose a novel approach called Counterfactual Variable Control (CVC) that explicitly mitigates any shortcut correlation.
  arXiv  Detail & Related papers  (2020-10-12T10:09:05Z)
• SRQA: Synthetic Reader for Factoid Question Answering [21.28441702154528]
  We introduce a new model called SRQA, which means Synthetic Reader for Factoid Question Answering. This model enhances the question answering system in the multi-document scenario from three aspects. We perform SRQA on the WebQA dataset, and experiments show that our model outperforms the state-of-the-art models.
  arXiv  Detail & Related papers  (2020-09-02T13:16:24Z)
• Harvesting and Refining Question-Answer Pairs for Unsupervised QA [95.9105154311491]
  We introduce two approaches to improve unsupervised Question Answering (QA) First, we harvest lexically and syntactically divergent questions from Wikipedia to automatically construct a corpus of question-answer pairs (named as RefQA) Second, we take advantage of the QA model to extract more appropriate answers, which iteratively refines data over RefQA.
  arXiv  Detail & Related papers  (2020-05-06T15:56:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.