SubjQA: A Dataset for Subjectivity and Review Comprehension

• URL: http://arxiv.org/abs/2004.14283v3
• Date: Tue, 6 Oct 2020 06:04:27 GMT
• Title: SubjQA: A Dataset for Subjectivity and Review Comprehension
• Authors: Johannes Bjerva, Nikita Bhutani, Behzad Golshan, Wang-Chiew Tan, and Isabelle Augenstein
• Abstract summary: We investigate the relationship between subjectivity and question answering (QA) We find that subjectivity is also an important feature in the case of QA, albeit with more intricate interactions between subjectivity and QA performance. We release an English QA dataset (SubjQA) based on customer reviews, containing subjectivity annotations for questions and answer spans across 6 distinct domains.
• Score: 52.13338191442912
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Subjectivity is the expression of internal opinions or beliefs which cannot be objectively observed or verified, and has been shown to be important for sentiment analysis and word-sense disambiguation. Furthermore, subjectivity is an important aspect of user-generated data. In spite of this, subjectivity has not been investigated in contexts where such data is widespread, such as in question answering (QA). We therefore investigate the relationship between subjectivity and QA, while developing a new dataset. We compare and contrast with analyses from previous work, and verify that findings regarding subjectivity still hold when using recently developed NLP architectures. We find that subjectivity is also an important feature in the case of QA, albeit with more intricate interactions between subjectivity and QA performance. For instance, a subjective question may or may not be associated with a subjective answer. We release an English QA dataset (SubjQA) based on customer reviews, containing subjectivity annotations for questions and answer spans across 6 distinct domains.

Related papers

• SubjECTive-QA: Measuring Subjectivity in Earnings Call Transcripts' QA Through Six-Dimensional Feature Analysis [4.368712652579087]
  SubjECTive-QA is a human annotated dataset on Earnings Call Transcripts' (ECTs) The dataset includes 49,446 annotations for long-form QA pairs across six features: Assertive, Cautious, Optimistic, Specific, Clear, and Relevant. Our findings are that the best-performing Pre-trained Language Model (PLM), RoBERTa-base, has similar weighted F1 scores to Llama-3-70b-Chat on features with lower subjectivity.
  arXiv  Detail & Related papers  (2024-10-28T01:17:34Z)
• Qsnail: A Questionnaire Dataset for Sequential Question Generation [76.616068047362]
  We present the first dataset specifically constructed for the questionnaire generation task, which comprises 13,168 human-written questionnaires. We conduct experiments on Qsnail, and the results reveal that retrieval models and traditional generative models do not fully align with the given research topic and intents. Despite enhancements through the chain-of-thought prompt and finetuning, questionnaires generated by language models still fall short of human-written questionnaires.
  arXiv  Detail & Related papers  (2024-02-22T04:14:10Z)
• UNK-VQA: A Dataset and a Probe into the Abstention Ability of Multi-modal Large Models [55.22048505787125]
  This paper contributes a comprehensive dataset, called UNK-VQA. We first augment the existing data via deliberate perturbations on either the image or question. We then extensively evaluate the zero- and few-shot performance of several emerging multi-modal large models.
  arXiv  Detail & Related papers  (2023-10-17T02:38:09Z)
• Discourse Analysis via Questions and Answers: Parsing Dependency Structures of Questions Under Discussion [57.43781399856913]
  This work adopts the linguistic framework of Questions Under Discussion (QUD) for discourse analysis. We characterize relationships between sentences as free-form questions, in contrast to exhaustive fine-grained questions. We develop the first-of-its-kind QUD that derives a dependency structure of questions over full documents.
  arXiv  Detail & Related papers  (2022-10-12T03:53:12Z)
• Competency Problems: On Finding and Removing Artifacts in Language Data [50.09608320112584]
  We argue that for complex language understanding tasks, all simple feature correlations are spurious. We theoretically analyze the difficulty of creating data for competency problems when human bias is taken into account.
  arXiv  Detail & Related papers  (2021-04-17T21:34:10Z)
• Effective FAQ Retrieval and Question Matching With Unsupervised Knowledge Injection [10.82418428209551]
  We propose a contextual language model for retrieving appropriate answers to frequently asked questions. We also explore to capitalize on domain-specific topically-relevant relations between words in an unsupervised manner. We evaluate variants of our approach on a publicly-available Chinese FAQ dataset, and further apply and contextualize it to a large-scale question-matching task.
  arXiv  Detail & Related papers  (2020-10-27T05:03:34Z)
• Analysing the Effect of Clarifying Questions on Document Ranking in Conversational Search [10.335808358080289]
  We investigate how different aspects of clarifying questions and user answers affect the quality of ranking. We introduce a simple-based lexical baseline, that significantly outperforms the existing naive baselines.
  arXiv  Detail & Related papers  (2020-08-09T12:55:16Z)
• Subjective Question Answering: Deciphering the inner workings of Transformers in the realm of subjectivity [0.0]
  I've exploited a recently released dataset for span-selection Question Answering, namely SubjQA. SubjQA is the first dataset that contains questions that ask for subjective opinions corresponding to review paragraphs from six different domains. I've investigated the inner workings of a Transformer-based architecture to contribute to a better understanding of these not yet well understood "black-box" models.
  arXiv  Detail & Related papers  (2020-06-02T13:48:14Z)
• SQuINTing at VQA Models: Introspecting VQA Models with Sub-Questions [66.86887670416193]
  We show that state-of-the-art VQA models have comparable performance in answering perception and reasoning questions, but suffer from consistency problems. To address this shortcoming, we propose an approach called Sub-Question-aware Network Tuning (SQuINT) We show that SQuINT improves model consistency by 5%, also marginally improving performance on the Reasoning questions in VQA, while also displaying better attention maps.
  arXiv  Detail & Related papers  (2020-01-20T01:02:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.