KQA Pro: A Dataset with Explicit Compositional Programs for Complex Question Answering over Knowledge Base

• URL: http://arxiv.org/abs/2007.03875v4
• Date: Thu, 23 Jun 2022 09:23:52 GMT
• Title: KQA Pro: A Dataset with Explicit Compositional Programs for Complex Question Answering over Knowledge Base
• Authors: Shulin Cao, Jiaxin Shi, Liangming Pan, Lunyiu Nie, Yutong Xiang, Lei Hou, Juanzi Li, Bin He, Hanwang Zhang
• Abstract summary: We introduce KQA Pro, a dataset for Complex KBQA including 120K diverse natural language questions. For each question, we provide the corresponding KoPL program and SPARQL query, so that KQA Pro serves for both KBQA and semantic parsing tasks.
• Score: 67.87878113432723
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Complex question answering over knowledge base (Complex KBQA) is challenging because it requires various compositional reasoning capabilities, such as multi-hop inference, attribute comparison, set operation. Existing benchmarks have some shortcomings that limit the development of Complex KBQA: 1) they only provide QA pairs without explicit reasoning processes; 2) questions are poor in diversity or scale. To this end, we introduce KQA Pro, a dataset for Complex KBQA including ~120K diverse natural language questions. We introduce a compositional and interpretable programming language KoPL to represent the reasoning process of complex questions. For each question, we provide the corresponding KoPL program and SPARQL query, so that KQA Pro serves for both KBQA and semantic parsing tasks. Experimental results show that SOTA KBQA methods cannot achieve promising results on KQA Pro as on current datasets, which suggests that KQA Pro is challenging and Complex KBQA requires further research efforts. We also treat KQA Pro as a diagnostic dataset for testing multiple reasoning skills, conduct a thorough evaluation of existing models and discuss further directions for Complex KBQA. Our codes and datasets can be obtained from https://github.com/shijx12/KQAPro_Baselines.

Related papers

• MarkQA: A large scale KBQA dataset with numerical reasoning [11.072552105311484]
  We propose a new task, NR-KBQA, which requires the ability to perform both multi-hop reasoning and numerical reasoning. We design a logic form in Python format called PyQL to represent the reasoning process of numerical reasoning questions. We present a large dataset called MarkQA, which is automatically constructed from a small set of seeds.
  arXiv  Detail & Related papers  (2023-10-24T04:50:59Z)
• Question Decomposition Tree for Answering Complex Questions over Knowledge Bases [9.723321745919186]
  We propose Question Decomposition Tree (QDT) to represent the structure of complex questions. Inspired by recent advances in natural language generation (NLG), we present a two-staged method called Clue-Decipher to generate QDT. To verify that QDT can enhance KBQA task, we design a decomposition-based KBQA system called QDTQA.
  arXiv  Detail & Related papers  (2023-06-13T07:44:29Z)
• Do I have the Knowledge to Answer? Investigating Answerability of Knowledge Base Questions [25.13991044303459]
  We create GrailQAbility, a new benchmark KBQA dataset with unanswerability. Experimenting with three state-of-the-art KBQA models, we find that all three models suffer a drop in performance. This underscores the need for further research in making KBQA systems robust to unanswerability.
  arXiv  Detail & Related papers  (2022-12-20T12:00:26Z)
• RoMQA: A Benchmark for Robust, Multi-evidence, Multi-answer Question Answering [87.18962441714976]
  We introduce RoMQA, the first benchmark for robust, multi-evidence, multi-answer question answering (QA) We evaluate state-of-the-art large language models in zero-shot, few-shot, and fine-tuning settings, and find that RoMQA is challenging. Our results show that RoMQA is a challenging benchmark for large language models, and provides a quantifiable test to build more robust QA methods.
  arXiv  Detail & Related papers  (2022-10-25T21:39:36Z)
• PACIFIC: Towards Proactive Conversational Question Answering over Tabular and Textual Data in Finance [96.06505049126345]
  We present a new dataset, named PACIFIC. Compared with existing CQA datasets, PACIFIC exhibits three key features: (i) proactivity, (ii) numerical reasoning, and (iii) hybrid context of tables and text. A new task is defined accordingly to study Proactive Conversational Question Answering (PCQA), which combines clarification question generation and CQA. UniPCQA performs multi-task learning over all sub-tasks in PCQA and incorporates a simple ensemble strategy to alleviate the error propagation issue in the multi-task learning by cross-validating top-$k$ sampled Seq2Seq
  arXiv  Detail & Related papers  (2022-10-17T08:06:56Z)
• ProQA: Structural Prompt-based Pre-training for Unified Question Answering [84.59636806421204]
  ProQA is a unified QA paradigm that solves various tasks through a single model. It concurrently models the knowledge generalization for all QA tasks while keeping the knowledge customization for every specific QA task. ProQA consistently boosts performance on both full data fine-tuning, few-shot learning, and zero-shot testing scenarios.
  arXiv  Detail & Related papers  (2022-05-09T04:59:26Z)
• ConditionalQA: A Complex Reading Comprehension Dataset with Conditional Answers [93.55268936974971]
  We describe a Question Answering dataset that contains complex questions with conditional answers. We call this dataset ConditionalQA. We show that ConditionalQA is challenging for many of the existing QA models, especially in selecting answer conditions.
  arXiv  Detail & Related papers  (2021-10-13T17:16:46Z)
• Complex Knowledge Base Question Answering: A Survey [41.680033017518376]
  Knowledge base question answering (KBQA) aims to answer a question over a knowledge base (KB) In recent years, researchers propose a large number of novel methods, which looked into the challenges of answering complex questions. We present two mainstream categories of methods for complex KBQA, namely semantic parsing-based (SP-based) methods and information retrieval-based (IR-based) methods.
  arXiv  Detail & Related papers  (2021-08-15T08:14:54Z)
• Generating Diverse and Consistent QA pairs from Contexts with Information-Maximizing Hierarchical Conditional VAEs [62.71505254770827]
  We propose a conditional variational autoencoder (HCVAE) for generating QA pairs given unstructured texts as contexts. Our model obtains impressive performance gains over all baselines on both tasks, using only a fraction of data for training.
  arXiv  Detail & Related papers  (2020-05-28T08:26:06Z)
• A Complex KBQA System using Multiple Reasoning Paths [42.007327947635595]
  Multi-hop knowledge based question answering (KBQA) is a complex task for natural language understanding. We introduce an end-to-end KBQA system which can leverage multiple reasoning paths' information and only requires labeled answer as supervision.
  arXiv  Detail & Related papers  (2020-05-22T02:35:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.