KET-QA: A Dataset for Knowledge Enhanced Table Question Answering

• URL: http://arxiv.org/abs/2405.08099v1
• Date: Mon, 13 May 2024 18:26:32 GMT
• Title: KET-QA: A Dataset for Knowledge Enhanced Table Question Answering
• Authors: Mengkang Hu, Haoyu Dong, Ping Luo, Shi Han, Dongmei Zhang,
• Abstract summary: We propose to use a knowledge base (KB) as the external knowledge source for TableQA. Every question requires the integration of information from both the table and the sub-graph to be answered. We design a retriever-reasoner structured pipeline model to extract pertinent information from the vast knowledge sub-graph.
• Score: 63.56707527868466
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Due to the concise and structured nature of tables, the knowledge contained therein may be incomplete or missing, posing a significant challenge for table question answering (TableQA) and data analysis systems. Most existing datasets either fail to address the issue of external knowledge in TableQA or only utilize unstructured text as supplementary information for tables. In this paper, we propose to use a knowledge base (KB) as the external knowledge source for TableQA and construct a dataset KET-QA with fine-grained gold evidence annotation. Each table in the dataset corresponds to a sub-graph of the entire KB, and every question requires the integration of information from both the table and the sub-graph to be answered. To extract pertinent information from the vast knowledge sub-graph and apply it to TableQA, we design a retriever-reasoner structured pipeline model. Experimental results demonstrate that our model consistently achieves remarkable relative performance improvements ranging from 1.9 to 6.5 times and absolute improvements of 11.66% to 44.64% on EM scores across three distinct settings (fine-tuning, zero-shot, and few-shot), in comparison with solely relying on table information in the traditional TableQA manner. However, even the best model achieves a 60.23% EM score, which still lags behind the human-level performance, highlighting the challenging nature of KET-QA for the question-answering community. We also provide a human evaluation of error cases to analyze further the aspects in which the model can be improved. Project page: https://ketqa.github.io/.

Related papers

• TANQ: An open domain dataset of table answered questions [15.323690523538572]
  TANQ is the first open domain question answering dataset where the answers require building tables from information across multiple sources. We release the full source attribution for every cell in the resulting table and benchmark state-of-the-art language models in open, oracle, and closed book setups. Our best-performing baseline, GPT4 reaches an overall F1 score of 29.1, lagging behind human performance by 19.7 points.
  arXiv  Detail & Related papers  (2024-05-13T14:07:20Z)
• Augment before You Try: Knowledge-Enhanced Table Question Answering via Table Expansion [57.53174887650989]
  Table question answering is a popular task that assesses a model's ability to understand and interact with structured data. Existing methods either convert both the table and external knowledge into text, which neglects the structured nature of the table. We propose a simple yet effective method to integrate external information in a given table.
  arXiv  Detail & Related papers  (2024-01-28T03:37:11Z)
• Localize, Retrieve and Fuse: A Generalized Framework for Free-Form Question Answering over Tables [46.039687237878105]
  TableQA aims at generating answers to questions grounded on a provided table. Table-to- Graph conversion, cell localizing, external knowledge retrieval, and the fusion of table and text are proposed. Experiments showcase the superior capabilities of TAG-QA in generating sentences that are both faithful and coherent.
  arXiv  Detail & Related papers  (2023-09-20T03:52:34Z)
• RobuT: A Systematic Study of Table QA Robustness Against Human-Annotated Adversarial Perturbations [13.900589860309488]
  RobuT builds upon existing Table QA datasets (WTQ, Wiki-Weak, and SQA) Our results indicate that both state-of-the-art Table QA models and large language models (e.g., GPT-3) with few-shot learning falter in these adversarial sets. We propose to address this problem by using large language models to generate adversarial examples to enhance training.
  arXiv  Detail & Related papers  (2023-06-25T19:23:21Z)
• QTSumm: Query-Focused Summarization over Tabular Data [58.62152746690958]
  People primarily consult tables to conduct data analysis or answer specific questions. We define a new query-focused table summarization task, where text generation models have to perform human-like reasoning. We introduce a new benchmark named QTSumm for this task, which contains 7,111 human-annotated query-summary pairs over 2,934 tables.
  arXiv  Detail & Related papers  (2023-05-23T17:43:51Z)
• MultiTabQA: Generating Tabular Answers for Multi-Table Question Answering [61.48881995121938]
  Real-world queries are complex in nature, often over multiple tables in a relational database or web page. Our model, MultiTabQA, not only answers questions over multiple tables, but also generalizes to generate tabular answers.
  arXiv  Detail & Related papers  (2023-05-22T08:25:15Z)
• OmniTab: Pretraining with Natural and Synthetic Data for Few-shot Table-based Question Answering [106.73213656603453]
  We develop a simple table-based QA model with minimal annotation effort. We propose an omnivorous pretraining approach that consumes both natural and synthetic data.
  arXiv  Detail & Related papers  (2022-07-08T01:23:45Z)
• Table Retrieval May Not Necessitate Table-specific Model Design [83.27735758203089]
  We focus on the task of table retrieval, and ask: "is table-specific model design necessary for table retrieval?" Based on an analysis on a table-based portion of the Natural Questions dataset (NQ-table), we find that structure plays a negligible role in more than 70% of the cases. We then experiment with three modules to explicitly encode table structures, namely auxiliary row/column embeddings, hard attention masks, and soft relation-based attention biases. None of these yielded significant improvements, suggesting that table-specific model design may not be necessary for table retrieval.
  arXiv  Detail & Related papers  (2022-05-19T20:35:23Z)
• Knowledge Graph-based Question Answering with Electronic Health Records [30.901617020638124]
  Question Answering (QA) is a widely-used framework for developing and evaluating an intelligent machine. This paper proposes a graph-based EHR QA where natural language queries are converted to SPARQL. All datasets are open-sourced to encourage further EHR QA research in both directions.
  arXiv  Detail & Related papers  (2020-10-19T11:31:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.