Multi-modal Retrieval of Tables and Texts Using Tri-encoder Models

• URL: http://arxiv.org/abs/2108.04049v1
• Date: Mon, 9 Aug 2021 14:02:00 GMT
• Title: Multi-modal Retrieval of Tables and Texts Using Tri-encoder Models
• Authors: Bogdan Kosti\'c, Julian Risch, Timo M\"oller
• Abstract summary: Some questions cannot be answered by text alone but require information stored in tables. We present an approach for retrieving both texts and tables relevant to a question by jointly encoding texts, tables and questions into a single vector space. We release the newly created multi-modal dataset to the community so that it can be used for training and evaluation.
• Score: 2.5621280373733604
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Open-domain extractive question answering works well on textual data by first retrieving candidate texts and then extracting the answer from those candidates. However, some questions cannot be answered by text alone but require information stored in tables. In this paper, we present an approach for retrieving both texts and tables relevant to a question by jointly encoding texts, tables and questions into a single vector space. To this end, we create a new multi-modal dataset based on text and table datasets from related work and compare the retrieval performance of different encoding schemata. We find that dense vector embeddings of transformer models outperform sparse embeddings on four out of six evaluation datasets. Comparing different dense embedding models, tri-encoders, with one encoder for each question, text and table, increase retrieval performance compared to bi-encoders with one encoder for the question and one for both text and tables. We release the newly created multi-modal dataset to the community so that it can be used for training and evaluation.

Related papers

• PixT3: Pixel-based Table-To-Text Generation [66.96636025277536]
  We present PixT3, a multimodal table-to-text model that overcomes the challenges of linearization and input size limitations. Experiments on the ToTTo and Logic2Text benchmarks show that PixT3 is competitive and superior to generators that operate solely on text.
  arXiv  Detail & Related papers  (2023-11-16T11:32:47Z)
• UNITE: A Unified Benchmark for Text-to-SQL Evaluation [72.72040379293718]
  We introduce a UNIfied benchmark for Text-to-domain systems. It is composed of publicly available text-to-domain datasets and 29K databases. Compared to the widely used Spider benchmark, we introduce a threefold increase in SQL patterns.
  arXiv  Detail & Related papers  (2023-05-25T17:19:52Z)
• QTSumm: Query-Focused Summarization over Tabular Data [58.62152746690958]
  People primarily consult tables to conduct data analysis or answer specific questions. We define a new query-focused table summarization task, where text generation models have to perform human-like reasoning. We introduce a new benchmark named QTSumm for this task, which contains 7,111 human-annotated query-summary pairs over 2,934 tables.
  arXiv  Detail & Related papers  (2023-05-23T17:43:51Z)
• Multi-View Graph Representation Learning for Answering Hybrid Numerical Reasoning Question [13.321467396155116]
  The paper proposes a Multi-View Graph (MVG) to take the relations among the granularity into account and capture the relations from multiple view. We validate our model on the publicly available table-text hybrid QA benchmark (TAT-QA) and outperform the state-of-the-art model.
  arXiv  Detail & Related papers  (2023-05-05T12:00:58Z)
• Diverse Parallel Data Synthesis for Cross-Database Adaptation of Text-to-SQL Parsers [21.272952382662215]
  Adapting to new databases is a challenging problem due to the lack of natural language queries in the new schemas. We present ReFill, a framework for adapting a Text-to-edit to a target schema.
  arXiv  Detail & Related papers  (2022-10-29T14:30:53Z)
• Answering Numerical Reasoning Questions in Table-Text Hybrid Contents with Graph-based Encoder and Tree-based Decoder [19.429216786198577]
  This paper proposes a textbfRelational textbfGraph enhanced textbfHybrid table-text textbfNumerical reasoning model with textbfRegHNT. It models the numerical question answering over table-text hybrid contents as an expression tree generation task. We validated our model on the publicly available table-text hybrid QA benchmark (TAT-QA)
  arXiv  Detail & Related papers  (2022-09-16T03:15:12Z)
• Table Retrieval May Not Necessitate Table-specific Model Design [83.27735758203089]
  We focus on the task of table retrieval, and ask: "is table-specific model design necessary for table retrieval?" Based on an analysis on a table-based portion of the Natural Questions dataset (NQ-table), we find that structure plays a negligible role in more than 70% of the cases. We then experiment with three modules to explicitly encode table structures, namely auxiliary row/column embeddings, hard attention masks, and soft relation-based attention biases. None of these yielded significant improvements, suggesting that table-specific model design may not be necessary for table retrieval.
  arXiv  Detail & Related papers  (2022-05-19T20:35:23Z)
• Pay More Attention to History: A Context Modeling Strategy for Conversational Text-to-SQL [8.038535788630542]
  One of the most intractable problem of conversational text-to- domain is modeling the semantics of multi-turn queries. This paper shows that explicit modeling the semantic changes by adding each turn and the summarization of the whole context can bring better performance.
  arXiv  Detail & Related papers  (2021-12-16T09:41:04Z)
• Benchmarking Multimodal AutoML for Tabular Data with Text Fields [83.43249184357053]
  We assemble 18 multimodal data tables that each contain some text fields. Our benchmark enables researchers to evaluate their own methods for supervised learning with numeric, categorical, and text features.
  arXiv  Detail & Related papers  (2021-11-04T09:29:16Z)
• Open Question Answering over Tables and Text [55.8412170633547]
  In open question answering (QA), the answer to a question is produced by retrieving and then analyzing documents that might contain answers to the question. Most open QA systems have considered only retrieving information from unstructured text. We present a new large-scale dataset Open Table-and-Text Question Answering (OTT-QA) to evaluate performance on this task.
  arXiv  Detail & Related papers  (2020-10-20T16:48:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.