Table-To-Text generation and pre-training with TabT5
- URL: http://arxiv.org/abs/2210.09162v1
- Date: Mon, 17 Oct 2022 15:05:53 GMT
- Title: Table-To-Text generation and pre-training with TabT5
- Authors: Ewa Andrejczuk, Julian Martin Eisenschlos, Francesco Piccinno, Syrine
Krichene, Yasemin Altun
- Abstract summary: We present TABT5, an encoder-decoder model that generates natural language text based on tables and textual inputs.
TABT5 overcomes the encoder-only limitation by incorporating a decoder component and leverages the input structure with table specific embeddings and pre-training.
- Score: 11.456825732936538
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Encoder-only transformer models have been successfully applied to different
table understanding tasks, as in TAPAS (Herzig et al., 2020). A major
limitation of these architectures is that they are constrained to
classification-like tasks such as cell selection or entailment detection. We
present TABT5, an encoder-decoder model that generates natural language text
based on tables and textual inputs. TABT5 overcomes the encoder-only limitation
by incorporating a decoder component and leverages the input structure with
table specific embeddings and pre-training. TABT5 achieves new state-of-the-art
results on several domains, including spreadsheet formula prediction with a 15%
increase in sequence accuracy, QA with a 2.5% increase in sequence accuracy and
data-to-text generation with a 2.5% increase in BLEU.
Related papers
- Making Pre-trained Language Models Great on Tabular Prediction [50.70574370855663]
The transferability of deep neural networks (DNNs) has made significant progress in image and language processing.
We present TP-BERTa, a specifically pre-trained LM for tabular data prediction.
A novel relative magnitude tokenization converts scalar numerical feature values to finely discrete, high-dimensional tokens, and an intra-feature attention approach integrates feature values with the corresponding feature names.
arXiv Detail & Related papers (2024-03-04T08:38:56Z) - A Text-to-Text Model for Multilingual Offensive Language Identification [19.23565690468299]
This study presents the first pre-trained model with encoder-decoder architecture for offensive language identification with text-to-text transformers (T5)
Our pre-trained T5 model outperforms other transformer-based models fine-tuned for offensive language detection, such as fBERT and HateBERT, in multiple English benchmarks.
Following a similar approach, we also train the first multilingual pre-trained model for offensive language identification using mT5.
arXiv Detail & Related papers (2023-12-06T09:37:27Z) - Graphix-T5: Mixing Pre-Trained Transformers with Graph-Aware Layers for
Text-to-SQL Parsing [56.232873134174056]
One of the major challenges in text-to-text parsing is domain generalization, i.e., how to well generalize to unseen databases.
In this work, we explore ways to further augment the pre-trained text-to-text transformer model with specialized components for text-to-text parsing.
To this end, we propose a new architecture GRAPHIX-T5, augmented by some specially-designed graph-aware model with layers.
arXiv Detail & Related papers (2023-01-18T13:29:05Z) - Improving Cross-task Generalization of Unified Table-to-text Models with
Compositional Task Configurations [63.04466647849211]
Methods typically encode task information with a simple dataset name as a prefix to the encoder.
We propose compositional task configurations, a set of prompts prepended to the encoder to improve cross-task generalization.
We show this not only allows the model to better learn shared knowledge across different tasks at training, but also allows us to control the model by composing new configurations.
arXiv Detail & Related papers (2022-12-17T02:20:14Z) - Text Embeddings by Weakly-Supervised Contrastive Pre-training [98.31785569325402]
E5 is a family of state-of-the-art text embeddings that transfer well to a wide range of tasks.
E5 can be readily used as a general-purpose embedding model for any tasks requiring a single-vector representation of texts.
arXiv Detail & Related papers (2022-12-07T09:25:54Z) - EdiT5: Semi-Autoregressive Text-Editing with T5 Warm-Start [21.4394742421462]
EdiT5 is a novel semi-autoregressive text-editing approach.
It combines the strengths of non-autoregressive text-editing and autoregressive decoding.
arXiv Detail & Related papers (2022-05-24T17:13:22Z) - Parameter-Efficient Abstractive Question Answering over Tables or Text [60.86457030988444]
A long-term ambition of information seeking QA systems is to reason over multi-modal contexts and generate natural answers to user queries.
Memory intensive pre-trained language models are adapted to downstream tasks such as QA by fine-tuning the model on QA data in a specific modality like unstructured text or structured tables.
To avoid training such memory-hungry models while utilizing a uniform architecture for each modality, parameter-efficient adapters add and train small task-specific bottle-neck layers between transformer layers.
arXiv Detail & Related papers (2022-04-07T10:56:29Z) - Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text
Models [10.645591218689058]
We provide the first exploration of text-to-text transformers (T5) sentence embeddings.
We investigate three methods for extracting T5 sentence embeddings.
Our encoder-only models outperforms BERT-based sentence embeddings on both transfer tasks and semantic textual similarity.
arXiv Detail & Related papers (2021-08-19T18:58:02Z) - NT5?! Training T5 to Perform Numerical Reasoning [0.8827543048499855]
Numerical reasoning over text (NRoT) presents unique challenges that are not well addressed by existing pre-training objectives.
We show that training the T5 multitasking framework with multiple numerical reasoning datasets of increasing difficulty can be achieved without manually engineering partitioned functionality.
arXiv Detail & Related papers (2021-04-15T08:34:44Z) - mT5: A massively multilingual pre-trained text-to-text transformer [60.0210636815514]
"Text-to-Text Transfer Transformer" (T5) leveraged a unified text-to-text format and scale to attain state-of-the-art results on English-language NLP tasks.
We introduce mT5, a multilingual variant of T5 that was pre-trained on a new Common Crawl-based dataset covering 101 languages.
arXiv Detail & Related papers (2020-10-22T17:58:14Z) - Understanding tables with intermediate pre-training [11.96734018295146]
We adapt TAPAS, a table-based BERT model, to recognize entailment.
We evaluate table pruning techniques as a pre-processing step to drastically improve the training and prediction efficiency.
arXiv Detail & Related papers (2020-10-01T17:43:27Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.