Enhancing Tabular Reasoning with Pattern Exploiting Training

• URL: http://arxiv.org/abs/2210.12259v1
• Date: Fri, 21 Oct 2022 21:28:18 GMT
• Title: Enhancing Tabular Reasoning with Pattern Exploiting Training
• Authors: Abhilash Reddy Shankarampeta and Vivek Gupta and Shuo Zhang
• Abstract summary: Recent methods based on pre-trained language models have exhibited superior performance over tabular tasks. In this work, we utilize Pattern-Exploiting Training (PET) on pre-trained language models to strengthen these reasoning models' pre-existing knowledge and reasoning abilities.
• Score: 14.424742483714846
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent methods based on pre-trained language models have exhibited superior performance over tabular tasks (e.g., tabular NLI), despite showing inherent problems such as not using the right evidence and inconsistent predictions across inputs while reasoning over the tabular data. In this work, we utilize Pattern-Exploiting Training (PET) (i.e., strategic MLM) on pre-trained language models to strengthen these tabular reasoning models' pre-existing knowledge and reasoning abilities. Our upgraded model exhibits a superior understanding of knowledge facts and tabular reasoning compared to current baselines. Additionally, we demonstrate that such models are more effective for underlying downstream tasks of tabular inference on InfoTabs. Furthermore, we show our model's robustness against adversarial sets generated through various character and word level perturbations.

Related papers

• Unleashing the Potential of Large Language Models for Predictive Tabular Tasks in Data Science [17.910306140400046]
  This research endeavors to apply Large Language Models (LLMs) towards addressing these predictive tasks. Our research aims to mitigate this gap by compiling a comprehensive corpus of tables annotated with instructions and executing large-scale training of Llama-2.
  arXiv  Detail & Related papers  (2024-03-29T14:41:21Z)
• Making Pre-trained Language Models Great on Tabular Prediction [50.70574370855663]
  The transferability of deep neural networks (DNNs) has made significant progress in image and language processing. We present TP-BERTa, a specifically pre-trained LM for tabular data prediction. A novel relative magnitude tokenization converts scalar numerical feature values to finely discrete, high-dimensional tokens, and an intra-feature attention approach integrates feature values with the corresponding feature names.
  arXiv  Detail & Related papers  (2024-03-04T08:38:56Z)
• Learning to Reason and Memorize with Self-Notes [51.17609489687686]
  Large language models have been shown to struggle with multi-step reasoning. We propose a simple method for solving both of these problems by allowing the model to take Self-Notes.
  arXiv  Detail & Related papers  (2023-05-01T14:02:48Z)
• Large Language Models with Controllable Working Memory [64.71038763708161]
  Large language models (LLMs) have led to a series of breakthroughs in natural language processing (NLP) What further sets these models apart is the massive amounts of world knowledge they internalize during pretraining. How the model's world knowledge interacts with the factual information presented in the context remains under explored.
  arXiv  Detail & Related papers  (2022-11-09T18:58:29Z)
• PTab: Using the Pre-trained Language Model for Modeling Tabular Data [5.791972449406902]
  Recent studies show that neural-based models are effective in learning contextual representation for Tabular data. We propose a novel framework PTab, using the Pre-trained language model to model Tabular data. Our method has achieved a better average AUC score in supervised settings compared to the state-of-the-art baselines.
  arXiv  Detail & Related papers  (2022-09-15T08:58:42Z)
• Measuring Causal Effects of Data Statistics on Language Model's `Factual' Predictions [59.284907093349425]
  Large amounts of training data are one of the major reasons for the high performance of state-of-the-art NLP models. We provide a language for describing how training data influences predictions, through a causal framework. Our framework bypasses the need to retrain expensive models and allows us to estimate causal effects based on observational data alone.
  arXiv  Detail & Related papers  (2022-07-28T17:36:24Z)
• Is My Model Using The Right Evidence? Systematic Probes for Examining Evidence-Based Tabular Reasoning [26.168211982441875]
  Neural models routinely report state-of-the-art performance across NLP tasks involving reasoning. Our experiments demonstrate that a BERT-based model representative of today's state-of-the-art fails to properly reason on the following counts.
  arXiv  Detail & Related papers  (2021-08-02T01:14:19Z)
• Turning Tables: Generating Examples from Semi-structured Tables for Endowing Language Models with Reasoning Skills [32.55545292360155]
  We propose to leverage semi-structured tables, and automatically generate at scale question-paragraph pairs. We add a pre-training step over this synthetic data, which includes examples that require 16 different reasoning skills. We show that our model, PReasM, substantially outperforms T5, a popular pre-trained encoder-decoder model.
  arXiv  Detail & Related papers  (2021-07-15T11:37:14Z)
• Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Little [74.49773960145681]
  A possible explanation for the impressive performance of masked language model (MLM)-training is that such models have learned to represent the syntactic structures prevalent in NLP pipelines. In this paper, we propose a different explanation: pre-trains succeed on downstream tasks almost entirely due to their ability to model higher-order word co-occurrence statistics. Our results show that purely distributional information largely explains the success of pre-training, and underscore the importance of curating challenging evaluation datasets that require deeper linguistic knowledge.
  arXiv  Detail & Related papers  (2021-04-14T06:30:36Z)
• Leap-Of-Thought: Teaching Pre-Trained Models to Systematically Reason Over Implicit Knowledge [96.92252296244233]
  Large pre-trained language models (LMs) acquire some reasoning capacity, but this ability is difficult to control. We show that LMs can be trained to reliably perform systematic reasoning combining both implicit, pre-trained knowledge and explicit natural language statements. Our work paves a path towards open-domain systems that constantly improve by interacting with users who can instantly correct a model by adding simple natural language statements.
  arXiv  Detail & Related papers  (2020-06-11T17:02:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.