Comprehensive Solution Program Centric Pretraining for Table-and-Text Hybrid Numerical Reasoning

• URL: http://arxiv.org/abs/2305.07475v1
• Date: Fri, 12 May 2023 13:44:40 GMT
• Title: Comprehensive Solution Program Centric Pretraining for Table-and-Text Hybrid Numerical Reasoning
• Authors: Qianying Liu, Dongsheng Yang, Wenjie Zhong, Fei Cheng, Sadao Kurohashi
• Abstract summary: Numerical reasoning over table-and-text hybrid passages, such as financial reports, poses significant challenges. coarse-grained supervision of the whole solution program has impeded the model's ability to learn the underlying numerical reasoning process. We propose three pretraining tasks that operate at both the whole program and sub-program level.
• Score: 21.708394374594082
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Numerical reasoning over table-and-text hybrid passages, such as financial reports, poses significant challenges and has numerous potential applications. Noise and irrelevant variables in the model input have been a hindrance to its performance. Additionally, coarse-grained supervision of the whole solution program has impeded the model's ability to learn the underlying numerical reasoning process. In this paper, we propose three pretraining tasks that operate at both the whole program and sub-program level: Variable Integrity Ranking, which guides the model to focus on useful variables; Variable Operator Prediction, which decomposes the supervision into fine-grained single operator prediction; and Variable Keyphrase Masking, which encourages the model to identify key evidence that sub-programs are derived from. Experimental results demonstrate the effectiveness of our proposed methods, surpassing transformer-based model baselines.

Related papers

• Explanatory Model Monitoring to Understand the Effects of Feature Shifts on Performance [61.06245197347139]
  We propose a novel approach to explain the behavior of a black-box model under feature shifts. We refer to our method that combines concepts from Optimal Transport and Shapley Values as Explanatory Performance Estimation.
  arXiv  Detail & Related papers  (2024-08-24T18:28:19Z)
• Investigating the Role of Instruction Variety and Task Difficulty in Robotic Manipulation Tasks [50.75902473813379]
  This work introduces a comprehensive evaluation framework that systematically examines the role of instructions and inputs in the generalisation abilities of such models. The proposed framework uncovers the resilience of multimodal models to extreme instruction perturbations and their vulnerability to observational changes.
  arXiv  Detail & Related papers  (2024-07-04T14:36:49Z)
• Beyond Self-learned Attention: Mitigating Attention Bias in Transformer-based Models Using Attention Guidance [9.486558126032639]
  We introduce SyntaGuid, a novel approach to guide Transformer-based models towards critical source code tokens. We show that SyntaGuid can improve overall performance up to 3.25% and fix up to 28.3% wrong predictions.
  arXiv  Detail & Related papers  (2024-02-26T18:03:50Z)
• Leveraging Training Data in Few-Shot Prompting for Numerical Reasoning [10.889271604723312]
  Chain-of-thought (CoT) prompting with large language models has proven effective in numerous natural language processing tasks. We investigate two approaches to leverage the training data in a few-shot prompting scenario: dynamic program prompting and program distillation. Our experiments on three standard math word problem (MWP) datasets demonstrate the effectiveness of these approaches.
  arXiv  Detail & Related papers  (2023-05-29T16:01:40Z)
• Explaining Language Models' Predictions with High-Impact Concepts [11.47612457613113]
  We propose a complete framework for extending concept-based interpretability methods to NLP. We optimize for features whose existence causes the output predictions to change substantially. Our method achieves superior results on predictive impact, usability, and faithfulness compared to the baselines.
  arXiv  Detail & Related papers  (2023-05-03T14:48:27Z)
• On the Reliability and Explainability of Language Models for Program Generation [15.569926313298337]
  We study the capabilities and limitations of automated program generation approaches. We employ advanced explainable AI approaches to highlight the tokens that significantly contribute to the code transformation. Our analysis reveals that, in various experimental scenarios, language models can recognize code grammar and structural information, but they exhibit limited robustness to changes in input sequences.
  arXiv  Detail & Related papers  (2023-02-19T14:59:52Z)
• Confident Adaptive Language Modeling [95.45272377648773]
  CALM is a framework for dynamically allocating different amounts of compute per input and generation timestep. We demonstrate the efficacy of our framework in reducing compute -- potential speedup of up to $times 3$ -- while provably maintaining high performance.
  arXiv  Detail & Related papers  (2022-07-14T17:00:19Z)
• Exploring the Trade-off between Plausibility, Change Intensity and Adversarial Power in Counterfactual Explanations using Multi-objective Optimization [73.89239820192894]
  We argue that automated counterfactual generation should regard several aspects of the produced adversarial instances. We present a novel framework for the generation of counterfactual examples.
  arXiv  Detail & Related papers  (2022-05-20T15:02:53Z)
• BLISS: Robust Sequence-to-Sequence Learning via Self-Supervised Input Representation [92.75908003533736]
  We propose a framework-level robust sequence-to-sequence learning approach, named BLISS, via self-supervised input representation. We conduct comprehensive experiments to validate the effectiveness of BLISS on various tasks, including machine translation, grammatical error correction, and text summarization.
  arXiv  Detail & Related papers  (2022-04-16T16:19:47Z)
• SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Generative Capabilities [76.97949110580703]
  We introduce SUPERB-SG, a new benchmark to evaluate pre-trained models across various speech tasks. We use a lightweight methodology to test the robustness of representations learned by pre-trained models under shifts in data domain. We also show that the task diversity of SUPERB-SG coupled with limited task supervision is an effective recipe for evaluating the generalizability of model representation.
  arXiv  Detail & Related papers  (2022-03-14T04:26:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.