Are NLP Models really able to Solve Simple Math Word Problems?

• URL: http://arxiv.org/abs/2103.07191v1
• Date: Fri, 12 Mar 2021 10:23:47 GMT
• Title: Are NLP Models really able to Solve Simple Math Word Problems?
• Authors: Arkil Patel, Satwik Bhattamishra, Navin Goyal
• Abstract summary: We show that MWP solvers that do not have access to the question asked in the MWP can still solve a large fraction of MWPs. We introduce a challenge dataset, SVAMP, created by applying carefully chosen variations over sampled from existing datasets. The best accuracy achieved by state-of-the-art models is substantially lower on SVAMP, thus showing that much remains to be done even for the simplest of the MWPs.
• Score: 7.433931244705934
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The problem of designing NLP solvers for math word problems (MWP) has seen sustained research activity and steady gains in the test accuracy. Since existing solvers achieve high performance on the benchmark datasets for elementary level MWPs containing one-unknown arithmetic word problems, such problems are often considered "solved" with the bulk of research attention moving to more complex MWPs. In this paper, we restrict our attention to English MWPs taught in grades four and lower. We provide strong evidence that the existing MWP solvers rely on shallow heuristics to achieve high performance on the benchmark datasets. To this end, we show that MWP solvers that do not have access to the question asked in the MWP can still solve a large fraction of MWPs. Similarly, models that treat MWPs as bag-of-words can also achieve surprisingly high accuracy. Further, we introduce a challenge dataset, SVAMP, created by applying carefully chosen variations over examples sampled from existing datasets. The best accuracy achieved by state-of-the-art models is substantially lower on SVAMP, thus showing that much remains to be done even for the simplest of the MWPs.

Related papers

• Cutting Through the Noise: Boosting LLM Performance on Math Word Problems [52.99006895757801]
  Large Language Models excel at solving math word problems, but struggle with real-world problems containing irrelevant information. We propose a prompting framework that generates adversarial variants of MWPs by adding irrelevant variables. Fine-tuning on adversarial training instances improves performance on adversarial MWPs by 8%.
  arXiv  Detail & Related papers  (2024-05-30T18:07:13Z)
• MWPRanker: An Expression Similarity Based Math Word Problem Retriever [12.638925774492403]
  Math Word Problems (MWPs) in online assessments help test the ability of the learner to make critical inferences. We propose a tool in this work for MWP retrieval.
  arXiv  Detail & Related papers  (2023-07-03T15:44:18Z)
• LogicSolver: Towards Interpretable Math Word Problem Solving with Logical Prompt-enhanced Learning [135.8654475934613]
  We first construct a high-quality MWP dataset named InterMWP which consists of 11,495 MWPs. We propose a novel approach with logical prompt and interpretation, called Logicr. With these improved semantic representations, our Logicr generates corresponding solution expressions and interpretable knowledge in accord with the generated solution expressions.
  arXiv  Detail & Related papers  (2022-05-17T11:01:52Z)
• Unbiased Math Word Problems Benchmark for Mitigating Solving Bias [72.8677805114825]
  Current solvers exist solving bias which consists of data bias and learning bias due to biased dataset and improper training strategy. Our experiments verify MWP solvers are easy to be biased by the biased training datasets which do not cover diverse questions for each problem narrative of all MWPs. An MWP can be naturally solved by multiple equivalent equations while current datasets take only one of the equivalent equations as ground truth.
  arXiv  Detail & Related papers  (2022-05-17T06:07:04Z)
• Adversarial Examples for Evaluating Math Word Problem Solvers [4.266990593059533]
  Math Word Problem (MWP) solvers have achieved high performance on benchmark datasets. The extent to which existing MWP solvers truly understand language and its relation with numbers is still unclear. We generate adversarial attacks to evaluate the robustness of state-of-the-art MWP solvers.
  arXiv  Detail & Related papers  (2021-09-13T12:47:40Z)
• Generate & Rank: A Multi-task Framework for Math Word Problems [48.99880318686938]
  Math word problem (MWP) is a challenging and critical task in natural language processing. We propose Generate & Rank, a framework based on a generative pre-trained language model. By joint training with generation and ranking, the model learns from its own mistakes and is able to distinguish between correct and incorrect expressions.
  arXiv  Detail & Related papers  (2021-09-07T12:21:49Z)
• MWPToolkit: An Open-Source Framework for Deep Learning-Based Math Word Problem Solvers [29.611442087779896]
  MWPToolkit is the first open-source framework for solving Math Word Problem (MWP) solvers. We implement and compare 17 MWP solvers on 4 widely-used single equation generation benchmarks and 2 multiple equations generation benchmarks.
  arXiv  Detail & Related papers  (2021-09-02T09:18:09Z)
• MWP-BERT: A Strong Baseline for Math Word Problems [47.51572465676904]
  Math word problem (MWP) solving is the task of transforming a sequence of natural language problem descriptions to executable math equations. Although recent sequence modeling MWP solvers have gained credits on the math-text contextual understanding, pre-trained language models (PLM) have not been explored for solving MWP. We introduce MWP-BERT to obtain pre-trained token representations that capture the alignment between text description and mathematical logic.
  arXiv  Detail & Related papers  (2021-07-28T15:28:41Z)
• Semantically-Aligned Universal Tree-Structured Solver for Math Word Problems [129.90766822085132]
  A practical automatic textual math word problems (MWPs) solver should be able to solve various textual MWPs. We propose a simple but efficient method called Universal Expression Tree (UET) to make the first attempt to represent the equations of various MWPs uniformly. Then a semantically-aligned universal tree-structured solver (SAU-r) based on an encoder-decoder framework is proposed to resolve multiple types of MWPs in a unified model.
  arXiv  Detail & Related papers  (2020-10-14T06:27:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.