Learning by Analogy: Enhancing Few-Shot Prompting for Math Word Problem Solving with Computational Graph-Based Retrieval

• URL: http://arxiv.org/abs/2411.16454v1
• Date: Mon, 25 Nov 2024 15:01:25 GMT
• Title: Learning by Analogy: Enhancing Few-Shot Prompting for Math Word Problem Solving with Computational Graph-Based Retrieval
• Authors: Xiaocong Yang, Jiacheng Lin, Ziqi Wang, Chengxiang Zhai,
• Abstract summary: We present how analogy from similarly structured questions can improve large language models' problem-solving capabilities. Specifically, we rely on the retrieval of problems with similar computational graphs to the given question to serve as exemplars in the prompt. Empirical results across six math word problem datasets demonstrate the effectiveness of our proposed method.
• Score: 22.865124583257987
• License:
• Abstract: Large language models (LLMs) are known to struggle with complicated reasoning tasks such as math word problems (MWPs). In this paper, we present how analogy from similarly structured questions can improve LLMs' problem-solving capabilities for MWPs. Specifically, we rely on the retrieval of problems with similar computational graphs to the given question to serve as exemplars in the prompt, providing the correct reasoning path for the generation model to refer to. Empirical results across six math word problem datasets demonstrate the effectiveness of our proposed method, which achieves a significant improvement of up to 6.7 percent on average in absolute value, compared to baseline methods. These results highlight our method's potential in addressing the reasoning challenges in current LLMs.

Related papers

• Improving Math Problem Solving in Large Language Models Through Categorization and Strategy Tailoring [0.0]
  We develop a machine learning model for problem categorization and show that its accuracy can be significantly improved through the creation of well-designed training datasets. We believe that our approach works by helping reduce hallucinations in LLMs, which is a critical step toward unlocking their potential to tackle advanced mathematical problems.
  arXiv  Detail & Related papers  (2024-10-29T16:06:26Z)
• HARDMath: A Benchmark Dataset for Challenging Problems in Applied Mathematics [1.5716764919736026]
  We introduce HARDMath, a dataset featuring challenging applied mathematics problems that require analytical approximation techniques. Our framework auto-generates a large number of problems with solutions validated against numerical ground truths. We evaluate both open- and closed-source LLMs on HARDMath-mini, a sub-sampled test set of 366 problems, as well as on 40 word problems formulated in applied science contexts.
  arXiv  Detail & Related papers  (2024-10-13T20:09:41Z)
• SIaM: Self-Improving Code-Assisted Mathematical Reasoning of Large Language Models [54.78329741186446]
  We propose a novel paradigm that uses a code-based critic model to guide steps including question-code data construction, quality control, and complementary evaluation. Experiments across both in-domain and out-of-domain benchmarks in English and Chinese demonstrate the effectiveness of the proposed paradigm.
  arXiv  Detail & Related papers  (2024-08-28T06:33:03Z)
• MindStar: Enhancing Math Reasoning in Pre-trained LLMs at Inference Time [51.5039731721706]
  MindStar is a purely inference-based searching method for large language models. It formulates reasoning tasks as searching problems and proposes two search ideas to identify the optimal reasoning paths. It significantly enhances the reasoning abilities of open-source models, such as Llama-2-13B and Mistral-7B, and achieves comparable performance to GPT-3.5 and Grok-1.
  arXiv  Detail & Related papers  (2024-05-25T15:07:33Z)
• Achieving >97% on GSM8K: Deeply Understanding the Problems Makes LLMs Better Solvers for Math Word Problems [50.76385564061713]
  Chain-of-Thought (CoT) prompting has enhanced the performance of Large Language Models (LLMs) across various reasoning tasks. CoT usually suffers from three pitfalls: semantic misunderstanding errors, calculation errors, and step-missing errors. We propose Deeply Understanding the Problems (DUP) to improve the LLMs' math problem-solving ability by addressing semantic misunderstanding errors.
  arXiv  Detail & Related papers  (2024-04-23T12:16:05Z)
• Data Augmentation with In-Context Learning and Comparative Evaluation in Math Word Problem Solving [0.0]
  This study aims to provide MWP solvers with a more diverse training set, ultimately improving their ability to solve various math problems. We propose several methods for data augmentation by modifying the problem texts and equations, such as synonym replacement, rule-based: question replacement, and rule based: reversing question methodologies. This study extends by introducing a new in-context learning augmentation method, employing the Llama-7b language model.
  arXiv  Detail & Related papers  (2024-04-05T07:57:03Z)
• From Large to Tiny: Distilling and Refining Mathematical Expertise for Math Word Problems with Weakly Supervision [12.023661884821554]
  We introduce an innovative two-stage framework that adeptly transfers mathematical Expertise from large to tiny language models. Our method fully leverages the semantic understanding capabilities during the searching 'problem-equation' pair. It demonstrates significantly improved performance on the Math23K and Weak12K datasets compared to existing small model methods.
  arXiv  Detail & Related papers  (2024-03-21T13:29:54Z)
• Benchmarking Hallucination in Large Language Models based on Unanswerable Math Word Problem [58.3723958800254]
  Large language models (LLMs) are highly effective in various natural language processing (NLP) tasks. They are susceptible to producing unreliable conjectures in ambiguous contexts called hallucination. This paper presents a new method for evaluating LLM hallucination in Question Answering (QA) based on the unanswerable math word problem (MWP)
  arXiv  Detail & Related papers  (2024-03-06T09:06:34Z)
• GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solvers [68.77382332826167]
  Large language models (LLMs) have achieved impressive performance across various mathematical reasoning benchmarks. One essential and frequently occurring evidence is that when the math questions are slightly changed, LLMs can behave incorrectly. This motivates us to evaluate the robustness of LLMs' math reasoning capability by testing a wide range of question variations.
  arXiv  Detail & Related papers  (2024-02-29T15:26:14Z)
• VerityMath: Advancing Mathematical Reasoning by Self-Verification Through Unit Consistency [33.760209585322606]
  We study the performance of strong open-source LLMs on math word problems using program-based solving techniques. We propose a systematic approach by defining the units for each quantity and ensuring the consistency of these units during mathematical operations. Our approach, which incorporates unit consistency, currently slightly underperforms compared to an approach that does not.
  arXiv  Detail & Related papers  (2023-11-13T09:06:58Z)
• Evaluating and Improving Tool-Augmented Computation-Intensive Math Reasoning [75.74103236299477]
  Chain-of-thought prompting(CoT) and tool augmentation have been validated as effective practices for improving large language models. We propose a new approach that can deliberate the reasoning steps with tool interfaces, namely textbfDELI. Experimental results on CARP and six other datasets show that the proposed DELI mostly outperforms competitive baselines.
  arXiv  Detail & Related papers  (2023-06-04T17:02:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.