A Careful Examination of Large Language Model Performance on Grade School Arithmetic

• URL: http://arxiv.org/abs/2405.00332v3
• Date: Fri, 3 May 2024 17:53:26 GMT
• Title: A Careful Examination of Large Language Model Performance on Grade School Arithmetic
• Authors: Hugh Zhang, Jeff Da, Dean Lee, Vaughn Robinson, Catherine Wu, Will Song, Tiffany Zhao, Pranav Raja, Dylan Slack, Qin Lyu, Sean Hendryx, Russell Kaplan, Michele Lunati, Summer Yue,
• Abstract summary: Large language models (LLMs) have achieved impressive success on many benchmarks for mathematical reasoning. There is growing concern that some of this performance actually reflects dataset contamination.
• Score: 4.667380916143971
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) have achieved impressive success on many benchmarks for mathematical reasoning. However, there is growing concern that some of this performance actually reflects dataset contamination, where data closely resembling benchmark questions leaks into the training data, instead of true reasoning ability. To investigate this claim rigorously, we commission Grade School Math 1000 (GSM1k). GSM1k is designed to mirror the style and complexity of the established GSM8k benchmark, the gold standard for measuring elementary mathematical reasoning. We ensure that the two benchmarks are comparable across important metrics such as human solve rates, number of steps in solution, answer magnitude, and more. When evaluating leading open- and closed-source LLMs on GSM1k, we observe accuracy drops of up to 13%, with several families of models (e.g., Phi and Mistral) showing evidence of systematic overfitting across almost all model sizes. At the same time, many models, especially those on the frontier, (e.g., Gemini/GPT/Claude) show minimal signs of overfitting. Further analysis suggests a positive relationship (Spearman's r^2=0.32) between a model's probability of generating an example from GSM8k and its performance gap between GSM8k and GSM1k, suggesting that many models may have partially memorized GSM8k.

Related papers

• GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models [24.266973481633755]
  The GSM8K benchmark is widely used to assess the mathematical reasoning of models on grade-school-level questions. We introduce GSM-Symbolic, an improved benchmark created from symbolic templates. Our findings reveal that LLMs exhibit noticeable variance when responding to different instantiations of the same question.
  arXiv  Detail & Related papers  (2024-10-07T17:36:37Z)
• Common 7B Language Models Already Possess Strong Math Capabilities [61.61442513067561]
  This paper shows that the LLaMA-2 7B model with common pre-training already exhibits strong mathematical abilities. The potential for extensive scaling is constrained by the scarcity of publicly available math questions.
  arXiv  Detail & Related papers  (2024-03-07T18:00:40Z)
• Chain-of-Thought Unfaithfulness as Disguised Accuracy [0.0]
  Chain-of-Thought (CoT) generations align with a large language model's (LLM) internal computations. A metric that measures a model's dependence on its CoT for producing an answer is proposed.
  arXiv  Detail & Related papers  (2024-02-22T17:23:53Z)
• Orca-Math: Unlocking the potential of SLMs in Grade School Math [10.206509967833664]
  A recent study hypothesized that the smallest model size, needed to achieve over 80% accuracy on the GSM8K benchmark, is 34 billion parameters. To reach this level of performance with smaller models, researcher often train SLMs to generate Python code or use tools to help avoid calculation errors. Our approach has the following key elements: A high quality synthetic dataset of 200K math problems created using a multi-agent setup where agents collaborate to create the data.
  arXiv  Detail & Related papers  (2024-02-16T23:44:38Z)
• MR-GSM8K: A Meta-Reasoning Benchmark for Large Language Model Evaluation [60.65820977963331]
  We introduce a novel evaluation paradigm for Large Language Models (LLMs) This paradigm shifts the emphasis from result-oriented assessments, which often neglect the reasoning process, to a more comprehensive evaluation. By applying this paradigm in the GSM8K dataset, we have developed the MR-GSM8K benchmark.
  arXiv  Detail & Related papers  (2023-12-28T15:49:43Z)
• TinyGSM: achieving >80% on GSM8k with small language models [49.21136294791747]
  Small-scale models offer various computational advantages, and yet to which extent size is critical for problem-solving abilities remains an open question. Specifically for solving grade school math, the smallest model size so far required to break the 80% barrier on the GSM8K benchmark remains to be 34B. Our work studies how high-quality datasets may be the key for small language models to acquire mathematical reasoning.
  arXiv  Detail & Related papers  (2023-12-14T18:58:28Z)
• MuggleMath: Assessing the Impact of Query and Response Augmentation on Math Reasoning [54.2093509928664]
  In math reasoning with large language models, fine-tuning data augmentation by query evolution and diverse reasoning paths is empirically verified effective. We conduct an investigation for such data augmentation in math reasoning and are intended to answer these questions. We release our codes and augmented data in https://github.com/OFA-Sys/8k-Scel.
  arXiv  Detail & Related papers  (2023-10-09T08:18:58Z)
• Holistic Evaluation of Language Models [183.94891340168175]
  Language models (LMs) are becoming the foundation for almost all major language technologies, but their capabilities, limitations, and risks are not well understood. We present Holistic Evaluation of Language Models (HELM) to improve the transparency of language models.
  arXiv  Detail & Related papers  (2022-11-16T18:51:34Z)
• Language Models are Multilingual Chain-of-Thought Reasoners [83.37148309771378]
  We introduce the Multilingual Grade School Math benchmark, by manually translating 250 grade-school math problems into ten typologically diverse languages. We find that the ability to solve MGSM problems via chain-of-thought prompting emerges with increasing model scale. We show that the multilingual reasoning abilities of language models extend to other tasks such as commonsense reasoning and word-in-context semantic judgment.
  arXiv  Detail & Related papers  (2022-10-06T17:03:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.