MatheMagic: Generating Dynamic Mathematics Benchmarks Robust to Memorization

• URL: http://arxiv.org/abs/2510.05962v1
• Date: Tue, 07 Oct 2025 14:19:21 GMT
• Title: MatheMagic: Generating Dynamic Mathematics Benchmarks Robust to Memorization
• Abstract summary: Current benchmarks are prone to overfitting due to having limited diversity of symbols and rules, coupled with closed-ended answers.<n>This paper proposes a method to leverage these shortcomings as useful features to a construct dynamic, counterfactual benchmark.<n>We demonstrate this via MatheMagic, which generates math test instances with the interpretations of numbers and operators altered, yet has automatically verifiable answers.
• Score: 24.225957201610118
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Conducting contamination-free evaluation of mathematical capabilities can be difficult for two reasons: models may memorize a test set once it is made public, and current mathematical benchmarks are prone to overfitting due to having limited diversity of symbols and rules, coupled with closed-ended answers. This paper proposes a method to leverage these shortcomings as useful features to a construct dynamic, counterfactual benchmark, which can be used to both reveal overfitting and measure true reasoning. We demonstrate this via MatheMagic, which generates math test instances with the interpretations of numbers and operators altered, yet has automatically verifiable answers. Test instances are randomly seeded and constructed at test time to evaluate a model's induction or deduction capability, offering stability, extensibility, comparability, and robustness to overfitting. Our experiments find that models solve deduction more easily than induction, but they revert to standard math. Further analysis reveals that math-adapted models fail to exhibit a general "skill" of reasoning, and fine-tuning on induction tasks generalizes poorly.

Related papers

• MathAdv: What Theorem Provers Know, Reason, Formalize, and Generalize [38.37332307303914]
  MathAdv is a diagnostic benchmark spanning 13 domains across undergraduate- and graduate-level mathematics.<n>We show how component-wise evaluation can reveal model capabilities and failure modes that aggregate theorem-proving accuracy obscures.
  arXiv  Detail & Related papers  (2026-08-26T07:12:54Z)
• ReverseMath: Answer Inversion for Scalable and Verifiable Mathematical Problem Generation [49.267054319001346]
  We introduce ReverseMath, a method for generating new math problems through answer inversion.<n>For training, ReverseMath provides automatically labeled reversed problems as data augmentation for reinforcement learning.<n>Experiments show that including ReverseMath-generated data improves mathematical reasoning performance across multiple benchmarks.
  arXiv  Detail & Related papers  (2026-05-26T21:32:47Z)
• LiveMathematicianBench: A Live Benchmark for Mathematician-Level Reasoning with Proof Sketches [61.30693283718321]
  We present LiveMathematicianBench, a dynamic multiple-choice benchmark for research-level mathematical reasoning.<n>By grounding evaluation in newly published theorems, it provides a realistic testbed beyond memorized patterns.<n>It employs a proof-sketchguided distractor pipeline that uses high-level proof strategies to construct plausible but invalid answer choices.
  arXiv  Detail & Related papers  (2026-04-02T08:22:17Z)
• Math Takes Two: A test for emergent mathematical reasoning in communication [1.2891210250935148]
  Math Takes Two is a new benchmark designed to assess the emergence of mathematical reasoning through communication.<n>Motivated by the hypothesis that mathematical cognition in humans co-evolved with the need for precise communication, our benchmark tests whether two agents, without prior mathematical knowledge, can develop a shared symbolic protocol.
  arXiv  Detail & Related papers  (2026-03-30T08:28:48Z)
• Pessimistic Verification for Open Ended Math Questions [6.715841196629822]
  Key limitation of verification performance lies in the ability of error detection.<n>In pessimistic verification we construct multiple parallel verifications for the same proof, and the proof is deemed incorrect if any one of them reports an error.<n>This simple technique significantly improves the performance across many math verification benchmarks without incurring substantial computational resources.
  arXiv  Detail & Related papers  (2025-11-26T15:52:52Z)
• Proof2Hybrid: Automatic Mathematical Benchmark Synthesis for Proof-Centric Problems [9.041749463376599]
  We propose Proof2Hybrid, a framework that synthesizes high-quality, proof-centric benchmarks from natural language mathematical corpora.<n>Our framework and benchmark pave the way for a new wave of in-depth research into the mathematical intelligence of AI systems.
  arXiv  Detail & Related papers  (2025-08-04T08:59:36Z)
• RealMath: A Continuous Benchmark for Evaluating Language Models on Research-Level Mathematics [21.453837660747844]
  Existing benchmarks for evaluating mathematical reasoning in large language models (LLMs) rely primarily on competition problems, formal proofs, or artificially challenging questions.<n>We introduce RealMath, a novel benchmark derived directly from research papers and mathematical forums that assesses LLMs' abilities on authentic mathematical tasks.
  arXiv  Detail & Related papers  (2025-05-18T23:32:46Z)
• MATH-Perturb: Benchmarking LLMs' Math Reasoning Abilities against Hard Perturbations [90.07275414500154]
  We observe significant performance drops on MATH-P-Hard across various models.<n>We also raise concerns about a novel form of memorization where models blindly apply learned problem-solving skills.
  arXiv  Detail & Related papers  (2025-02-10T13:31:46Z)
• Examining False Positives under Inference Scaling for Mathematical Reasoning [83.97128486951999]
  We systematically examine the prevalence of false positive solutions in mathematical problem solving for language models.<n>Our experimental results reveal that: (1) false positive solutions persist across different models, datasets, and decoding methods, (2) sampling-based inference time scaling methods do not alleviate the problem, and (3) the pass@N evaluation metric is more susceptible to false positives.
  arXiv  Detail & Related papers  (2025-02-10T07:49:35Z)
• Data for Mathematical Copilots: Better Ways of Presenting Proofs for Machine Learning [85.635988711588]
  We argue that enhancing the capabilities of large language models requires a paradigm shift in the design of mathematical datasets.<n>We advocate for mathematical dataset developers to consider the concept of "motivated proof", introduced by G. P'olya in 1949, which can serve as a blueprint for datasets that offer a better proof learning signal.<n>We provide a questionnaire designed specifically for math datasets that we urge creators to include with their datasets.
  arXiv  Detail & Related papers  (2024-12-19T18:55:17Z)
• UTMath: Math Evaluation with Unit Test via Reasoning-to-Coding Thoughts [7.856746367263317]
  This paper introduces the UTMath Benchmark, a robust evaluation framework designed to assess Large Language Models.<n>It comprises 1,053 cutting-edge problems spanning nine mathematical domains, with an average of 68 test cases per problem.<n>The best-performing model, o1-mini, solving only 32.57% of the problems, followed by o1-preview at 27.16%, and GPT-4o at 26.93%.
  arXiv  Detail & Related papers  (2024-11-11T18:59:02Z)
• Is Your Model Really A Good Math Reasoner? Evaluating Mathematical Reasoning with Checklist [46.670206614087334]
  We argue that if a model really understands a problem, it should be robustly applied across a diverse array of tasks. MathCheck is a well-designed checklist for testing task generalization and reasoning. MathCheck better reflects true mathematical abilities and represents mathematical intelligence more linearly.
  arXiv  Detail & Related papers  (2024-07-11T17:58:58Z)
• GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solvers [68.77382332826167]
  Large language models (LLMs) have achieved impressive performance across various mathematical reasoning benchmarks. One essential and frequently occurring evidence is that when the math questions are slightly changed, LLMs can behave incorrectly. This motivates us to evaluate the robustness of LLMs' math reasoning capability by testing a wide range of question variations.
  arXiv  Detail & Related papers  (2024-02-29T15:26:14Z)
• InternLM-Math: Open Math Large Language Models Toward Verifiable Reasoning [98.53491178426492]
  We open-source our math reasoning LLMs InternLM-Math which is continue pre-trained from InternLM2. We unify chain-of-thought reasoning, reward modeling, formal reasoning, data augmentation, and code interpreter in a unified seq2seq format. Our pre-trained model achieves 30.3 on the MiniF2F test set without fine-tuning.
  arXiv  Detail & Related papers  (2024-02-09T11:22:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.