Less Is More: Cognitive Load and the Single-Prompt Ceiling in LLM Mathematical Reasoning

• URL: http://arxiv.org/abs/2604.18897v1
• Date: Mon, 20 Apr 2026 22:55:23 GMT
• Title: Less Is More: Cognitive Load and the Single-Prompt Ceiling in LLM Mathematical Reasoning
• Abstract summary: We study prompt engineering for formal mathematical reasoning in the context of the SAIR Equational Theories Stage 1 competition.<n>The task requires deciding whether one equational law implies another over all magmas.<n>Over five weeks, we designed, tested, and analyzed more than 40 prompt variants.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present a systematic empirical study of prompt engineering for formal mathematical reasoning in the context of the SAIR Equational Theories Stage 1 competition. The task requires deciding whether one equational law implies another over all magmas -- a problem that is undecidable in general but decidable for FALSE via finite model search. Over five weeks, we designed, tested, and analyzed more than 40 prompt variants, ranging from 0 to 4,878 bytes, across four evaluation splits and three language models (gpt-oss-120b, Llama 3.3 70B, Gemma 4 31B). Our central finding is a single-prompt ceiling: despite substantial engineering effort, balanced hard accuracy plateaus in an empirical saturation region of approximately 60--79% for gpt-oss-120b, compared to a 59.75% no-cheatsheet baseline. We identify three mechanisms underlying this ceiling: (1) the mathematical undecidability of the TRUE case limits what any finite prompt can encode; (2) complex rule systems decrease performance on weaker models (Llama 3.3 70B collapses to 0% TRUE recall with prompts exceeding 2KB); and (3) prompt ordering effects interact with model attention in fragile, non-monotonic ways. Our best submission (AN45c, 2,252 bytes) achieves 79.25% accuracy on hard3 (n=400; 95% CI: [75.0%, 82.9%]), with TRUE recall of 95.9% and FALSE recall of 63.4%, representing a +19.5 percentage-point improvement over the no-cheatsheet baseline (59.75%). We release all prompt variants, evaluation scripts, and results at https://github.com/israelcazares/sair-prompt-engineering

Related papers

• Capability-Stratified Degradation in Ternary Language Models [0.0]
  Ternary language models restrict weights to $-1,0,+1$, approaching the limit of $log 3 approx 1.585$ bits/weight.<n>We convert Qwen3.5-0.8B (752M parameters) to ternary weights using 72.4M tokens of quantisation-aware training (QAT)<n>The resulting model, Cloe, is evaluated across 29 benchmarks, representation diagnostics, and downstream fine-tuning.
  arXiv  Detail & Related papers  (2026-08-28T19:22:38Z)
• When Does Few-Shot Prompting Help? A Systematic Empirical Study of Shot-Count Effects Across Model Scale, Architecture, and Output Parsing Robustness [0.0]
  Few-shot prompting, the practice of prepending a small number of input-output demonstration pairs to a query before presenting it to a large language model, is widely adopted in NLP.<n>Yet little systematic work investigates how shot count interacts with model scale, architecture, and output format compliance in classification performance.
  arXiv  Detail & Related papers  (2026-07-25T00:31:40Z)
• Automated Proving of Shannon-Type Entropy Inequalities via Fine-Tuned Language Models and Guided Tree Search [50.16356451328644]
  Proving Shannon-type entropy inequalities is a fundamental task in information theory.<n>We investigate whether small-scale large language models, fine-tuned on atomic proof steps, can automate this process.<n>We show that GPT-5.5 solves 1.7% samples under zero-shot prompting while Psitip solves 33.3% samples.
  arXiv  Detail & Related papers  (2026-06-04T05:43:12Z)
• Knowledge Index of Noah's Ark [63.143852586221534]
  We introduce KINA, an 899-item benchmark across 261 fine-grained disciplines.<n>We prove a bonus-on-bar tournament weakly FOSD-dominates flat payment in released-review quality.<n>The top model, Gemini-3.1-Pro-Preview, reaches 53.17%, followed by Claude-Opus-4.6 at 49.92% and GPT-5.4 at 48.55%.
  arXiv  Detail & Related papers  (2026-06-03T17:06:49Z)
• Structured Synthetic Reasoning Data for Arithmetic Fine-Tuning of Small Language Models [0.0]
  We study whether structured synthetic reasoning data can improve arithmetic reasoning in small language models.<n>We generate a 21,250-example corpus of grade-school arithmetic word-problem variants using GPT-5-mini.<n>Fine-tuned models produce shorter reasoning traces, make fewer arithmetic and distractor-use errors, and benefit more consistently from self-consistency sampling.
  arXiv  Detail & Related papers  (2026-05-21T15:50:29Z)
• Universal Transformers Need Memory: Depth-State Trade-offs in Adaptive Recursive Reasoning [0.0]
  We study learned memory tokens as computational scratchpad for a single-block Universal Transformer (UT) on SudokuExtreme.<n>We find that memory tokens are empirically necessary: across all configurations tested, no configuration without memory tokens non-trivial performance.
  arXiv  Detail & Related papers  (2026-04-23T18:30:01Z)
• FregeLogic at SemEval 2026 Task 11: A Hybrid Neuro-Symbolic Architecture for Content-Robust Syllogistic Validity Prediction [0.0017904458681854366]
  We present FregeLogic, a hybrid neuro-symbolic system for SemEval-2026 Task 11 (Subtask 1)<n>Our system achieves 94.3% accuracy with a content effect of 2.85 and a combined score of 41.88 in nested 5-fold cross-validation on the dataset (N=960)<n>Our results suggest that targeted neuro-symbolic integration, applying formal methods precisely where ensemble consensus is lowest, can improve the combined accuracy-plus-content-effect metric used by this task.
  arXiv  Detail & Related papers  (2026-04-20T14:31:00Z)
• One Token Away from Collapse: The Fragility of Instruction-Tuned Helpfulness [12.183451602438753]
  We show that simple lexical constraints (banning a single punctuation character or common word) cause instruction-tuned LLMs to collapse their responses.<n>We show that base models show no systematic collapse under identical constraints, with effects that are small, noisy, and bidirectional.
  arXiv  Detail & Related papers  (2026-04-14T17:40:01Z)
• THEIA: Learning Complete Kleene Three-Valued Logic in a Pure-Neural Modular Architecture [0.0]
  THEIA is a 2.75M modular neural architecture that learns the complete Kleene three-valued logic (K3) truth table from task data without external symbolic inference or hand-encoded K3 gate primitives.<n> Transformer baselines also reach >99% on all 39 rules, and flats match THEIA on Phase-1 accuracy within 0.04pp.
  arXiv  Detail & Related papers  (2026-04-13T10:44:15Z)
• LiveMathematicianBench: A Live Benchmark for Mathematician-Level Reasoning with Proof Sketches [61.30693283718321]
  We present LiveMathematicianBench, a dynamic multiple-choice benchmark for research-level mathematical reasoning.<n>By grounding evaluation in newly published theorems, it provides a realistic testbed beyond memorized patterns.<n>It employs a proof-sketchguided distractor pipeline that uses high-level proof strategies to construct plausible but invalid answer choices.
  arXiv  Detail & Related papers  (2026-04-02T08:22:17Z)
• BeyondBench: Benchmark-Free Evaluation of Reasoning in Language Models [13.380359214677176]
  We introduce BeyondBench, an evaluation framework that avoids contamination from internet-scale training data.<n>Our framework covers 44 algorithmic tasks with a total of 117 variations, grouped into three difficulty levels.<n>We evaluate 101 language models, including 85 open-source and 16 closed-source models, spanning sizes from 0.5B to 141B parameters.
  arXiv  Detail & Related papers  (2025-09-29T02:49:01Z)
• WirelessMathLM: Teaching Mathematical Reasoning for LLMs in Wireless Communications with Reinforcement Learning [51.13280433665446]
  Large language models (LLMs) excel at general mathematical reasoning but fail catastrophically on specialized technical mathematics.<n>In wireless communications, where problems require precise manipulation of information-theoretic bounds, even state-of-the-art models struggle to achieve competent performance.<n>We present WirelessMathLM, demonstrating that compact models (0.5B-7B parameters) can match or exceed much larger models through domain-specific reinforcement learning.
  arXiv  Detail & Related papers  (2025-09-27T09:58:03Z)
• ReasonFlux: Hierarchical LLM Reasoning via Scaling Thought Templates [51.633266497799745]
  hierarchical LLM reasoning via scaling thought templates can effectively optimize the reasoning search space.<n>We introduce three innovations: (i) a structured and generic thought template library, containing around 500 high-level thought templates capable of generalizing to similar or relevant reasoning problems; (ii) performing hierarchical reinforcement learning on a sequence of thought templates instead of long CoTs; and (iii) a brand new inference scaling system.
  arXiv  Detail & Related papers  (2025-02-10T18:51:47Z)
• ProofAug: Efficient Neural Theorem Proving via Fine-grained Proof Structure Analysis [50.020850767257095]
  We propose ProofAug, a procedure that equips LLMs with automation methods at various granularities.<n>Our method is validated on the miniF2F benchmark using the open-source deep-math-7b-base model and the Isabelle proof assistant.<n>We also implement a Lean 4 version of ProofAug that can improve the pass@1 performance of Kimina-Prover-seek-Distill-1.5B from 44.3% to 50.4%.
  arXiv  Detail & Related papers  (2025-01-30T12:37:06Z)
• Not All Votes Count! Programs as Verifiers Improve Self-Consistency of Language Models for Math Reasoning [24.386388107656334]
  We introduce Prove, a framework that leverages translated programs derived from natural language solutions as a verification mechanism.<n>Unlike vanilla majority voting, our approach filters out solutions whose corresponding program output is inconsistent with the generated solution, aggregating only those that pass verification.<n>Our results show that Prove consistently outperforms vanilla majority voting for solving mathematical reasoning tasks across all model sizes and datasets.
  arXiv  Detail & Related papers  (2024-10-16T14:24:55Z)
• Common 7B Language Models Already Possess Strong Math Capabilities [61.61442513067561]
  This paper shows that the LLaMA-2 7B model with common pre-training already exhibits strong mathematical abilities. The potential for extensive scaling is constrained by the scarcity of publicly available math questions.
  arXiv  Detail & Related papers  (2024-03-07T18:00:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.