Placing Puzzle Pieces Where They Matter: A Question Augmentation Framework for Reinforcement Learning

• URL: http://arxiv.org/abs/2604.15830v1
• Date: Fri, 17 Apr 2026 08:34:51 GMT
• Title: Placing Puzzle Pieces Where They Matter: A Question Augmentation Framework for Reinforcement Learning
• Abstract summary: Reinforcement learning has become a powerful approach for enhancing large language model reasoning, but faces a fundamental dilemma.<n>We propose textbfPieceHint, a hint injection framework that strategically identifies and provides critical reasoning steps during training.
• Score: 1.338679434374531
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reinforcement learning has become a powerful approach for enhancing large language model reasoning, but faces a fundamental dilemma: training on easy problems can cause overfitting and pass@k degradation, while training on hard problems often results in sparse rewards. Recent question augmentation methods address this by prepending partial solutions as hints. However, uniform hint provision may introduce redundant information while missing critical reasoning bottlenecks, and excessive hints can reduce reasoning diversity, causing pass@k degradation. We propose \textbf{PieceHint}, a hint injection framework that strategically identifies and provides critical reasoning steps during training. By scoring the importance of different reasoning steps, selectively allocating hints based on problem difficulty, and progressively withdrawing scaffolding, PieceHint enables models to transition from guided learning to independent reasoning. Experiments on six mathematical reasoning benchmarks show that our 1.5B model achieves comparable average performance to 32B baselines while preserving pass@k diversity across all $k$ values.

Related papers

• Stabilizing Reinforcement Learning for Honesty Alignment in Language Models on Deductive Reasoning [27.42733470720954]
  We propose a reinforcement learning method that injects ground truth trajectories into rollouts, preventing early training collapse.<n>Our results demonstrate that this method stabilizes learning and significantly improves the overall reasoning performance.
  arXiv  Detail & Related papers  (2025-11-12T11:34:19Z)
• Learning a Dense Reasoning Reward Model from Expert Demonstration via Inverse Reinforcement Learning [50.20267980386502]
  We learn a dense, token-level reward model for process supervision directly from expert demonstrations.<n>The learned reasoning reward serves two complementary roles: (i) it provides step-level feedback to optimise a reasoning policy during training; and (ii) it functions at inference as a critic to rerank sampled traces under fixed compute budgets.
  arXiv  Detail & Related papers  (2025-10-02T09:55:26Z)
• Less is More Tokens: Efficient Math Reasoning via Difficulty-Aware Chain-of-Thought Distillation [82.2288581878096]
  We present a framework for difficulty-aware reasoning that teaches models to dynamically adjust reasoning depth based on problem complexity.<n>We show that models can be endowed with such dynamic inference pathways without any architectural modifications.
  arXiv  Detail & Related papers  (2025-09-05T16:40:13Z)
• Beyond Scaling Law: A Data-Efficient Distillation Framework for Reasoning [10.186434946738201]
  Large language models (LLMs) demonstrate remarkable reasoning capabilities in tasks such as algorithmic coding and mathematical problem-solving.<n>Recent methods have improved reasoning through expanded corpus and multistage training combining reinforcement learning and supervised fine-tuning.
  arXiv  Detail & Related papers  (2025-08-13T15:32:25Z)
• Decoupling Understanding from Reasoning via Problem Space Mapping for Small-scale Model Reasoning [22.582715282848795]
  We propose a new framework that decouples understanding from reasoning by mapping natural language problems into a canonical problem space.<n>Within this framework, we introduce DURIT, a three-step algorithm that iteratively aligns reasoning trajectories through self-distillation.<n>Experiments show that DURIT substantially improves SLMs' performance on both in-domain and out-of-domain mathematical and logical reasoning tasks.
  arXiv  Detail & Related papers  (2025-08-07T01:13:30Z)
• Fractional Reasoning via Latent Steering Vectors Improves Inference Time Compute [60.151643048803145]
  We propose Fractional Reasoning, a framework that enables continuous control over reasoning intensity at inference time.<n>Our method operates by extracting the latent steering vector associated with deeper reasoning and reapplying it with a tunable scaling factor.<n> Experiments on GSM8K, MATH500, and GPQA demonstrate that Fractional Reasoning consistently improves performance across diverse reasoning tasks and models.
  arXiv  Detail & Related papers  (2025-06-18T21:15:59Z)
• The Price of a Second Thought: On the Evaluation of Reasoning Efficiency in Large Language Models [54.88805865447848]
  We show that instruct models achieve higher efficiency overall, and problem difficulty affects efficiency.<n>We propose COTHINK, a simple two-stage pipeline: an instruct model drafts a brief outline, and a thinking model expands it.<n>On GSM8K, MATH500, and AIME24, COTHINK cuts token usage by 21.1% while keeping accuracy on four thinking models, and remains competitive with strong efficiency baselines.
  arXiv  Detail & Related papers  (2025-05-28T06:24:45Z)
• Unlocking the Potential of Difficulty Prior in RL-based Multimodal Reasoning [69.64809103333839]
  We investigate how explicitly modeling problem's difficulty prior information shapes the effectiveness of reinforcement learning based fine-tuning for multimodal reasoning.<n>Our approach demonstrates significant performances across various multi-modal mathematical reasoning benchmarks with only 2K+0.6K two-stage training data.
  arXiv  Detail & Related papers  (2025-05-19T15:43:10Z)
• Reasoning Paths Optimization: Learning to Reason and Explore From Diverse Paths [69.39559168050923]
  We introduce Reasoning Paths Optimization (RPO), which enables learning to reason and explore from diverse paths. Our approach encourages favorable branches at each reasoning step while penalizing unfavorable ones, enhancing the model's overall problem-solving performance. We focus on multi-step reasoning tasks, such as math word problems and science-based exam questions.
  arXiv  Detail & Related papers  (2024-10-07T06:37:25Z)
• Flow of Reasoning: Training LLMs for Divergent Reasoning with Minimal Examples [12.48027669682156]
  Flow of Reasoning (FoR) aims at improving diversity with minimal data.<n>FoR formulates multi-step LLM reasoning as a Markovian flow on a DAG-structured reasoning graph.<n>Experiments show that, with limited training examples, FoR enables the discovery of diverse, creative, high-quality solutions.
  arXiv  Detail & Related papers  (2024-06-09T07:06:58Z)
• DCR: Divide-and-Conquer Reasoning for Multi-choice Question Answering with LLMs [9.561022942046279]
  We propose Divide and Conquer Reasoning (DCR) to enhance the reasoning capability of large language models (LLMs) We first categorize questions into two subsets based on confidence score ($mathcalCS$), which is estimated by statistical frequency of generated answers. In particular, we first categorize questions into two subsets based on confidence score ($mathcalCS$), which is estimated by statistical frequency of generated answers.
  arXiv  Detail & Related papers  (2024-01-10T14:38:46Z)
• Learning by Fixing: Solving Math Word Problems with Weak Supervision [70.62896781438694]
  Previous neural solvers of math word problems (MWPs) are learned with full supervision and fail to generate diverse solutions. We introduce a textitweakly-supervised paradigm for learning MWPs. Our method only requires the annotations of the final answers and can generate various solutions for a single problem.
  arXiv  Detail & Related papers  (2020-12-19T03:10:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.