Beyond Trajectory Imitation: Strategy-Guided Policy Optimization for LLM Reasoning
Abstract Overview
This paper argues that reasoning distillation for language models should target reusable problem-solving strategies rather than instance-specific solution trajectories. It proposes Strategy-Guided Policy Optimization (SGPO), which extracts structured strategy descriptions from strong-model responses and uses them to compare the student model’s guided and unguided behavior on the same problem. SGPO combines autonomous GRPO-based exploration with token-level forward-KL distillation, plus proximal constraints and adaptive instance weighting to stabilize training and control when strategy guidance is applied. Experiments are conducted on four mathematical reasoning benchmarks across Qwen2.5 and Llama-3.2 model families.
Novelty
The main novelty is shifting distillation from trajectory imitation to strategy-level transfer, so the student is trained to internalize how strategic guidance changes its own token distribution rather than to copy a teacher’s exact reasoning trace. The method’s forward-KL objective, reachable-target selection, and adaptive weighting together make this a policy optimization framework for strategy distillation rather than standard supervised imitation.
Results
Across four math benchmarks and three base models, SGPO achieves the best average performance among the compared methods, outperforming SFT, SFT+GRPO, HPT, and LUFFY. On Qwen2.5-7B-Instruct, it improves the average score from 49.9 for the strongest baseline to 52.1, and ablations show that removing strategy distillation or the proximal constraints lowers performance substantially. Additional analysis indicates that KL-based distillation is more effective than direct SFT on guided trajectories and that gains tend to increase with stronger base-model capability.
Key Points
- SGPO builds both autonomous and strategy-guided trajectories for each problem, then distills the distributional shift induced by strategy conditioning back into the unguided policy.
- The method uses token-level forward KL, reachable-target selection, KL clipping, and adaptive instance weighting to transfer strategy information while maintaining training stability.
- Empirically, SGPO consistently outperforms trajectory-based baselines on mathematical reasoning benchmarks, and analyses suggest its learning signal focuses on strategy-critical decision points rather than routine tokens.