Strategically Diverse Sampling for Self-Training
Abstract Overview
This paper investigates self-training data construction for large language models, arguing that repeated samples should vary in problem-solving approach rather than merely in surface form or correctness. The authors introduce two strategy-oriented sampling techniques: Groot, which builds a hierarchical tree of approaches to sample distinct decision paths, and an approach-level adaptation of Verbalized Sampling (VS), which elicits multiple alternative approaches before generating full responses. They evaluate these methods on competitive programming and next-chapter prediction, assessing direct generation performance, test-time aggregation, and reinforcement learning initialization. Across both domains, fine-tuning on strategically diverse data delivers superior performance on difficult tasks compared to IID self-training, even when IID sampling uses elevated temperature, larger sampling budgets, or distillation from a much larger teacher model.
Novelty
The paper establishes strategic diversity of solution approaches as a primary design principle for constructing self-training data rather than relying strictly on correctness filtering or teacher scale. Methodologically, it introduces Groot for hierarchical approach generation and adapts Verbalized Sampling to separate strategic planning from solution execution during training data synthesis.
Results
On frontier coding benchmarks, models trained on Groot or VS data substantially exceed IID-trained baselines on pass@k metrics and serve as superior initializations for both Recursive Self-Aggregation and downstream RL. In next-chapter prediction, strategic diversity training significantly increases the coverage of high-quality narrative plans where standard IID training offers minimal gain over the base model. Furthermore, self-training on strategically diverse but incorrect traces from a 4B student model outperforms IID distillation from a 235B teacher model.
Key Points
- Groot and VS explicitly diversify problem-solving strategies prior to full solution generation, eliciting distinct decision paths rather than surface-level variants of a single approach.
- Performance improvements are pronounced on difficult frontier tasks, where strategic sampling outperforms standard IID sampling scaled with higher temperatures or up to 16x larger generation budgets.
- Strategic diversity proves more critical than strict correctness or teacher scale, as student-generated diverse incorrect traces and diverse self-training outperform standard IID data from a 235B teacher model.