From Reasoning Traces to Reusable Modules: Understanding Compositional Generalization in Language Model Reasoning
Abstract Overview
This paper studies why post-training with supervised fine-tuning (SFT) followed by reinforcement learning (RL) improves reasoning generalization in language models. It frames reasoning traces as outputs of a hierarchical latent selection model, where reusable atomic modules include both local skills and routing mechanisms that determine how intermediate results are reused and composed. Within this framework, the paper argues that SFT mainly provides compositional traces containing the raw module material, while RL helps disentangle those traces into identifiable atomic modules by enriching reward-relevant local events. The claims are supported by theory on latent structure identification and module recomposition, along with controlled synthetic experiments and a complementary analysis of open-source SFT/RL model pairs.
Novelty
The paper's main novelty is a unified theoretical and empirical account of compositional generalization in language-model reasoning based on reusable atomic skills and routing mechanisms. It also proposes a specific division of labor between SFT and RL, and studies how the relationship between their training supports affects generalization.
Results
In controlled string-transformation experiments, RL trained on compound traces improved atomic-task accuracy over SFT-only training and delivered much larger gains on unseen compositions than on seen ones. RL on compound traces outperformed RL on isolated atomic modules for out-of-distribution composition, and re-injecting compositions containing held-out skills or routing mechanisms restored performance more effectively than isolated atom-only data. The best unseen generalization was observed when SFT covered the atomic inventory and RL focused on compositions outside the SFT support, with complementary evidence from open-source models showing richer short skill-pattern recombination after RL.
Key Points
- The paper models reasoning traces as compositions of reusable atomic skills and routing mechanisms within a hierarchical latent selection framework.
- It argues that SFT supplies compositional trace material while RL decomposes those traces into reusable modules by exposing identification-critical local events.
- Experiments indicate that compound-trace training and an SFT/RL curriculum with broad SFT atom coverage plus RL exploration beyond SFT support are important for compositional generalization.