From Atomic to Composite: Reinforcement Learning Enables Generalization in Complementary Reasoning

• URL: http://arxiv.org/abs/2512.01970v2
• Date: Tue, 02 Dec 2025 04:17:18 GMT
• Title: From Atomic to Composite: Reinforcement Learning Enables Generalization in Complementary Reasoning
• Authors: Sitao Cheng, Xunjian Yin, Ruiwen Zhou, Yuxuan Li, Xinyi Wang, Liangming Pan, William Yang Wang, Victor Zhong,
• Abstract summary: We study Complementary Reasoning, a complex task that requires integrating internal parametric knowledge with external contextual information.<n>We find that RL acts as a reasoning synthesizer rather than a probability amplifier.
• Score: 83.94543243783285
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The mechanism by which RL contributes to reasoning capabilities-whether it incentivizes the synthesis of new skills or merely amplifies existing behaviors-remains a subject of intense debate. In this work, we investigate this question through the lens of Complementary Reasoning, a complex task that requires integrating internal parametric knowledge with external contextual information. Using a controlled synthetic dataset of human biographies, we strictly decouple this ability into two atomic skills: Parametric Reasoning (relying on internal knowledge) and Contextual Reasoning (depending on external information). To rigorously assess capability boundaries, we evaluate generalization across three distinct levels of difficulty: I.I.D., Composition, and Zero-shot settings. We find that while SFT is sufficient for in-distribution performance, it struggles with O.O.D. generalization, particularly in Zero-shot settings where relational combinations are novel. Crucially, we identify the SFT Generalization Paradox: Models supervised solely on the composite task achieve near-perfect in-distribution accuracy but collapse on out-of-distribution generalization, indicating their reliance on rote memorization of path shortcuts. In contrast, we find that RL acts as a reasoning synthesizer rather than a probability amplifier. However, we uncover a strict atomic prerequisite: RL can only synthesize these complex strategies if the base model has first mastered the independent atomic skills (Parametric and Contextual) via SFT. These findings challenge the view of RL as a mere amplifier, suggesting that given sufficient atomic foundations, RL can actively synthesize complex reasoning strategies from learned primitives without explicit supervision on such complex strategies. This indicates that decoupled atomic training followed by RL offers a scalable path to generalization for complex reasoning tasks.

Related papers

• Issues with Measuring Task Complexity via Random Policies in Robotic Tasks [0.005771104869225669]
  Key challenge inReinforcement learning (RL) is measuring task complexity.<n>Few metrics exist for assessing task complexity in non-tabular domains.
  arXiv  Detail & Related papers  (2026-02-21T14:38:02Z)
• Sample-Efficient Neurosymbolic Deep Reinforcement Learning [49.60927398960061]
  We propose a neuro-symbolic Deep RL approach that integrates background symbolic knowledge to improve sample efficiency.<n>Online reasoning is performed to guide the training process through two mechanisms.<n>We show improved performance over a state-of-the-art reward machine baseline.
  arXiv  Detail & Related papers  (2026-01-06T09:28:53Z)
• RL for Reasoning by Adaptively Revealing Rationales [36.50924054394857]
  Supervised fine-tuning (SFT) relies on dense ground-truth labels, which become increasingly costly as sequence length grows.<n>We address this by adaptive backtracking (AdaBack), a per-sample curriculum learning algorithm that reveals only a partial prefix of the target output during training.<n>We show that our adaptive curriculum over partial answers reliably solves problems that are otherwise intractable.
  arXiv  Detail & Related papers  (2025-06-22T17:46:14Z)
• Generalization or Hallucination? Understanding Out-of-Context Reasoning in Transformers [80.70134000599391]
  We argue that both behaviors stem from a single mechanism known as out-of-context reasoning (OCR)<n>OCR drives both generalization and hallucination, depending on whether the associated concepts are causally related.<n>Our work provides a theoretical foundation for understanding the OCR phenomenon, offering a new lens for analyzing and mitigating undesirable behaviors from knowledge injection.
  arXiv  Detail & Related papers  (2025-06-12T16:50:45Z)
• Beyond Accuracy: Dissecting Mathematical Reasoning for LLMs Under Reinforcement Learning [93.00629872970364]
  Reinforcement learning (RL) has become the dominant paradigm for improving the performance of language models on complex reasoning tasks.<n>We introduce SPARKLE, a fine-grained analytic framework to dissect the effects of RL across three key dimensions.<n>We study whether difficult problems -- those yielding no RL signals and mixed-quality reasoning traces -- can still be effectively used for training.
  arXiv  Detail & Related papers  (2025-06-05T07:53:59Z)
• AtomR: Atomic Operator-Empowered Large Language Models for Heterogeneous Knowledge Reasoning [49.24235059299745]
  We introduce AtomR, a framework for large language models to conduct accurate heterogeneous knowledge reasoning at the atomic level.<n>AtomR decomposes a complex question into a reasoning tree where each leaf node corresponds to an atomic knowledge operator.<n>In the reasoning execution stage, AtomR executes each atomic knowledge operator, which flexibly selects, retrieves, and operates atomic level knowledge from heterogeneous sources.
  arXiv  Detail & Related papers  (2024-11-25T15:35:51Z)
• Sparse Mixture-of-Experts for Compositional Generalization: Empirical Evidence and Theoretical Foundations of Optimal Sparsity [89.81738321188391]
  This study investigates the relationship between task complexity and optimal sparsity in SMoE models.<n>We show that the optimal sparsity lies between minimal activation (1-2 experts) and full activation, with the exact number scaling proportionally to task complexity.
  arXiv  Detail & Related papers  (2024-10-17T18:40:48Z)
• Unlock the Correlation between Supervised Fine-Tuning and Reinforcement Learning in Training Code Large Language Models [12.656574142412484]
  We make an attempt to understand the correlation between supervised fine-tuning and reinforcement learning.<n>We find that both atomic and synthetic functions are indispensable for SFT's generalization.
  arXiv  Detail & Related papers  (2024-06-14T03:39:01Z)
• Laying the Foundation First? Investigating the Generalization from Atomic Skills to Complex Reasoning Tasks [40.7766635942194]
  We propose a probing framework to investigate whether the atomic skill can spontaneously generalize to complex reasoning tasks. We then introduce a hierarchical curriculum learning training strategy to achieve better skill generalization. By leveraging hierarchical curriculum learning, we successfully induce generalization, significantly improve the performance of open-source LMs on complex reasoning tasks.
  arXiv  Detail & Related papers  (2024-03-14T15:20:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.