Reinforcement Learning for Compositional Generalization with Outcome-Level Optimization

• URL: http://arxiv.org/abs/2605.04920v1
• Date: Wed, 06 May 2026 13:47:31 GMT
• Title: Reinforcement Learning for Compositional Generalization with Outcome-Level Optimization
• Abstract summary: Existing approaches often rely on supervised fine-tuning, which encourages models to imitate target outputs.<n>In this work, we investigate whether compositional generalization can instead be improved through outcome-level reinforcement learning.
• Score: 11.743149066177642
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Compositional generalization refers to correctly interpret novel combinations of known primitives, which remains a major challenge. Existing approaches often rely on supervised fine-tuning, which encourages models to imitate target outputs. This token-level training paradigm fails to capture the global compositional structure required for generalizing to unseen combinations. In this work, we investigate whether compositional generalization can instead be improved through outcome-level reinforcement learning. We adopt Group Relative Policy Optimization to optimize models based on feedback on their final outputs. Within this framework, we explore both a simple binary outcome reward and a composite reward that provides additional composition feedback. Experiments on multiple compositional benchmarks show that reinforcement learning improves compositional generalization compared to supervised fine-tuning. Further analysis reveals that supervised models tend to overfit frequent training compositions, whereas reinforcement learning improves compositional generalization by reshaping the output distribution, particularly for more complex composition types.

Related papers

• TPR-Attention for Combinatorial Generalization [1.1688030627514532]
  We introduce a new architectural component that embeds structured inductive bias into deep learning.<n>We show that this TPR-attention mechanism outperforms existing architectural components in generalization.
  arXiv  Detail & Related papers  (2026-08-31T01:18:44Z)
• Composition-Incremental Learning for Compositional Generalization [35.44592461934844]
  An ideal model is supposed to gradually improve the capability of compositional generalization in an incremental manner.<n>We develop a benchmark construction pipeline leveraging existing datasets, yielding MIT-States-CompIL and C-GQA-CompIL.<n>We propose a pseudo-replay framework utilizing a visual synthesizer to synthesize visual representations of learned compositions and a linguistic primitive distillation mechanism to maintain aligned primitive representations.
  arXiv  Detail & Related papers  (2025-11-12T08:00:03Z)
• Learning to Substitute Components for Compositional Generalization [70.96410435337967]
  We propose a novel compositional augmentation strategy called CompSub, which enables multi-grained composition of substantial substructures.<n>We also introduce the Learning Component Substitution (LCS) framework, which empowers the learning of component substitution probabilities in CompSub.<n>Our results demonstrate the superiority of CompSub, LCS, and LCS-ICL, with improvements of up to 66.5%, 10.3%, 1.4%, and 8.8%, respectively.
  arXiv  Detail & Related papers  (2025-02-28T08:30:47Z)
• Compositional Generalization from First Principles [27.243195680442533]
  We investigate compositionality as a property of the data-generating process rather than the data itself. This reformulation enables us to derive mild conditions on only the support of the training distribution and the model architecture. Our results set the stage for a principled theoretical study of compositional generalization.
  arXiv  Detail & Related papers  (2023-07-10T19:30:32Z)
• Compositional Generalisation with Structured Reordering and Fertility Layers [121.37328648951993]
  Seq2seq models have been shown to struggle with compositional generalisation. We present a flexible end-to-end differentiable neural model that composes two structural operations.
  arXiv  Detail & Related papers  (2022-10-06T19:51:31Z)
• Compositional Generalization Requires Compositional Parsers [69.77216620997305]
  We compare sequence-to-sequence models and models guided by compositional principles on the recent COGS corpus. We show structural generalization is a key measure of compositional generalization and requires models that are aware of complex structure.
  arXiv  Detail & Related papers  (2022-02-24T07:36:35Z)
• Grounded Graph Decoding Improves Compositional Generalization in Question Answering [68.72605660152101]
  Question answering models struggle to generalize to novel compositions of training patterns, such as longer sequences or more complex test structures. We propose Grounded Graph Decoding, a method to improve compositional generalization of language representations by grounding structured predictions with an attention mechanism. Our model significantly outperforms state-of-the-art baselines on the Compositional Freebase Questions (CFQ) dataset, a challenging benchmark for compositional generalization in question answering.
  arXiv  Detail & Related papers  (2021-11-05T17:50:14Z)
• Disentangled Sequence to Sequence Learning for Compositional Generalization [62.954842223732435]
  We propose an extension to sequence-to-sequence models which allows us to learn disentangled representations by adaptively re-encoding the source input. Experimental results on semantic parsing and machine translation empirically show that our proposal yields more disentangled representations and better generalization.
  arXiv  Detail & Related papers  (2021-10-09T22:27:19Z)
• Improving Compositional Generalization in Semantic Parsing [54.4720965813889]
  Generalization of models to out-of-distribution (OOD) data has captured tremendous attention recently. We investigate compositional generalization in semantic parsing, a natural test-bed for compositional generalization.
  arXiv  Detail & Related papers  (2020-10-12T12:34:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.