GCPO: Diagnosing and Constraining Subspace Geometry in Rollout RL for LLMs

• URL: http://arxiv.org/abs/2608.11674v1
• Date: Wed, 12 Aug 2026 05:32:22 GMT
• Title: GCPO: Diagnosing and Constraining Subspace Geometry in Rollout RL for LLMs
• Abstract summary: We introduce Principal-Subspace Overlap, a dimension-corrected measure of individual rollout updates relative to the dominant singular subspaces of pretrained weights.<n>We propose GCPO, which applies hard bilateral projections to constrain updates to the complementary subspaces, preventing such excursions by construction.<n>Our findings provide a new diagnostic lens and a principled design perspective for stable reinforcement learning post-training.
• Score: 40.53507207838462
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: On-policy rollout methods such as GRPO are central to post-training of large language models, yet they frequently suffer from training instabilities, cross-task capability degradation, and response-length inflation. Although prior work has characterized the subspace geometry of aggregate updates, the stepwise variation of this geometry and its relationship to model performance remain unclear. We introduce Principal-Subspace Overlap, a dimension-corrected measure of individual rollout updates relative to the dominant singular subspaces of pretrained weights. Despite low average overlap, transient spikes often precede performance degradation. To address this, we propose GCPO (Geometrically Constrained Policy Optimization), which applies hard bilateral orthogonal projections to constrain updates to the complementary subspaces, preventing such excursions by construction. Across mathematical reasoning, code generation, and tool-use tasks on Qwen3-8B and GLM4-9B, GCPO consistently outperforms GRPO and recent variants, including DAPO and GSPO, improving over the base models and the strongest baseline by up to 27.69 and 2.37 points, respectively. Furthermore, GCPO preserves general capabilities, eliminates response-length inflation, and stabilizes policy entropy. Our findings provide a new diagnostic lens and a principled design perspective for stable reinforcement learning post-training.

Related papers

• Self-Evolving Spatial Reasoning in Vision Language Models via Geometric Logic Consistency [53.28076739140119]
  We propose a framework that enforces logical consistency in Vision-Language Models (VLMs) through geometric and linguistic duality operations.<n>SAGE is model-agnostic, data-efficient compared to prior GRPO methods, and can be applied as a lightweight post-training stage to any existing VLM.<n> Experiments on video and spatial reasoning benchmarks demonstrate consistent improvements over strong baselines.
  arXiv  Detail & Related papers  (2026-05-18T10:05:21Z)
• Power Reinforcement Post-Training of Text-to-Image Models with Super-Linear Advantage Shaping [66.25536973294726]
  Post-training methods for text-to-image (T2I) models are prone to reward hacking.<n>Super-Linear Advantage Shaping (SLAS) reshapes the local policy space.<n>SLAS consistently surpasses the DanceGRPO baseline across multiple backbones and benchmarks.
  arXiv  Detail & Related papers  (2026-05-11T17:59:25Z)
• Beyond Alignment: Expanding Reasoning Capacity via Manifold-Reshaping Policy Optimization [1.974921946982281]
  Reinforcement Learning with Verifiable Rewards (RLVR) has demonstrated remarkable success in enhancing the reasoning capabilities of Large Language Models (LLMs)<n>Recent studies question whether RL genuinely expands reasoning capacity or merely aligns existing latent capabilities, arguing that exploration remains confined within the pre-trained model's low-rank bias manifold.<n>We propose Manifold-Reshaping Policy Optimization (MRPO), a geometric framework designed to fundamentally restructure the inference space of LLMs.
  arXiv  Detail & Related papers  (2026-01-30T05:38:44Z)
• One Ring to Rule Them All: Unifying Group-Based RL via Dynamic Power-Mean Geometry [40.539393367855176]
  Group-based reinforcement learning has evolved from the arithmetic mean of GRPO to the geometric mean of GMPO.<n>We unify these approaches under Power-Mean Policy Optimization (PMPO), a generalized framework that parameterizes the aggregation geometry.<n>We show that adjusting p modulates the concentration of gradient updates, effectively reweighting tokens based on their advantage contribution.
  arXiv  Detail & Related papers  (2026-01-30T03:58:54Z)
• Adaptive-Boundary-Clipping GRPO: Ensuring Bounded Ratios for Stable and Generalizable Training [7.404779700134294]
  Adaptive-Boundary-Clipping GRPO (ABC-GRPO) is an asymmetric and adaptive refinement of the original GRPO framework.<n>ABC-GRPO achieves superior performance over standard GRPO on mathematical reasoning tasks.
  arXiv  Detail & Related papers  (2026-01-07T13:04:52Z)
• Can LLMs Guide Their Own Exploration? Gradient-Guided Reinforcement Learning for LLM Reasoning [44.07085022671951]
  Trajectories that introduce novel gradient directions receive a bounded multiplicative reward scaler.<n>G2RL consistently improves pass@1, maj@16, and pass@k over entropy based GRPO and external embedding methods.
  arXiv  Detail & Related papers  (2025-12-17T18:44:45Z)
• GRPO-Guard: Mitigating Implicit Over-Optimization in Flow Matching via Regulated Clipping [63.33669214116784]
  GRPO-Guard is a simple yet effective enhancement to existing GRPO frameworks.<n>It restores a balanced and step-consistent importance ratio, ensuring that PPO clipping properly constrains harmful updates.<n>It substantially mitigates implicit over-optimization without relying on heavy KL regularization.
  arXiv  Detail & Related papers  (2025-10-25T14:51:17Z)
• PointNSP: Autoregressive 3D Point Cloud Generation with Next-Scale Level-of-Detail Prediction [87.33016661440202]
  Autoregressive point cloud generation has long lagged behind diffusion-based approaches in quality.<n>We propose PointNSP, a coarse-to-fine generative framework that preserves global shape structure at low resolutions.<n> Experiments on ShapeNet show that PointNSP establishes state-of-the-art (SOTA) generation quality for the first time within the autoregressive paradigm.
  arXiv  Detail & Related papers  (2025-10-07T06:31:02Z)
• GeometryZero: Improving Geometry Solving for LLM with Group Contrastive Policy Optimization [63.107398132743825]
  Group Contrastive Policy Optimization (GCPO) is a novel reinforcement learning framework featuring two key innovations.<n>We develop GeometryZero, a family of affordable-size geometric reasoning models that judiciously determine when to employ auxiliary construction.
  arXiv  Detail & Related papers  (2025-06-08T14:18:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.