GEOALIGN: Geometric Rollout Curation for Robust LLM Reinforcement Learning

• URL: http://arxiv.org/abs/2606.26917v1
• Date: Thu, 25 Jun 2026 11:53:37 GMT
• Title: GEOALIGN: Geometric Rollout Curation for Robust LLM Reinforcement Learning
• Abstract summary: Geoalign is a plug-in for rollout curation in iterative policy optimization.<n>It forms within-prompt preference pairs, learns an online projector on per-rollout hidden states to concentrate reward-ordered displacement directions, and rectifies them with within-prompt stable alternatives.<n>Across dialogue alignment with a learned reward model and mathematical reasoning with binary verified rewards, Geoalign improves final performance and reduces training oscillation.
• Score: 25.144030941321237
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Online reinforcement learning is widely used to align large language models (LLMs) with reward signals, yet training can be unstable under noisy or misspecified rewards. We identify a failure mode we call directional inconsistency: within a batch, a small set of high-reward rollouts induces representation-space preference directions that sharply disagree with the batch majority, resulting in high-variance and destabilizing updates. We propose geoalign, a lightweight plug-in for rollout curation in iterative policy optimization. Geoalign (i) forms within-prompt preference pairs, (ii) learns an online projector on per-rollout hidden states to concentrate reward-ordered displacement directions, and (iii) detects directionally inconsistent rollouts via their angular deviation from a batch consensus prototype and rectifies them with within-prompt stable alternatives. Geoalign is forward-pass only and adds negligible overhead. Across dialogue alignment with a learned reward model and mathematical reasoning with binary verified rewards, Geoalign improves final performance and reduces training oscillation, outperforming PF-PPO, PAR, PODS, and Seed-GRPO. These results suggest latent directional consensus as an effective reliability signal for online LLM RL.

Related papers

• LEAP: Lean Environment-Feedback via Adaptive Pruning for Code RL in GPU Kernel Generation [3.4535054007206636]
  Post-training large language models (LLMs) via reinforcement learning (RL) has significantly advanced code generation capabilities.<n>Current state-of-the-art frameworks leverage critic-free paradigms like Group Relative Policy Optimization (GRPO) tied to rule-based verification sandboxes.<n>We introduce LEAPLean Environment-Feedback via Adaptive Pruning, a scalable and computationally efficient multi-turn framework optimized for low-level hardware accelerator alignment.
  arXiv  Detail & Related papers  (2026-08-03T07:12:52Z)
• What and When to Distill: Selective Hindsight Distillation for Multi-Turn Agents [70.6980022118038]
  Reinforcement learning can train LLM agents from sparse task rewards, but long-horizon credit assignment remains challenging.<n>Existing methods rely on trajectory-level rewards or proxy signals, without fully leveraging per-step environmental feedback.<n>We introduce SERL, a selective environment-reweighted learning framework.
  arXiv  Detail & Related papers  (2026-05-19T07:00:55Z)
• Learning from Failures: Correction-Oriented Policy Optimization with Verifiable Rewards [73.44333771806282]
  Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as an effective paradigm for improving the reasoning capabilities of large language models.<n>We propose Correction-Oriented Policy Optimization (CIPO), a simple and effective extension to RLVR that converts on-policy failed trajectories into correction-oriented supervision.<n>CIPO improves learning effectiveness while explicitly enhancing the model's ability to correct its own errors.
  arXiv  Detail & Related papers  (2026-05-14T08:22:21Z)
• Power Reinforcement Post-Training of Text-to-Image Models with Super-Linear Advantage Shaping [66.25536973294726]
  Post-training methods for text-to-image (T2I) models are prone to reward hacking.<n>Super-Linear Advantage Shaping (SLAS) reshapes the local policy space.<n>SLAS consistently surpasses the DanceGRPO baseline across multiple backbones and benchmarks.
  arXiv  Detail & Related papers  (2026-05-11T17:59:25Z)
• Same Signal, Opposite Meaning: Direction-Informed Adaptive Learning for LLM Agents [24.71914991820339]
  LLM agents aim to invoke extra computation only when it improves performance.<n>Existing methods typically use confidence-, uncertainty-, or difficulty-based gates, assuming a fixed direction from the gating signal through compute need to the value of computation.<n>We show that this alignment is unstable, with reversals across environments and backbones even when the task is fixed.<n>We propose DIAL (Direction-Informed Adaptive Learning), a sparse gate trained from signal-agnostic counterfactual exploration to learn the utility direction of state features per.
  arXiv  Detail & Related papers  (2026-05-07T20:15:51Z)
• GradAlign: Gradient-Aligned Data Selection for LLM Reinforcement Learning [55.03441672267886]
  We propose GradAlign, a gradient-aligned data selection method for reinforcement learning.<n>We evaluate GradAlign across three data regimes: unreliable reward signals, distribution imbalance, and low-utility training corpus.
  arXiv  Detail & Related papers  (2026-02-25T01:54:50Z)
• Improving LLM-based Recommendation with Self-Hard Negatives from Intermediate Layers [80.55429742713623]
  ILRec is a novel preference fine-tuning framework for LLM-based recommender systems.<n>We introduce a lightweight collaborative filtering model to assign token-level rewards for negative signals.<n>Experiments on three datasets demonstrate ILRec's effectiveness in enhancing the performance of LLM-based recommender systems.
  arXiv  Detail & Related papers  (2026-02-19T14:37:43Z)
• Learning Where It Matters: Geometric Anchoring for Robust Preference Alignment [6.428964221372943]
  We propose Geometric Anchor Preference Optimization (GAPO), which replaces the fixed reference with a dynamic, geometry-aware anchor.<n>GAPO consistently improves robustness while matching or improving performance on standard LLM alignment and reasoning benchmarks.
  arXiv  Detail & Related papers  (2026-02-04T00:40:21Z)
• Local Pairwise Distance Matching for Backpropagation-Free Reinforcement Learning [0.9065034043031668]
  Training neural networks with reinforcement learning (RL) typically relies on backpropagation (BP)<n>BP requires storage of activations from the forward pass for subsequent backward updates.<n>We propose a novel approach that trains each layer of the neural network using local signals during the forward pass in RL settings.
  arXiv  Detail & Related papers  (2025-07-15T14:39:41Z)
• Towards Overcoming False Positives in Visual Relationship Detection [95.15011997876606]
  We investigate the cause of the high false positive rate in Visual Relationship Detection (VRD) This paper presents Spatially-Aware Balanced negative pRoposal sAmpling (SABRA) as a robust VRD framework that alleviates the influence of false positives.
  arXiv  Detail & Related papers  (2020-12-23T06:28:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.