Perturb the Thought, Not the Pixels: Latent-Space Rollout Diversification for Reinforcement Learning of Vision-Language Models

• URL: http://arxiv.org/abs/2608.21595v1
• Date: Fri, 21 Aug 2026 19:56:50 GMT
• Title: Perturb the Thought, Not the Pixels: Latent-Space Rollout Diversification for Reinforcement Learning of Vision-Language Models
• Abstract summary: Noise-Contrastive GRPO injects scale-calibrated Gaussian noise into the last hidden layer of the prompt-encoding pass.<n>On Qwen2.5-VL-7B trained on Geometry3K, NC-GRPO significantly improves out-of-domain mathematical reasoning.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reinforcement learning with verifiable rewards (RLVR) improves the reasoning ability of vision-language models (VLMs), and diversifying the rollouts within each optimization group amplifies its gains. Existing approaches diversify through decoding temperature or pixel-space image distortion; we ask whether the perturbation belongs in the model's latent space instead. We introduce Noise-Contrastive GRPO (NC-GRPO), which injects scale-calibrated Gaussian noise into the last hidden layer of the prompt-encoding pass for half of each rollout group, branching those rollouts from a displaced departure state. Branches that reach the answer despite the displacement are reinforced over those derailed by it, converting sensitivity at the branch point into policy-gradient signal; the objective, reward, and inference protocol are untouched. On Qwen2.5-VL-7B trained on Geometry3K, NC-GRPO significantly improves out-of-domain mathematical reasoning over vanilla GRPO across five held-out benchmarks (pooled McNemar $p \le 0.001$) while also improving in-domain accuracy and hallucination robustness -- the latter an axis on which image-space noise regresses even while posting a larger OOD average on perception-heavy benchmarks. Mechanism ablations indicate that independent stochastic diversity, not noise budget or direction, is the active ingredient, and a noise-scale study exposes a dial between reasoning specialization and general capability. NC-GRPO is designed to be modality-agnostic and integrates into a standard RLVR pipeline as a ~50-line change to the inference engine.

Related papers

• N-GRPO: Embedding-Level Neighbor Mixing for Enhanced Policy Optimization [55.14402862283128]
  We introduce N-GRPO, a novel exploration strategy integrated into the Group Relative Policy Optimization framework.<n>Rather than relying on token-level sampling or native embedding-level noise, our approach leverages Semantic Neighbor Mixing.<n>N-GRPO achieves consistent improvements over strong baselines on math reasoning benchmarks and also exhibits robust generalization capabilities on out-of-distribution tasks.
  arXiv  Detail & Related papers  (2026-06-09T12:21:27Z)
• Clipping Bottleneck: Stabilizing RLVR via Stochastic Recovery of Near-Boundary Signals [83.0127582612634]
  Near-boundary Rescue (NSR) is a minimal, plug-and-play modification that retains out-of-bound tokens to recover lost signals.<n>NSR substantially improves training stability and delivers consistent gains over strong baselines such as DAPO and GSPO.
  arXiv  Detail & Related papers  (2026-05-21T16:45:31Z)
• Advantage Collapse in Group Relative Policy Optimization: Diagnosis and Mitigation [13.272542054938258]
  We introduce the Advantage Collapse Rate (ACR), the first metric quantifying the proportion of training batches with ineffective gradients.<n>We then propose Adaptive Virtual Sample Policy Optimization (AVSPO), a lightweight extension of GRPO that injects virtual reward samples.<n>AVSPO reduces advantage collapse by 58-63% relative to GRPO and yields consistent accuracy gains of 4-6 percentage points across all model scales.
  arXiv  Detail & Related papers  (2026-05-20T12:57:37Z)
• MoASE++: Mixture of Activation Sparsity Experts with Domain-Adaptive On-policy Distillation for Continual Test Time Adaptation [65.068801413044]
  Continual test-time adaptation adapts a source-pretrained model to non-stationary, unlabeled target streams.<n>We introduce MoASE, a mixture-of-experts that disentangles domain-agnostic structure from domain-specific texture.
  arXiv  Detail & Related papers  (2026-05-18T01:52:12Z)
• PDCR: Perception-Decomposed Confidence Reward for Vision-Language Reasoning [80.94559742826083]
  Reinforcement Learning with Verifiable Rewards (RLVR) traditionally relies on a sparse, outcome-based signal.<n>Recent work shows that providing a fine-grained, model-intrinsic signal effectively improves language reasoning training by providing step-level guidance without costly external models.<n>While effective for unimodal text, we find that naively applying this global reward to vision-language (V-L) reasoning is a suboptimal strategy.<n>We propose Perception-Decomposed Confidence Reward (PDCR), a framework that solves this by aligning the reward structure with the task's heterogeneous nature.
  arXiv  Detail & Related papers  (2026-05-13T12:55:18Z)
• Too Correct to Learn: Reinforcement Learning on Saturated Reasoning Data [55.84428098924793]
  We propose Constrained Uniform Top-K Sampling (CUTS), a parameter-free decoding strategy enforcing structure-preserving exploration.<n>We integrate this into Mixed-CUTS, a training framework synergizing exploitative and exploratory rollouts to amplify intra-group advantage variance.<n> Notably, Mixed-CUTS improves Pass@1 accuracy on the challenging AIME25 benchmark by up to 15.1% over standard GRPO.
  arXiv  Detail & Related papers  (2026-04-20T16:43:28Z)
• GIRL: Generative Imagination Reinforcement Learning via Information-Theoretic Hallucination Control [0.0]
  We introduce GIRL (Generative Imagination Reinforcement Learning), a latent world-model framework that addresses this failure mode with two key components.<n> GIRL reduces latent rollout drift by 38 to 61 percent across tasks relative to DreamerV3, improves return, and requires fewer environment interactions on long-horizon tasks.<n>A distilled-prior variant reduces overhead and improves computational efficiency relative to the full model.
  arXiv  Detail & Related papers  (2026-04-08T17:14:21Z)
• Not All Tokens See Equally: Perception-Grounded Policy Optimization for Large Vision-Language Models [38.47027398567909]
  Perception-Grounded Policy Optimization (PGPO) is a novel fine-grained credit assignment framework that dynamically reshapes advantages at the token level.<n>We show that PGPO actively amplifies learning signals for visually-dependent tokens while suppressing gradient noise from linguistic priors.<n>Both theoretical and empirical analyses confirm that PGPO effectively reduces gradient variance, prevents training collapse, and acts as a potent regularizer for robust, perception-grounded multimodal reasoning.
  arXiv  Detail & Related papers  (2026-04-02T09:53:20Z)
• ConsistentRFT: Reducing Visual Hallucinations in Flow-based Reinforcement Fine-Tuning [85.20505958752928]
  Reinforcement Fine-Tuning (RFT) on flow-based models is crucial for preference alignment.<n>RFT often introduce visual hallucinations like over-optimized details and semantic misalignment.<n>This work preliminarily explores why visual hallucinations arise and how to reduce them.
  arXiv  Detail & Related papers  (2026-02-03T11:49:46Z)
• Learning to Explore with Parameter-Space Noise: A Deep Dive into Parameter-Space Noise for Reinforcement Learning with Verifiable Rewards [16.22162269278471]
  PSN-RLVR perturbs policy parameters before rollout generation to induce temporally consistent, trajectory-level exploration.<n>We propose a computationally efficient real-time adaptive noise scheduler driven by a lightweight surrogate that combines semantic diversity with normalized self-certainty.
  arXiv  Detail & Related papers  (2026-01-30T13:10:30Z)
• Smooth Operator: Smooth Verifiable Reward Activates Spatial Reasoning Ability of Vision-Language Model [18.526821056010384]
  Vision-Language Models (VLMs) face a critical bottleneck in achieving precise numerical prediction for 3D scene understanding.<n>Traditional reinforcement learning approaches, primarily based on relative ranking, often suffer from severe reward sparsity and gradient instability.<n>We introduce the Smooth Numerical Reward Activation (SNRA) operator and the Absolute-Preserving GRPO framework.
  arXiv  Detail & Related papers  (2026-01-12T16:26:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.