SIVA-RL: Sensitivity-Invariance Visual Alignment for Multimodal Reinforcement Learning

• URL: http://arxiv.org/abs/2607.13931v1
• Date: Wed, 15 Jul 2026 15:13:02 GMT
• Title: SIVA-RL: Sensitivity-Invariance Visual Alignment for Multimodal Reinforcement Learning
• Abstract summary: We propose SIVA-RL to replace operator-conditioned regularization with sample-wise, outcome-conditioned supervision.<n>SIVA-RL constructs localized interventions through token-aligned, distance-constrained within-image Patch.<n>It yields an 8.79 percentage-point gain on vision-dependent reasoning and up to 14.9% relative overall improvement across all four GRPO- and DAPO-based configurations.
• Score: 19.079142995255506
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reinforcement learning with verifiable rewards (RLVR) drives multimodal reasoning, but answer-level correctness does not guarantee that a vision-language model grounds its predictions in visual evidence. Existing visual-intervention methods contrast policy behavior on original and modified images, yet assign supervision by the type of intervention rather than its observed effect. This assumption fails: identical operators produce heterogeneous outcomes across samples. We propose SIVA-RL, a Sensitivity-Invariance Visual Alignment framework that replaces operator-conditioned regularization with sample-wise, outcome-conditioned supervision. SIVA-RL constructs localized interventions through token-aligned, distance-constrained within-image PatchSwap. A frozen audit policy then scores each clean-intervention pair, and the observed reward drop becomes soft routing weights. Large-drop pairs drive sensitivity alignment, low-drop pairs drive clean-anchored invariance alignment, and ambiguous pairs are down-weighted. This design decouples intervention construction from supervision assignment and is compatible with both GRPO and DAPO backbones. Across nine multimodal reasoning benchmarks spanning mathematical, logical, and vision-dependent tasks, SIVA-RL improves 3B and 7B models over matched RL baselines in every setting. It yields an 8.79 percentage-point gain on vision-dependent reasoning and up to 14.9% relative overall improvement across all four GRPO- and DAPO-based configurations.

Related papers

• Consistency as Inductive Bias: Learning Cross-View Invariance for Robust Multimodal Reasoning [40.68014726208686]
  We propose ConsistRoll to inject cross-view consistency into RLVR training.<n>It places original and semantically invariant transformed views in the same generation group.<n>It assigns a joint reward only when paired completions are both correct and consistent.
  arXiv  Detail & Related papers  (2026-06-29T05:45:18Z)
• DyCo-RL: Dynamic Cross-Modal Coordination for Visual Reasoning [49.985643703496514]
  We propose DyCo-RL, which integrates dynamic cross-modal coordination into RLVR optimization.<n>Experiments demonstrate that DyCo-RL, when applied to Qwen2.5-VL-3B/7B, consistently improves four representative RLVR algorithms.
  arXiv  Detail & Related papers  (2026-06-06T07:56:25Z)
• Leveraging Error Diversity in Group Rollouts for Reinforcement Learning [61.10980882949414]
  Reinforcement Learning from Verifiable Rewards (RLVR) typically samples multiple responses per prompt and assigns binary rewards based on individual correctness.<n>We propose Error Diversity Advantage Shaping (EDAS) to modulate the advantage signal for incorrect rollouts based on intra-group error diversity.<n>EDAS operates as a simple post-hoc adjustment that can be seamlessly integrated into any RLVR algorithm.
  arXiv  Detail & Related papers  (2026-05-17T08:52:31Z)
• VSPO: Vector-Steered Policy Optimization for Behavioral Control [30.80095775190934]
  We introduce Vector-Steered Policy optimization (VSPO) for language models.<n>VSPO employs a steering vector associated with the target behavior to control the behavior intensity of the generated rollouts.<n>We show that VSPO consistently improves the control along target behavior while maintaining or improving task accuracy.
  arXiv  Detail & Related papers  (2026-05-15T04:31:06Z)
• What to Ignore, What to React: Visually Robust RL Fine-Tuning of VLA Models [35.49750676677204]
  Reinforcement learning (RL) fine-tuning has shown promise for Vision-Language-Action (VLA) models in robotic manipulation.<n>A key difficulty is that standard task rewards supervise task success, but offer limited guidance on whether a visual change is task-irrelevant or changes the behavior required for manipulation.<n>We propose PAIR-VLA to address this difficulty by adding two auxiliary objectives over paired visual variants during PPO optimization.
  arXiv  Detail & Related papers  (2026-05-13T07:15:37Z)
• Faithful GRPO: Improving Visual Spatial Reasoning in Multimodal Language Models via Constrained Policy Optimization [31.411469692692766]
  Chain-of-Thought traces are frequently inconsistent with the final answer and poorly grounded in the visual evidence.<n>We propose Faithful GRPO, which enforces consistency and grounding as constraints via Lagrangian dual ascent.<n>Our results show that FGRPO substantially improves reasoning quality, reducing the inconsistency rate from 24.5% to 1.7% and improving visual grounding scores by +13%.
  arXiv  Detail & Related papers  (2026-04-09T17:15:47Z)
• Diffusion Controller: Framework, Algorithms and Parameterization [54.82539154511621]
  We present a unified control-theoretic view that casts reverse diffusion sampling as state-only control within (generalized) linearly-solvable Markov Decision Processes.<n>Under this framework, control acts by reweighting the pretrained reverse-time transition kernels, balancing terminal objectives against an $f$-divergence cost.<n>Experiments on Stable Diffusion v1.4 show consistent gains in preference-alignment win rates and improved quality-efficiency trade-offs.
  arXiv  Detail & Related papers  (2026-03-07T01:49:59Z)
• CASHEW: Stabilizing Multimodal Reasoning via Iterative Trajectory Aggregation [6.356820150960838]
  We introduce two complementary approaches inspired by test-time scaling to stabilize vision-language models.<n>CASHEW is an inference-time framework that stabilizes reasoning by iteratively aggregating multiple candidate trajectories into higher-quality reasoning traces.<n>CASHEW-RL is trained using Group Sequence Policy Optimization (GSPO) with a composite reward that encourages correct answers grounded in minimal yet sufficient visual evidence.
  arXiv  Detail & Related papers  (2026-01-12T21:24:45Z)
• ArenaRL: Scaling RL for Open-Ended Agents via Tournament-based Relative Ranking [84.07076200941474]
  ArenaRL is a reinforcement learning paradigm that shifts from pointwise scalar scoring to intra-group relative ranking.<n>We construct an intra-group adversarial arena and devise a tournament-based ranking scheme to obtain stable advantage signals.<n>Experiments show that ArenaRL substantially outperforms standard RL baselines.
  arXiv  Detail & Related papers  (2026-01-10T08:43:07Z)
• Enhancing the Outcome Reward-based RL Training of MLLMs with Self-Consistency Sampling [90.87033586963828]
  Outcome-reward reinforcement learning (RL) is a common and increasingly significant way to refine the step-by-step reasoning of multimodal large language models (MLLMs)<n>We propose Self-Consistency Sampling (SCS) to correct this issue.<n>Based on Qwen2.5-VL-7B-Instruct, SCS improves accuracy by up to 7.7 percentage points on six multimodal benchmarks with negligible extra computation.
  arXiv  Detail & Related papers  (2025-11-13T18:59:57Z)
• Token-Level Inference-Time Alignment for Vision-Language Models [58.41370989069588]
  Vision-Language Models (VLMs) have become essential backbones of modern multimodal intelligence.<n>We present TITA, a lightweight framework that freezes the base VLM and instead trains a reward model to approximate its distribution.<n>During inference, implicit preference signals are extracted as log-probability ratios between the reward model and the target VLM, yielding dense autoregressive feedback.
  arXiv  Detail & Related papers  (2025-10-20T09:58:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.