Hints, Critics, and Teachers: Prior Injection for Sparse-Reward RL in Vision-Language Math Reasoning

• URL: http://arxiv.org/abs/2608.21811v1
• Date: Sat, 22 Aug 2026 07:17:14 GMT
• Title: Hints, Critics, and Teachers: Prior Injection for Sparse-Reward RL in Vision-Language Math Reasoning
• Abstract summary: On a pool of 20,830 visual-math problems where Qwen2-VL-2B answers 3.6% of rollouts correctly, 85-97% of GRPO rollouts are entirely wrong and zero gradient.<n>We train eleven methods under identical conditions in this regime, each injecting a different prior.<n>Among the methods, hint-guided exploration drives hint gains, and replacing the critic's MSE loss with HL-Gauss cross-entropy is worth +14.4 points in-domain.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reinforcement learning for vision-language math reasoning starves under sparse reward: on a pool of 20,830 visual-math problems where Qwen2-VL-2B answers 3.6% of rollouts correctly, 85-97% of GRPO rollout groups are entirely wrong and contribute zero gradient. We train eleven methods under identical conditions in this regime, each injecting a different prior: text (reference-solution hints), distribution (on-policy distillation from a 7B teacher), and value (a value-pretrained critic with an MSE or HL-Gauss categorical loss). A prior helps exactly when it is delivered: the six arms whose prior effectively reaches the policy separate with no overlap from the remaining five -- the no-prior baseline and four arms whose prior is teacher-capped, gated away, or lost to a mis-parameterized critic -- both on the pooled in-domain metric and on cross-domain transfer (DynaMath). The central finding, however, concerns evaluation: one slice of the in-domain pool -- long used as this project's general-distribution check -- anti-correlates with genuine cross-domain transfer (Spearman rho = -0.74, n = 11 arms, permutation p = 0.011), while the hardest in-domain slice predicts it closely (rho = +0.89, p < 0.001). We attribute the inversion to a near-chance multiple-choice subset that rewards models for not having changed; read through it, the best cross-domain method looked mediocre and the worst looked like the champion. Among the methods, hint-guided exploration -- not UFT's auxiliary loss -- drives hint gains, and replacing the critic's MSE loss with HL-Gauss cross-entropy is worth +14.4 points in-domain. All accuracies are blind-judged, with paired exact tests.

Related papers

• EGGROLL, Unrolled: Understanding and Improving Low-Rank Evolution Strategies at Scale [9.562123195246393]
  EGGROLL makes evolution strategies practical for LLMs by replacing dense Gaussian weight perturbations with low-rank Gaussian products.<n>We analyze the error of its finite-population estimator.<n>LOO-ROLL, a leave-one-out estimator, preserves the finite-rank population field while replacing EGGROLL's two antithetic evaluations per direction by one.
  arXiv  Detail & Related papers  (2026-09-10T01:58:02Z)
• Coverage, Not Targeting: A Structural Regime in Multi-Turn Agent Credit Assignment [5.864346468963738]
  Multi-turn agentic RL increasingly treats credit assignment as a targeting problem.<n>We show that terminal-state verifiers sit deep in a low-V_d regime where targeting is the wrong axis.
  arXiv  Detail & Related papers  (2026-09-02T10:37:12Z)
• Nothing Changed but the Model: CellFill -- Bounded In-Cell Learning for Bit-Identical, Revocable Updates to Quantized LLMs [4.455667003143275]
  We show that every way of teaching a language model, full fine-tuning, adapter merging, model editing, replaces the released checkpoint.<n>With the integer codes and scales of a 4-bit release frozen, new knowledge is written only into the per-weight residual that lives strictly inside each quantization decision cell.<n>We give six propositions and three training paths, including CellFill.
  arXiv  Detail & Related papers  (2026-08-21T08:49:07Z)
• Measuring Reward Hacking and Reasoning-Answer Decoupling Under Position-Confounded Optimization [1.1316247605466565]
  We train language models with GRPO on math problems where the correct answer is always option A, then evaluate on an unseen test set with unbiased answer positions.<n>Across Qwen2.5, Llama 3.x and Gemma-3 models, biased training often drives option-A rates above 0.90 in smaller models.<n>We track reasoning-answer decoupling: capable models generate reasoning that reaches the correct numeric answer while still selecting A.
  arXiv  Detail & Related papers  (2026-08-15T23:13:57Z)
• WDL-OPD: Weak-Driven On-Policy Distillation via Mixture-Constrained Co-Training [48.290951447286744]
  WDL-OPD is a mixture-constrained co-training method with two trainable policies.<n>An anchor policy generates every rollout, an auxiliary policy evaluates the same visited states, and a geometric mixture of their token distributions is matched to a frozen teacher by reverse KL.<n>We show that freezing the auxiliary recovers an anchor-plus-contrast proxy target closely related to OPD$2$ and W2S-OPD, whereas joint training creates branch-level degrees of freedom that a static delta cannot express.
  arXiv  Detail & Related papers  (2026-08-10T11:22:53Z)
• Right Makes Might: Aligning Verified Hidden States Empowers RL Reasoning [55.264863369127774]
  Current methods reduce each correct rollout to a single reward bit, ignoring the geometric structure shared among their hidden states.<n>We propose Hidden-Align, an auxiliary loss function that aligns the last-layer hidden states of correct rollouts at the anchor token during RL training, with zero overhead in both training and inference.<n>On eight mathematical reasoning benchmarks, Hidden-Align improves average pass@1 over the DAPO baseline by 3.8, 6.2, and 5.4 percentage points on Qwen3-1.7B, 4B, and 14B respectively, with consistent pass@k gains across all three scales.
  arXiv  Detail & Related papers  (2026-06-02T06:51:15Z)
• Gradient Starvation in Binary-Reward GRPO: Why Group-Mean Centering Fails and Why the Simplest Fix Works [19.92945464084119]
  Group Relative Policy Optimization (GRPO) is a standard algorithm for reinforcement learning from verifiable rewards.<n>We prove that the true degeneracy rate always exceeds the i.i.d. Bernoulli prediction by Jensen's inequality.<n>We show that the fixed-reference Sign advantage, $A=2r-1$, performs pass@$G$ failure descent by increasing the probability that at least one sample in the group succeeds.
  arXiv  Detail & Related papers  (2026-05-08T12:58:06Z)
• Transform-Augmented GRPO Improves Pass@k [50.3707071191733]
  Group Relative Policy Optimization (GRPO) was designed to improve reasoning, but it worsens this situation through two failure modes.<n>We propose TA-GRPO (Transform-Augmented GRPO), which generates semantically equivalent transformed variants of each question.<n>This pooled computation ensures mixed rewards even when the original question is too easy or too hard, while training on diverse phrasings promotes multiple solution strategies.
  arXiv  Detail & Related papers  (2026-01-30T02:43:29Z)
• CARE What Fails: Contrastive Anchored-REflection for Verifiable Multimodal [84.71254539482369]
  Group-relative reinforcement learning with verifiable rewards (RLVR) often wastes the most informative data it already has the failures.<n>We present CARE, a failure-centric post-training framework for multimodal reasoning that turns errors into supervision.<n> CARE improves accuracy and training smoothness while explicitly increasing the share of learning signal that comes from failures.
  arXiv  Detail & Related papers  (2025-12-22T16:34:21Z)
• Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifiers [90.50039419576807]
  Reinforcement Learning with Verifiable Rewards (RLVR) trains policies against automated verifiers to avoid costly human labeling.<n>To reduce vulnerability to verifier hacking, many RLVR systems collapse rewards to binary $0,1$ during training.<n>This choice carries a cost: it introduces textitfalse negatives (rejecting correct answers, FNs) and textitfalse positives (accepting incorrect ones, FPs)
  arXiv  Detail & Related papers  (2025-10-01T13:56:44Z)
• KL Guided Domain Adaptation [88.19298405363452]
  Domain adaptation is an important problem and often needed for real-world applications. A common approach in the domain adaptation literature is to learn a representation of the input that has the same distributions over the source and the target domain. We show that with a probabilistic representation network, the KL term can be estimated efficiently via minibatch samples.
  arXiv  Detail & Related papers  (2021-06-14T22:24:23Z)
• A Balanced and Uncertainty-aware Approach for Partial Domain Adaptation [142.31610972922067]
  This work addresses the unsupervised domain adaptation problem, especially in the case of class labels in the target domain being only a subset of those in the source domain. We build on domain adversarial learning and propose a novel domain adaptation method BA$3$US with two new techniques termed Balanced Adversarial Alignment (BAA) and Adaptive Uncertainty Suppression (AUS) Experimental results on multiple benchmarks demonstrate our BA$3$US surpasses state-of-the-arts for partial domain adaptation tasks.
  arXiv  Detail & Related papers  (2020-03-05T11:37:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.