Hi-TTRL: Regulating Consensus with Hints for Test-Time Reinforcement Learning

• URL: http://arxiv.org/abs/2608.03545v2
• Date: Wed, 05 Aug 2026 03:37:38 GMT
• Title: Hi-TTRL: Regulating Consensus with Hints for Test-Time Reinforcement Learning
• Abstract summary: We introduce Hi-TTRL, a test-time reinforcement learning framework that utilizes hints during sampling to regulate rollout consensus strength.<n>Experiments on multiple datasets and backbones show that Hi-TTRL consistently improves over standard TTRL.
• Score: 22.648368360495315
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Test-time reinforcement learning (TTRL) improves the reasoning capabilities of large language models without labeled data by updating the policy with pseudo-labels constructed through majority voting. While effective, the reward signal assigned from majority voting is highly sensitive to consensus strength, defined as the frequency of the most common answer within a rollout group. In TTRL, consensus strength plays a dual role: it reflects both the reliability of the pseudo-label and the distribution of advantages. Low consensus can amplify updates from unreliable pseudo-labels through disproportionately large advantages, whereas high consensus reduces reward contrast and ultimately yields vanishing gradients. In this paper, we introduce Hi-TTRL, a test-time reinforcement learning framework that utilizes hints during sampling to regulate rollout consensus strength. Hi-TTRL first estimates consensus strength from a partial rollout group. When the consensus strength falls outside a target interval, it invokes a Markov chain Monte Carlo (MCMC) hint sampler. The sampler targets the power-transformed prefix distribution and uses finite-step approximate sampling to generate rollout prefixes as hints. By tuning the power exponent, Hi-TTRL generates hints with a sharpened or flattened power target, steering rollout consensus strength toward the target interval. Experiments on multiple datasets and backbones show that Hi-TTRL consistently improves over standard TTRL, with ablations and consensus-steering analyses validating the effectiveness of adaptive hint-guided consensus regulation.

Related papers

• ARISE-RL: Agentic Rubric-Grounded Iterative Self-Evolution with Reinforcement Learning [74.19730275669227]
  ARISE-RL is a novel full-cycle self-evolution framework for training open-ended agents.<n>We present Reward-Gated Self-Evolution Distillation (RG-SED), which distills a memory-augmented variant of the same policy back into itself.<n>We also present ECR-Bench, an expert-calibrated rubric benchmark suite covering single-tool deep research and multi-tool travel planning.
  arXiv  Detail & Related papers  (2026-09-01T10:54:13Z)
• TRACE: A Unified Rollout Budget Allocation Framework for Efficient Agentic Reinforcement Learning [52.67756371231985]
  Reinforcement learning with verifiable rewards (RLVR) is a promising approach for enhancing reasoning and agentic behavior in large language models.<n>We introduce Tree Rollout Allocation for Contrastive Exploration (TRACE), a unified rollout allocation framework.<n>Technically, TRACE allocates rollout budget to both prompt roots and intermediate prefixes that are most likely to yield mixed terminal rewards.
  arXiv  Detail & Related papers  (2026-06-09T17:16:03Z)
• TARPO: Token-Wise Latent-Explicit Reasoning via Action-Routing Policy Optimization [18.977861031175756]
  TARPO is a pure RL framework that switches between discrete token generation and continuous latent reasoning at each step.<n> TARPO consistently outperforms existing explicit and latent reasoning RL baselines across diverse benchmarks.
  arXiv  Detail & Related papers  (2026-06-04T08:30:53Z)
• Understanding and Mitigating Spurious Signal Amplification in Test-Time Reinforcement Learning for Math Reasoning [37.981949917083746]
  Test-time reinforcement learning (TTRL) always adapts models at inference time via pseudo-labeling.<n>We propose a unified framework, Debiased and Denoised test-time Reinforcement Learning, to mitigate spurious signals.
  arXiv  Detail & Related papers  (2026-04-23T06:32:08Z)
• Balanced Aggregation: Understanding and Fixing Aggregation Bias in GRPO [70.38763678943648]
  Reinforcement learning with verifiable rewards (RLVR) has become a central paradigm for improving reasoning and code generation in large language models.<n>Standard GRPO uses sequence aggregation, while recent work has advocated token aggregation as a better alternative.<n>We show that these two rules induce different optimization biases; token aggregation introduces sign-length coupling, while sequence aggregation implicitly downweights longer responses.
  arXiv  Detail & Related papers  (2026-04-14T09:48:46Z)
• What If Consensus Lies? Selective-Complementary Reinforcement Learning at Test Time [57.533031432715084]
  Test-Time Reinforcement Learning (TTRL) enables Large Language Models (LLMs) to enhance reasoning capabilities on unlabeled test streams.<n>Existing TTRL methods rely exclusively on positive pseudo-labeling strategies.<n>We propose SCRL, a robust test-time reinforcement learning framework that effectively mitigates label noise amplification.
  arXiv  Detail & Related papers  (2026-03-20T11:47:12Z)
• Beyond Majority Voting: Towards Fine-grained and More Reliable Reward Signal for Test-Time Reinforcement Learning [12.354777054071379]
  Test-time reinforcement learning mitigates reliance on annotated data by using majority voting results as pseudo-labels.<n>This voting strategy often induces confirmation bias and suffers from sparse rewards, limiting the overall performance.<n>We propose subgroup-specific step-wise confidence-weighted pseudo-label estimation (SCOPE) to address these issues.
  arXiv  Detail & Related papers  (2025-12-17T07:21:54Z)
• ZIP-RC: Optimizing Test-Time Compute via Zero-Overhead Joint Reward-Cost Prediction [57.799425838564]
  We present ZIP-RC, an adaptive inference method that equips models with zero-overhead inference-time predictions of reward and cost.<n> ZIP-RC improves accuracy by up to 12% over majority voting at equal or lower average cost.
  arXiv  Detail & Related papers  (2025-12-01T09:44:31Z)
• CG-TTRL: Context-Guided Test-Time Reinforcement Learning for On-Device Large Language Models [37.06397567773862]
  Test-time Reinforcement Learning (TTRL) has shown promise in adapting foundation models for complex tasks at test-time.<n>We propose context-guided TTRL, integrating context dynamically into both sampling phases and propose a method for efficient context selection for on-device applications.
  arXiv  Detail & Related papers  (2025-11-09T15:51:52Z)
• Reinforcing Video Reasoning with Focused Thinking [65.85683941058916]
  We propose TW-GRPO, a novel framework that enhances visual reasoning with focused thinking and dense reward granularity.<n>Specifically, we employ a token weighting mechanism that prioritizes tokens with high informational density.<n>We also reformulate RL training by shifting from single-choice to multi-choice QA tasks.
  arXiv  Detail & Related papers  (2025-05-30T15:42:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.