Daily Paper Briefing

Abstract Overview

This paper introduces Test-Time Policy Optimization (TTPO), a label-free test-time training method for mathematical reasoning models. The method is motivated by an observed asymmetry in majority-vote pseudo-labeling on hard competition problems: pseudo-labels are often wrong, but rollouts that disagree with the pseudo-label are still usually incorrect. TTPO therefore applies on-policy self-distillation to rollouts that agree with the pseudo-label and Group Relative Policy Optimization penalties to rollouts that disagree, with token-level weighting and masking in the two branches. Experiments on Qwen3-1.7B, 4B, and 8B across five competition-level math benchmarks evaluate both training on labeled benchmark data without using labels and pure test-time training on unlabeled test sets.

Novelty

The distinctive contribution is an asymmetric objective for label-free test-time training that treats pseudo-label agreement and disagreement differently instead of using a single distillation or RL signal for all rollouts. The paper also introduces token-level selection in both branches, using weighting for positive distillation and masking for negative RL penalties to reduce the effect of pseudo-label noise and collateral token-level errors.

Results

Across five competition-level benchmarks, TTPO matches or exceeds label-supervised OPSD despite using only majority-vote pseudo-labels. In the pure test-time training setting, it improves Qwen3-1.7B average accuracy from 38.0% to 45.2% and outperforms both TTRL and OPSD-TTT across the reported model scales. The paper also reports gains in non-thinking evaluation and cross-benchmark transfer, suggesting that the method improves reasoning behavior beyond a single target task.

Key Points

1. TTPO is built on the empirical observation that majority-vote pseudo-labels are frequently wrong on hard problems, yet disagreeing rollouts are still usually incorrect, making negative penalties more reliable than naive full distillation.
2. The method combines OPSD on pseudo-label-consistent rollouts with GRPO penalties on inconsistent rollouts, and further refines both using token-level weighting and masking.
3. Empirically, TTPO achieves stronger label-free test-time training performance than prior baselines and remains competitive with or better than label-supervised OPSD on the reported benchmarks.

References

• arXiv: https://arxiv.org/abs/2608.27448v1
• Fugu-MT: https://fugumt.com/fugumt/paper_check/2608.27448v1
• Hugging Face Papers: https://huggingface.co/papers/2608.27448