Rethinking Multiple-Choice Questions for RLVR: Unlocking Potential via Distractor Design

• URL: http://arxiv.org/abs/2603.12826v1
• Date: Fri, 13 Mar 2026 09:28:36 GMT
• Title: Rethinking Multiple-Choice Questions for RLVR: Unlocking Potential via Distractor Design
• Abstract summary: Reinforcement Learning with Verifiable Rewards (RLVR) significantly enhances the reasoning capabilities of Large Language Models.<n>Current approaches often mitigate this by converting Multiple-Choice Questions (MCQs) to open-ended formats.<n>We propose Iterative Distractor Curation (IDC), a framework that actively constructs high-quality distractors to block elimination shortcuts.
• Score: 32.193718910105325
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) significantly enhances the reasoning capabilities of Large Language Models. When applied to RLVR, Multiple-Choice Questions (MCQs) offer a scalable source of verifiable data but risk inducing reward hacking, where models shortcut reasoning via random guessing or simple elimination. Current approaches often mitigate this by converting MCQs to open-ended formats, thereby discarding the contrastive signal provided by expert-designed distractors. In this work, we systematically investigate the impact of option design on RLVR. Our analysis highlights two primary insights: (1) Mismatches in option counts between training and testing degrade performance. (2) Strong distractors effectively mitigate random guessing, enabling effective RLVR training even with 2-way questions. Motivated by these findings, we propose Iterative Distractor Curation (IDC), a framework that actively constructs high-quality distractors to block elimination shortcuts and promote deep reasoning. Experiments on various benchmarks demonstrate that our method effectively enhances distractor quality and yields significant gains in RLVR training compared to the original data.

Related papers

• SVR-R1: Bootstrapping Multi-modal Reasoning with Self-verification in Reinforcement Learning [47.4480595602701]
  We introduce Self-Verified Reasoner (SVR-R1), a multi-turn RL framework that turns a model's own verification into a learning signal for multimodal reasoning.<n>For each query, the model proposes an answer using the same weights, and issues a binary self-verdict (Yes/No)<n>A 'No' triggers a second-chance rethink; a 'Yes,' or a turn cap, finalizes the output for computing the outcome-based reward.
  arXiv  Detail & Related papers  (2026-07-13T00:21:30Z)
• Outcome Rewards Do Not Guarantee Verifiable or Causally Important Reasoning [42.998601712623525]
  We show that RLVR does not always lead models to rely on reasoning in the way that is commonly thought.<n>This issue can be remedied with simple modifications to the post-training procedure.
  arXiv  Detail & Related papers  (2026-04-23T21:01:34Z)
• LongRLVR: Long-Context Reinforcement Learning Requires Verifiable Context Rewards [51.45138356629732]
  We introduce LongRLVR to augment the sparse answer reward with a dense and verifiable context reward.<n>This auxiliary signal directly incentivizes the model for selecting the correct grounding information.<n>LongRLVR consistently and significantly outperforms the standard RLVR across all models and benchmarks.
  arXiv  Detail & Related papers  (2026-03-02T18:07:53Z)
• Detecting RLVR Training Data via Structural Convergence of Reasoning [31.260852555788205]
  Reinforcement learning with verifiable rewards (RLVR) is central to training modern reasoning models.<n>We show that RLVR induces a distinctive behavioral signature.<n>We introduce Min-$k$NN Distance, a simple black-box detector that quantifies this collapse.
  arXiv  Detail & Related papers  (2026-02-12T10:17:32Z)
• Adaptive Ability Decomposing for Unlocking Large Reasoning Model Effective Reinforcement Learning [82.91265691530351]
  A$2$D is an Adaptive Ability Decomposing method for enhancing the effectiveness ofReinforcement Learning with verifiable rewards.<n>We first train a decomposer via RLVR without distillation, enabling it to decompose complex questions into a set of simpler sub-questions.<n>Next, we use this decomposer to annotate sub-questions for each question in the training dataset, and then train the reasoner under RLVR with sub-question guidance.
  arXiv  Detail & Related papers  (2026-01-31T14:48:23Z)
• From Verifiable Dot to Reward Chain: Harnessing Verifiable Reference-based Rewards for Reinforcement Learning of Open-ended Generation [52.62655622099456]
  We propose reinforcement learning with verifiable reference-based rewards (RLVRR)<n>Instead of checking the final answer, RLVRR extracts an ordered linguistic signal from high-quality references (i.e., reward chain)<n>In this way, RLVRR decomposes rewards into two dimensions: content, which preserves deterministic core concepts, and style, which evaluates adherence to stylistic properties.
  arXiv  Detail & Related papers  (2026-01-26T14:39:58Z)
• Spurious Rewards Paradox: Mechanistically Understanding How RLVR Activates Memorization Shortcuts in LLMs [16.74831908818562]
  Recent evidence shows models like Qwen 2.5 achieve significant gains even with spurious or incorrect rewards.<n>We identify a "Perplexity Paradox": spurious RLVR triggers a divergence where answer-token perplexity drops while prompt-side coherence degrades.<n>We uncover a hidden Anchor-Adapter circuit that facilitates this shortcut.
  arXiv  Detail & Related papers  (2026-01-16T07:55:38Z)
• Limits of Generalization in RLVR: Two Case Studies in Mathematical Reasoning [3.437656066916039]
  Reinforcement with Verifiable Rewards (RLVR) has emerged as a promising approach for enhancing such capabilities.<n>We investigate RLVR on two problems with fully verifiable solutions.<n>We find that RLVR improves evaluation metrics but often by reinforcing superficial Learning metrics rather than acquiring new reasoning strategies.
  arXiv  Detail & Related papers  (2025-10-30T23:16:02Z)
• The Reasoning Boundary Paradox: How Reinforcement Learning Constrains Language Models [31.773914661815393]
  Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a key method for improving Large Language Models' reasoning capabilities.<n>Recent evidence suggests it may paradoxically shrink the reasoning boundary rather than expand it.<n>This paper investigates the shrinkage issue of RLVR by analyzing its learning dynamics.
  arXiv  Detail & Related papers  (2025-10-02T17:17:27Z)
• CDE: Curiosity-Driven Exploration for Efficient Reinforcement Learning in Large Language Models [85.315711639214]
  We introduce Curiosity-Driven Exploration (CDE), a framework that leverages the model's own intrinsic sense of curiosity to guide exploration.<n>For the actor, we use perplexity over its generated response, and for the critic, we use the variance of value estimates from a multi-head architecture.<n>Our theoretical analysis shows that the actor-wise bonus inherently penalizes overconfident errors and promotes diversity among correct responses.
  arXiv  Detail & Related papers  (2025-09-11T17:59:17Z)
• Reinforcing Video Reasoning with Focused Thinking [65.85683941058916]
  We propose TW-GRPO, a novel framework that enhances visual reasoning with focused thinking and dense reward granularity.<n>Specifically, we employ a token weighting mechanism that prioritizes tokens with high informational density.<n>We also reformulate RL training by shifting from single-choice to multi-choice QA tasks.
  arXiv  Detail & Related papers  (2025-05-30T15:42:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.