Why Sample What You Can Enumerate? Exact Policy Optimization for Genomic Tool Selection

• URL: http://arxiv.org/abs/2609.10221v2
• Date: Thu, 10 Sep 2026 01:52:39 GMT
• Title: Why Sample What You Can Enumerate? Exact Policy Optimization for Genomic Tool Selection
• Abstract summary: Reinforcement learning over a frozen reasoner has become a common recipe for teaching a policy which external tools invoke.<n>We show that this recipe becomes structurally mismatched in specialist scientific settings where the complete tool-subset space is enumerable.<n>We introduce FGPO, which scores every tool subset and optimize the exact action expectation, so each update sees the complete action space.
• Score: 23.92446384027772
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reinforcement learning over a frozen reasoner has become a common recipe for teaching a policy which external tools to invoke. We show that this recipe becomes structurally mismatched in specialist scientific settings where the complete tool-subset space is enumerable. There, a small set of recurring computational capabilities covers the domain, so the space of tool subsets is combinatorial yet small enough to enumerate, and GRPO still estimates an action expectation from a handful of sampled rollouts. Worse, the approximation degrades as training succeeds: as the policy concentrates on preferred subsets it resamples them, sampled rewards collide, and the group-normalized advantage vanishes. On genomic reasoning the fraction of questions yielding no reward signal rises from 0.2% under a uniform reference policy to 20.8% after GRPO training. As a remedy, we introduce FGPO (Full-Group Policy Optimization), which (1) scores every tool subset and optimizes the exact action expectation, so each update sees the complete action space, and (2) precomputes the reward of each question--subset pair into an exhaustive table, removing frozen-reasoner calls from the training loop entirely. Across five frozen reasoners and three genomic benchmarks, FGPO outperforms GRPO in all 15 settings by 6.75 points on average and up to 14.20, while a standard on-demand GRPO schedule would require 2.4 times as many frozen-reasoner reward evaluations and, on GenomeQA, FGPO cuts invoked tools per question from 2.36 to 1.40.

Related papers

• Reusing Rollouts under Policy Lag: Prefix-Normalized Policy Optimization for LLM Reinforcement Learning [21.43528083212271]
  Autoregressive rollout generation is a major computational cost in reinforcement learning.<n>PNPO replaces the cumulative ratio with the mean of likelihood ratios along each causal prefix.<n>PNPO does not consistently outperform GSPO with one epoch.<n>Four-epoch PNPO reaches a final macro Avg@32 of 49.66 after 150 rollout batches, comparable to the 49.56 reached after 600 batches with one epoch.
  arXiv  Detail & Related papers  (2026-08-02T18:02:03Z)
• CIGPO: Contextual Information-Gain Policy Optimization for Multi-Turn Evidence-Reading LLM Agents [1.4298580363875282]
  Training multi-turn evidence-reading agents with outcome-only reinforcement learning is unstable because intermediate turns receive little direct credit.<n>In HotpotQA experiments with Qwen2.5-3B-Instruct, GRPO initially improves (standard F1 0.430) but subsequently collapses to 100% format-violating outputs.<n>We propose a variance-textual strategy: by assigning per-turn rewards to intermediate evidence-reading turns, we prevent the group reward distribution from collapsing to a single value.
  arXiv  Detail & Related papers  (2026-06-26T11:50:29Z)
• Agent Explorative Policy Optimization for Multimodal Agentic Reasoning [97.64835302176056]
  Agentic reasoning interleaves two behaviors with a structural asymmetry: thinking and tool use.<n>Under standard RL recipes like GRPO, the gap manifests as two diagnostic symptoms during training.<n>We propose AXPO (Agent eXplorative Policy Optimization) for each all-wrong tool-using subgroup.
  arXiv  Detail & Related papers  (2026-05-27T17:36:39Z)
• Advantage Collapse in Group Relative Policy Optimization: Diagnosis and Mitigation [13.272542054938258]
  We introduce the Advantage Collapse Rate (ACR), the first metric quantifying the proportion of training batches with ineffective gradients.<n>We then propose Adaptive Virtual Sample Policy Optimization (AVSPO), a lightweight extension of GRPO that injects virtual reward samples.<n>AVSPO reduces advantage collapse by 58-63% relative to GRPO and yields consistent accuracy gains of 4-6 percentage points across all model scales.
  arXiv  Detail & Related papers  (2026-05-20T12:57:37Z)
• Beyond Mode Collapse: Distribution Matching for Diverse Reasoning [69.88237286885065]
  On-policy reinforcement learning methods like GRPO suffer from mode collapse.<n>We show this stems from reverse KL minimization's mode-seeking behavior.<n>We propose DMPO, which prevents mode collapse through principled approximation of forward KL minimization.
  arXiv  Detail & Related papers  (2026-05-19T07:13:00Z)
• Selective Rollout: Mid-Trajectory Termination for Multi-Sample Agent RL [6.435733307123974]
  Group-relative RL training (GRPO) samples a small group of parallel rollouts for every training prompt.<n>Uses their within-group reward spread to compute per-trajectory advantages.<n>When every rollout of a prompt ends with the same reward, the group has zero reward variance and contributes no gradient.
  arXiv  Detail & Related papers  (2026-05-07T07:41:09Z)
• Unifying Group-Relative and Self-Distillation Policy Optimization via Sample Routing [79.88256756334327]
  Self-Distillation Policy Optimization (SDPO) addresses this by providing denser, more targeted logit-level supervision.<n>We propose Sample-Routed Policy Optimization (SRPO)<n>SRPO routes correct samples to GRPO's reward-aligned reinforcement and failed samples to SDPO's targeted logit-level correction.
  arXiv  Detail & Related papers  (2026-04-02T17:29:18Z)
• iGRPO: Self-Feedback-Driven LLM Reasoning [88.83313431248473]
  Large Language Models (LLMs) have shown promise in solving complex mathematical problems, yet they still fall short of producing accurate and consistent solutions.<n>We introduce Iterative Group Relative Policy Optimization (iGRPO), a two-stage extension of GRPO that adds dynamic self-conditioning through model-generated drafts.<n>Under matched rollout budgets, iGRPO consistently outperforms GRPO across base models.
  arXiv  Detail & Related papers  (2026-02-09T18:45:11Z)
• Self-Hinting Language Models Enhance Reinforcement Learning [37.311361929798714]
  We propose self-hint aligned GRPO with privileged supervision (SAGE)<n>SAGE injects privileged hints during training to reshape the rollout distribution under the same terminal verifier reward.<n> Experiments over 6 benchmarks with 3 LLMs show that SAGE consistently outperforms GRPO.
  arXiv  Detail & Related papers  (2026-02-03T05:56:20Z)
• GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimization [133.27496265096445]
  We show how to apply Group Relative Policy Optimization under multi-reward setting without examining its suitability.<n>We then introduce Group reward-Decoupled Normalization Policy Optimization (GDPO), a new policy optimization method to resolve these issues.<n>GDPO consistently outperforms GRPO, demonstrating its effectiveness and generalizability for multi-reward reinforcement learning optimization.
  arXiv  Detail & Related papers  (2026-01-08T18:59:24Z)
• CPPO: Accelerating the Training of Group Relative Policy Optimization-Based Reasoning Models [77.16976971950785]
  This paper introduces Completion Pruning Policy Optimization (CPPO) to accelerate the training of reasoning models.<n>CPPO prunes completions with low absolute advantages, significantly reducing the number needed for gradient calculation and updates.<n> Experiments show that CPPO achieves up to $7.98times$ speedup on GSM8K and $3.48times$ on Math while preserving or even enhancing the accuracy compared to the original GRPO.
  arXiv  Detail & Related papers  (2025-03-28T11:30:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.