GAPD: Gold-Action Policy Distillation for Agentic Reinforcement Learning in Knowledge Base Question Answering

• URL: http://arxiv.org/abs/2605.29584v2
• Date: Wed, 03 Jun 2026 07:07:10 GMT
• Title: GAPD: Gold-Action Policy Distillation for Agentic Reinforcement Learning in Knowledge Base Question Answering
• Abstract summary: We propose a training-time Gold-Action Policy Distillation framework that adds dense token-level guidance to outcome-based RL.<n>GAPD consistently surpasses the current state of the art on WebQSP, GrailQA, and GraphQ.
• Score: 64.23115520219609
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reinforcement learning (RL) is a natural fit for agentic knowledge base question answering (KBQA), where a model must issue executable actions, observe knowledge-base feedback, and eventually return an answer. However, current RL-based KBQA systems mainly optimize sparse rewards from the final answer, leaving intermediate action errors weakly supervised. This is especially limiting for logical-form annotated KBQA benchmarks: gold logical forms can be converted into executable action sequences, but existing pipelines use them mainly for warm-start data construction rather than for on-policy RL updates. We propose GAPD, a training-time Gold-Action Policy Distillation framework that adds dense token-level guidance to outcome-based RL. To align gold actions with on-policy student rollouts, GAPD uses MID-ANCHOR MATCHING: it treats the intermediate entities reached during student exploration and gold execution as state anchors, and matches student states to gold states through these explored entity sets. The current policy conditioned on this aligned gold action serves as a stop-gradient teacher, whose token distribution is distilled back to the ordinary student policy over generated action-token spans. GAPD consistently surpasses the current state of the art on WebQSP, GrailQA, and GraphQ.

Related papers

• GTA-RAG: Graph-Trajectory-Augmented Reinforcement Learning for Multi-Turn Retrieval-Augmented Reasoning [27.83191719981676]
  We present textscGTA-RAG, a graph-trajectory-augmented RL framework for retrieval-augmented reasoning.<n>Method consistently outperforms RL-based RAG baselines with both Qwen2.5-3B and Qwen2.5-7B backbones.
  arXiv  Detail & Related papers  (2026-08-23T16:05:20Z)
• AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learning [58.76655851910778]
  AgentOPSD is a critic-free, recursive method for turn-level credit assignment in agentic reinforcement learning.<n>We evaluate AgentOPSD on ALFWorld, WebShop, and Search-QA using Qwen2.5 models at two scales (3B and 7B)
  arXiv  Detail & Related papers  (2026-08-06T13:00:59Z)
• Graph Is the Verifier: Agentic Reinforcement Learning for Interprocedural Vulnerability Detection [13.409515289348532]
  VulAgentRL is a framework for interprocedural vulnerability detection built on a Code Property Graph (CPG)<n>We show that VulAgentRL outperforms state-of-the-art baselines, including frontier models, on the strict pair-wise-correct metric while issuing fewer tool calls.
  arXiv  Detail & Related papers  (2026-07-29T09:16:54Z)
• Test-Time Gradient Guidance of Flow Policies in Reinforcement Learning [50.738952715864116]
  Expressive continuous control policies form the backbone of advances in scaling imitation learning for simulated and real robot control.<n>We propose QGF (Q-Guided Flow), an RL algorithm that performs policy optimization entirely at test time.<n> Empirically, QGF outperforms prior test-time RL methods on single-task and goal-conditioned offline RL benchmarks.
  arXiv  Detail & Related papers  (2026-06-09T16:45:57Z)
• Text-to-SPARQL Generation with Reinforcement Learning: A GRPO-based Approach on DBLP [5.221431176702212]
  Group-Relative Policy Optimization is applied to the Qwen3-1.7B model on DBLP-QuAD.<n>This study examines whether reinforcement learning with outcome-based rewards can train a small instruction-tuned language model to perform zero-shot Text-to-SPARQL generation.
  arXiv  Detail & Related papers  (2026-05-19T16:20:57Z)
• Rubric-Grounded RL: Structured Judge Rewards for Generalizable Reasoning [1.517713730645682]
  We formalize emphrubric-grounded reinforcement learning (RL): a framework in which the policy is optimized against a structured, multi-criterion reward.<n>We instantiate the framework by deriving rubrics from an Office of Scientific and Technical Information (OSTI)-derived corpus of roughly 100,000 scientific and technical documents.
  arXiv  Detail & Related papers  (2026-05-08T17:48:58Z)
• Self-Induced Outcome Potential: Turn-Level Credit Assignment for Agents without Verifiers [26.97849381770806]
  Self-Induced Outcome Potential treats semantic clusters of final answers as latent future outcome states for potential-based turn-level credit assignment.<n>We formalize the framework, characterize its supervised gold-answer limit, and show that SIOP improves average performance over verifier-free outcome-level baselines.
  arXiv  Detail & Related papers  (2026-05-06T14:38:48Z)
• GoldenStart: Q-Guided Priors and Entropy Control for Distilling Flow Policies [22.819853466384686]
  Flow-matching policies hold great promise for reinforcement learning (RL)<n>Their practical application is often hindered by prohibitive latency inference and ineffective online exploration.<n>We propose GoldenStart (GSFlow), a policy distillation method with Q-guided priors and explicit entropy control.<n>Our integrated framework demonstrates that by designing the generative startpoint and explicitly controlling policy entropy, it is possible to achieve efficient and exploratory policies.
  arXiv  Detail & Related papers  (2026-03-15T06:39:09Z)
• TreePS-RAG: Tree-based Process Supervision for Reinforcement Learning in Agentic RAG [71.06073770344732]
  Agentic retrieval-augmented generation (RAG) formulates question answering as a multi-step interaction between reasoning and information retrieval.<n>We present TreePS-RAG, an online, tree-based RL framework for agentic RAG that enables step-wise credit assignment while retaining outcome-only rewards.
  arXiv  Detail & Related papers  (2026-01-11T14:07:30Z)
• KBQA-R1: Reinforcing Large Language Models for Knowledge Base Question Answering [64.62317305868264]
  We present textbfKBQA-R1, a framework that shifts the paradigm from text imitation to interaction optimization via Reinforcement Learning.<n>Treating KBQA as a multi-turn decision process, our model learns to navigate the knowledge base using a list of actions.<n>Experiments on WebQSP, GrailQA, and GraphQuestions demonstrate that KBQA-R1 achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2025-12-10T17:45:42Z)
• KAQG: A Knowledge-Graph-Enhanced RAG for Difficulty-Controlled Question Generation [0.0]
  This study introduces Knowledge Augmented Question Generation (KAQG)<n>It integrates Item Response Theory, abbreviated as IRT, Bloom's taxonomy, and knowledge graphs into a multi-agent Retrieval-Augmented Generation system.<n>The proposed approach overcomes limitations of existing methods by enabling fine-grained control over item difficulty, psychometric calibration, and cognitive alignment.
  arXiv  Detail & Related papers  (2025-05-12T14:42:19Z)
• Mildly Conservative Q-Learning for Offline Reinforcement Learning [63.2183622958666]
  offline reinforcement learning (RL) defines the task of learning from a static logged dataset without continually interacting with the environment. Existing approaches, penalizing the unseen actions or regularizing with the behavior policy, are too pessimistic. We propose Mildly Conservative Q-learning (MCQ), where OOD actions are actively trained by assigning them proper pseudo Q values.
  arXiv  Detail & Related papers  (2022-06-09T19:44:35Z)
• Offline Reinforcement Learning with Implicit Q-Learning [85.62618088890787]
  Current offline reinforcement learning methods need to query the value of unseen actions during training to improve the policy. We propose an offline RL method that never needs to evaluate actions outside of the dataset. This method enables the learned policy to improve substantially over the best behavior in the data through generalization.
  arXiv  Detail & Related papers  (2021-10-12T17:05:05Z)
• Robust Deep Reinforcement Learning against Adversarial Perturbations on State Observations [88.94162416324505]
  A deep reinforcement learning (DRL) agent observes its states through observations, which may contain natural measurement errors or adversarial noises. Since the observations deviate from the true states, they can mislead the agent into making suboptimal actions. We show that naively applying existing techniques on improving robustness for classification tasks, like adversarial training, is ineffective for many RL tasks.
  arXiv  Detail & Related papers  (2020-03-19T17:59:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.