SPARK: Self-Play with Asymmetric Reward from Knowledge Graphs

• URL: http://arxiv.org/abs/2605.05546v1
• Date: Thu, 07 May 2026 00:51:36 GMT
• Title: SPARK: Self-Play with Asymmetric Reward from Knowledge Graphs
• Abstract summary: Self-play reinforcement learning has shown strong performance in domains with formally verifiable structure.<n>We propose SPARK, a framework that automatically constructs a unified knowledge graph (KG) from multi-document scientific literature.<n>A single small vision-language model (sVLM) alternates between Proposer and Solver roles under information asymmetry against a fixed KG.<n>Results show that SPARK consistently outperforms flat-corpus-based self-play baselines.
• Score: 2.1636750837987195
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Self-play reinforcement learning has shown strong performance in domains with formally verifiable structure, such as mathematics and coding, where both problem generation and reward computation can be grounded in explicit rules. Extending this paradigm to scientific literature is more challenging: the relationships among multi-modal elements within and across documents are rarely made explicit in text, which makes automatic generation of relational reasoning questions difficult and weakens the reliability of reward signals. We propose SPARK (Self-Play with Asymmetric Reward from Knowledge Graphs), a framework that automatically constructs a unified knowledge graph (KG) from multi-document scientific literature and uses it as the structural basis for self-play. KG paths over multimodal nodes serve as a source for generating relational reasoning questions, and structured facts stored in the KG provide a basis for verifiable reward computation. A single small vision-language model (sVLM) alternates between Proposer and Solver roles under information asymmetry against a fixed KG, a design that we believe can be naturally extended toward online adaptation in future work. We evaluate SPARK on public benchmarks and a self-constructed cross-document multi-hop QA dataset. Results show that SPARK consistently outperforms flat-corpus-based self-play baselines, and the performance gap widens as hop count increases, suggesting that KG-structure grounding contributes to relational multi-hop reasoning beyond what unstructured corpus grounding can provide.

Related papers

• Structure-Internalized Rule Language Model for Faithful Knowledge Graph Reasoning [6.421907131279842]
  Knowledge Graph Reasoning (KGR) aims to discover latent facts by leveraging the structural evidence available in Knowledge Graphs ( KGs)<n>Recent studies have demonstrated that Large Language Models (LLMs) can achieve remarkable progress on KGR tasks via flexible in-context learning.<n>We propose a Structure-Internalized Rule Language Model (SIRLM) to address the representation inconsistency between KG structural context and LLM parametric knowledge.
  arXiv  Detail & Related papers  (2026-08-18T07:21:20Z)
• HyGRL: Adaptive Hybrid Graph Reasoning for Multi-Entity Questions [6.383838707344047]
  We propose textbffwa, a unified framework that embeds unstructured text into structured knowledge graphs.<n> Reasoning is formulated as adaptive structure induction, learned via a robust two-stage process.<n>Experiments demonstrate that fwa effectively merges textual richness with structural knowledge, outperforming SOTA baselines in answer accuracy and reasoning fidelity.
  arXiv  Detail & Related papers  (2026-07-04T07:07:15Z)
• RAGA: Reading-And-Graph-building-Agent for Autonomous Knowledge Graph Construction and Retrieval-Augmented Generation [1.7016397531234393]
  RAGA (Reading And Graph-building Agent) is an autonomous knowledge graph construction and retrieval framework.<n>It provides full KG lifecycle CRUD operations and embeds a Read-Search-Verify-Verify cognitive constraint into a ReAct tool loop.<n>Preliminary experiments on a subset of the QASPER scientific QA dataset indicate that RAGA's fusion retrieval outperforms zero-shot baselines.
  arXiv  Detail & Related papers  (2026-05-16T16:42:50Z)
• A Human-in-the-Loop, LLM-Centered Architecture for Knowledge-Graph Question Answering [41.99844472131922]
  Large Language Models excel at language understanding but are limited in knowledge-intensive domains.<n>This work introduces an interactive framework in which LLMs generate and explain Cypher graph queries.
  arXiv  Detail & Related papers  (2026-02-05T10:10:19Z)
• Plan Then Retrieve: Reinforcement Learning-Guided Complex Reasoning over Knowledge Graphs [52.16166558205338]
  Graph-RFT is a novel two-stage reinforcement fine-tuning KGQA framework with a 'plan-KGsearch-and-Websearch-during-think' paradigm.<n>It enables LLMs to perform autonomous planning and adaptive retrieval scheduling across KG and web sources under incomplete knowledge conditions.
  arXiv  Detail & Related papers  (2025-10-23T16:04:13Z)
• Enrich-on-Graph: Query-Graph Alignment for Complex Reasoning with LLM Enriching [61.824094419641575]
  Large Language Models (LLMs) struggle with hallucinations and factual errors in knowledge-intensive scenarios like knowledge graph question answering (KGQA)<n>We attribute this to the semantic gap between structured knowledge graphs (KGs) and unstructured queries, caused by inherent differences in their focuses and structures.<n>Existing methods usually employ resource-intensive, non-scalable reasoning on vanilla KGs, but overlook this gap.<n>We propose a flexible framework, Enrich-on-Graph (EoG), which leverages LLMs' prior knowledge to enrich KGs, bridge the semantic gap between graphs and queries.
  arXiv  Detail & Related papers  (2025-09-25T06:48:52Z)
• GRIL: Knowledge Graph Retrieval-Integrated Learning with Large Language Models [59.72897499248909]
  We propose a novel graph retriever trained end-to-end with Large Language Models (LLMs)<n>Within the extracted subgraph, structural knowledge and semantic features are encoded via soft tokens and the verbalized graph, respectively, which are infused into the LLM together.<n>Our approach consistently achieves state-of-the-art performance, validating the strength of joint graph-LLM optimization for complex reasoning tasks.
  arXiv  Detail & Related papers  (2025-09-20T02:38:00Z)
• Knowledge Graph-extended Retrieval Augmented Generation for Question Answering [10.49712834719005]
  This paper proposes a system that integrates Large Language Models (LLMs) and Knowledge Graphs (KGs) without requiring training.<n>The resulting approach can be classified as a specific form of a Retrieval Augmented Generation (RAG) with a KG.<n>It includes a question decomposition module to enhance multi-hop information retrieval and answerability.
  arXiv  Detail & Related papers  (2025-04-11T18:03:02Z)
• Grounding LLM Reasoning with Knowledge Graphs [4.279373869671241]
  We propose integrating reasoning strategies with Knowledge Graphs to anchor every step or "thought" of the reasoning chains in KG data.<n>We evaluate both agentic and automated search methods across several reasoning strategies, including Chain-of-Thought (CoT), Tree-of-Thought (ToT), and Graph-of-Thought (GoT)<n>Our experiments demonstrate that this approach consistently outperforms baseline models.
  arXiv  Detail & Related papers  (2025-02-18T19:20:46Z)
• Ontology-grounded Automatic Knowledge Graph Construction by LLM under Wikidata schema [60.42231674887294]
  We propose an ontology-grounded approach to Knowledge Graph (KG) construction using Large Language Models (LLMs) on a knowledge base.<n>We ground generation of KG with the authored ontology based on extracted relations to ensure consistency and interpretability.<n>Our work presents a promising direction for scalable KG construction pipeline with minimal human intervention, that yields high quality and human-interpretable KGs.
  arXiv  Detail & Related papers  (2024-12-30T13:36:05Z)
• Decoding on Graphs: Faithful and Sound Reasoning on Knowledge Graphs through Generation of Well-Formed Chains [66.55612528039894]
  Knowledge Graphs (KGs) can serve as reliable knowledge sources for question answering (QA) We present DoG (Decoding on Graphs), a novel framework that facilitates a deep synergy between LLMs and KGs. Experiments across various KGQA tasks with different background KGs demonstrate that DoG achieves superior and robust performance.
  arXiv  Detail & Related papers  (2024-10-24T04:01:40Z)
• GIVE: Structured Reasoning of Large Language Models with Knowledge Graph Inspired Veracity Extrapolation [108.2008975785364]
  Graph Inspired Veracity Extrapolation (GIVE) is a novel reasoning method that merges parametric and non-parametric memories to improve accurate reasoning with minimal external input.<n>GIVE guides the LLM agent to select the most pertinent expert data (observe), engage in query-specific divergent thinking (reflect), and then synthesize this information to produce the final output (speak)
  arXiv  Detail & Related papers  (2024-10-11T03:05:06Z)
• Learning Federated Neural Graph Databases for Answering Complex Queries from Distributed Knowledge Graphs [53.03085605769093]
  We propose to learn Federated Neural Graph DataBase (FedNGDB), a pioneering systematic framework that empowers privacy-preserving reasoning over multi-source graph data.<n>FedNGDB leverages federated learning to collaboratively learn graph representations across multiple sources, enriching relationships between entities, and improving the overall quality of graph data.
  arXiv  Detail & Related papers  (2024-02-22T14:57:44Z)
• ReasoningLM: Enabling Structural Subgraph Reasoning in Pre-trained Language Models for Question Answering over Knowledge Graph [142.42275983201978]
  We propose a subgraph-aware self-attention mechanism to imitate the GNN for performing structured reasoning. We also adopt an adaptation tuning strategy to adapt the model parameters with 20,000 subgraphs with synthesized questions. Experiments show that ReasoningLM surpasses state-of-the-art models by a large margin, even with fewer updated parameters and less training data.
  arXiv  Detail & Related papers  (2023-12-30T07:18:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.