From Memorization to Absorption: Mixed-Policy RL for Continual Knowledge Injection

• URL: http://arxiv.org/abs/2608.25243v1
• Date: Wed, 26 Aug 2026 00:04:46 GMT
• Title: From Memorization to Absorption: Mixed-Policy RL for Continual Knowledge Injection
• Abstract summary: We propose Golden-GRPO Injection (GRIN), a three-stage self-learning framework for continual knowledge injection.<n>Golden-GRPO is a mixed-policy reinforcement learning algorithm designed specifically for knowledge injection.<n>GRIN substantially outperforms SFT and mixed-policy RL baselines on the harder question types while matching them on basic fact recall.
• Score: 8.467797533190216
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Continual knowledge injection is essential for keeping large language models up-to-date in a fast-evolving world. Existing methods rely on supervised fine-tuning (SFT), which memorizes injected facts in their training format but fails to generalize across paraphrasing, document combinations, and reasoning. To address this, we propose Golden-GRPO Injection (GRIN), a three-stage self-learning framework for continual knowledge injection. Golden-GRPO is a mixed-policy reinforcement learning algorithm designed specifically for knowledge injection, which injects a golden answer to provide learning signal even when on-policy rollouts fail on novel facts. We further introduce Blank and Counter, two document-level benchmarks targeting novel acquisition and counterfactual overwrite respectively, each evaluating single-fact recall, multi-source retrieval, and inferential reasoning. Our experiments establish a clear empirical claim: mixed-policy reinforcement learning enables knowledge absorption beyond what supervised fine-tuning can achieve. GRIN substantially outperforms SFT and mixed-policy RL baselines on the harder question types while matching them on basic fact recall.

Related papers

• RING: Retrieval-Internalized Generation for Continual Large-Scale Knowledge Injection [88.56816599695391]
  Retrieval-augmented generation (RAG) improves factuality but adds latency and engineering overhead at serving time.<n>We propose RING (Retrieval-Internalized Generation), a holistic paradigm spanning both architecture and training.
  arXiv  Detail & Related papers  (2026-08-03T03:00:43Z)
• DeepRefine: Agent-Compiled Knowledge Refinement via Reinforcement Learning [60.35032268606525]
  Agent-compiled knowledge bases provide persistent external knowledge for large language model (LLM) agents in downstream tasks.<n>Yet their quality is systematically limited by emphincompleteness, emphincorrectness, and emphredundancy
  arXiv  Detail & Related papers  (2026-05-11T12:48:31Z)
• EruDiff: Refactoring Knowledge in Diffusion Models for Advanced Text-to-Image Synthesis [49.883192716595026]
  EruDiff aims to rectify the knowledge distribution of intractable implicit prompts with that of well-defined explicit anchors.<n>We develop the Diffusion Knowledge Distribution Matching (DK-DM) to register the knowledge distribution of implicit prompts with that of well-defined explicit anchors.<n>We employ the Negative-Only Reinforcement Learning (NO-RL) strategy for fine-grained correction.
  arXiv  Detail & Related papers  (2026-03-21T14:04:08Z)
• Retrieval-augmented Prompt Learning for Pre-trained Foundation Models [101.13972024610733]
  We present RetroPrompt, which aims to achieve a balance between memorization and generalization.<n>Unlike traditional prompting methods, RetroPrompt incorporates a retrieval mechanism throughout the input, training, and inference stages.<n>We conduct comprehensive experiments on a variety of datasets across natural language processing and computer vision tasks to demonstrate the superior performance of our proposed approach.
  arXiv  Detail & Related papers  (2025-12-23T08:15:34Z)
• Prompting is not Enough: Exploring Knowledge Integration and Controllable Generation [89.65955788873532]
  Open-domain question answering (OpenQA) represents a cornerstone in natural language processing (NLP)<n>We propose a novel framework named GenKI, which aims to improve the OpenQA performance by exploring Knowledge Integration and controllable Generation.
  arXiv  Detail & Related papers  (2025-05-26T08:18:33Z)
• Latent Paraphrasing: Perturbation on Layers Improves Knowledge Injection in Language Models [54.385486006684495]
  LaPael is a latent-level paraphrasing method that applies input-dependent noise to early Large Language Models layers. Our experiments on question-answering benchmarks demonstrate that LaPael improves knowledge injection over standard fine-tuning and existing noise-based approaches.
  arXiv  Detail & Related papers  (2024-11-01T15:47:05Z)
• KEHRL: Learning Knowledge-Enhanced Language Representations with Hierarchical Reinforcement Learning [32.086825891769585]
  Knowledge-enhanced pre-trained language models (KEPLMs) leverage relation triples from knowledge graphs (KGs) Previous works treat knowledge enhancement as two independent operations, i.e., knowledge injection and knowledge integration. This paper jointly addresses the problems of detecting positions for knowledge injection and integrating external knowledge into the model in order to avoid injecting inaccurate or irrelevant knowledge.
  arXiv  Detail & Related papers  (2024-06-24T07:32:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.