Procedural Memory Distillation: Online Reflection for Self-Improving Language Models

• URL: http://arxiv.org/abs/2607.01480v1
• Date: Wed, 01 Jul 2026 21:20:57 GMT
• Title: Procedural Memory Distillation: Online Reflection for Self-Improving Language Models
• Abstract summary: Reinforcement learning with verifiable rewards (RLVR) evaluates each rollout against a verifier and updates the policy from that episode-level signal.<n>Across episodes and epochs, the model repeatedly encounters related problems under a changing policy, producing cross-episode signals that episode-local updates cannot capture.<n>We propose Procedural Memory Distillation (PMD), which converts these crossepisode signals into reusable procedural memory and distills it into the policy's weights during training.
• Score: 51.938248694868584
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reinforcement learning with verifiable rewards (RLVR), along with recent selfdistillation variants such as SDPO, evaluates each rollout against a verifier and updates the policy from that episode-level signal. However, the richer procedural information in the rollout is rarely retained or reused. Across episodes and epochs, the model repeatedly encounters related problems under a changing policy, producing cross-episode signals that episode-local updates cannot capture: which strategies consistently pass verification, which failure modes persist, which patterns recur. We propose Procedural Memory Distillation (PMD), which converts these crossepisode signals into reusable procedural memory and distills it into the policy's weights during training. This memory functions as a training scaffold, absorbed into the policy itself, yielding a memory-free model at inference. PMD organizes the memory at three levels of abstraction: raw trajectories, self-reflected strategies and lessons, and higher-level behavioral patterns that recur across problems, all extracted online from the model's own trajectories. A memory-conditioned self-teacher draws on the accumulated experience to supervise the student on its own rollouts, enabling student to progressively internalize procedural knowledge within its parameters. The central design principle is co-evolution: the policy generates rollouts that update the memory, and memory shapes the supervision that updates the policy. Empirically, across Qwen3-8B and OLMo3-Instruct-7B, PMD improves over SDPO by 3.8-5.5% on SCIKNOWEVAL and 7.9-13.6% on LIVECODEBENCH. Co-evolution powers these gains: freezing either the memory or the policy trails PMD by more than 10% across SCIKNOWEVAL domains.

Related papers

• OnEvoMemory: Evolving Memory through Online Robot Rollouts for Pretrained Robot Policies [2.9005223064604078]
  Long-horizon robot manipulation requires policies to track completed subtasks and critical interaction events.<n>We propose OnEvoMemory, a value-guided memory module for pretrained robot policies.
  arXiv  Detail & Related papers  (2026-08-09T14:53:19Z)
• AttriMem: Attribution-Guided Process Feedback for Agent Memory Learning [13.898789808338634]
  We propose an attribution-guided process-feedback framework for learning memory-construction policies with RL.<n>AttriMem augments the global outcome reward with local rewards derived from token-level contributions to the final answer.
  arXiv  Detail & Related papers  (2026-07-23T09:35:34Z)
• EvolveMem:Self-Evolving Memory Architecture via AutoResearch for LLM Agents [80.59925936278162]
  We present EvolveMem, a self-evolving memory architecture that exposes its full retrieval configuration as a structured action space optimized by an LLM-powered diagnosis module.<n>This closedloop selfevolution realizes an AutoResearch process: the system autonomously conducts iterative research cycles on its own architecture, replacing manual configuration tuning.
  arXiv  Detail & Related papers  (2026-05-13T17:12:44Z)
• MemEvoBench: Benchmarking Memory MisEvolution in LLM Agents [78.95081012334116]
  Large Language Models (LLMs) with persistent memory enhances interaction continuity and personalization but introduces new safety risks.<n> contaminated or biased memory accumulation can trigger abnormal agent behaviors.<n>MemeEvoBench is the first benchmark evaluating memory safety in LLM agents against adversarial memory injection, noisy tool outputs, and biased feedback.
  arXiv  Detail & Related papers  (2026-04-17T07:29:52Z)
• MemRL: Self-Evolving Agents via Runtime Reinforcement Learning on Episodic Memory [46.632646462295234]
  We propose MemRL, a framework that enables agents to self-evolve via non-parametric reinforcement learning on episodic memory.<n>MemRL employs a Two-Phase Retrieval mechanism that filters candidates by semantic relevance and then selects them based on learned Q-values.<n>Our analysis experiments confirm that MemRL effectively reconciles the stability-plasticity dilemma, enabling continuous runtime improvement without weight updates.
  arXiv  Detail & Related papers  (2026-01-06T17:14:50Z)
• Memento 2: Learning by Stateful Reflective Memory [4.7052412989773975]
  We study continual learning in large language model (LLM) based agents that integrate episodic memory with reinforcement learning.<n>We focus on reflection, the ability of an agent to revisit past experience and adjust how it selects future actions.<n>We introduce the Stateful Reflective Decision Process (SRDP), in which an agent maintains and updates episodic memory and alternates between writing new experiences to memory and reading relevant cases to guide decisions.
  arXiv  Detail & Related papers  (2025-12-27T22:15:03Z)
• Memp: Exploring Agent Procedural Memory [72.41472703974935]
  Large Language Models (LLMs) based agents excel at diverse tasks, yet they suffer from brittle procedural memory that is manually engineered or entangled in static parameters.<n>We propose Memp that distills past agent trajectories into both fine-grained, step-by-step instructions and higher-level, script-like abstractions.<n>We show that as the memory repository is refined, agents achieve steadily higher success rates and greater efficiency on analogous tasks.
  arXiv  Detail & Related papers  (2025-08-08T16:20:56Z)
• Offline Reinforcement Learning with Implicit Q-Learning [85.62618088890787]
  Current offline reinforcement learning methods need to query the value of unseen actions during training to improve the policy. We propose an offline RL method that never needs to evaluate actions outside of the dataset. This method enables the learned policy to improve substantially over the best behavior in the data through generalization.
  arXiv  Detail & Related papers  (2021-10-12T17:05:05Z)
• Evolutionary Stochastic Policy Distillation [139.54121001226451]
  We propose a new method called Evolutionary Policy Distillation (ESPD) to solve GCRS tasks. ESPD enables a target policy to learn from a series of its variants through the technique of policy distillation (PD) The experiments based on the MuJoCo control suite show the high learning efficiency of the proposed method.
  arXiv  Detail & Related papers  (2020-04-27T16:19:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.