MemReward: Graph-Based Experience Memory for LLM Reward Prediction with Limited Labels

• URL: http://arxiv.org/abs/2603.19310v1
• Date: Fri, 13 Mar 2026 18:00:16 GMT
• Title: MemReward: Graph-Based Experience Memory for LLM Reward Prediction with Limited Labels
• Abstract summary: We introduce MemReward, a graph-based experience memory framework.<n>An initial LLM policy generates rollouts for each query, each comprising a thinking process and a final answer.<n>A GNN trained on labeled nodes propagates rewards to unlabeled rollouts during online optimization.
• Score: 32.633330452668766
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Training large language models (LLMs) for complex reasoning via reinforcement learning requires reward labels that specify whether the generated rollouts are correct. However, obtaining reward labels at scale often requires expensive human labeling or time-consuming verification procedures; for instance, evaluating mathematical proofs demands expert review, while open-ended question answering lacks definitive ground truth. When reward labels are limited, the effectiveness of reinforcement learning fine-tuning is constrained by the scarcity of reward labels. We introduce MemReward, a graph-based experience memory framework: an initial LLM policy generates rollouts for each query, each comprising a thinking process and a final answer, and these rollouts are stored as experience memory. Queries, thinking processes, and answers form nodes in a heterogeneous graph with similarity and structural edges; a GNN trained on labeled nodes propagates rewards to unlabeled rollouts during online optimization. Experiments on Qwen2.5-3B and 1.5B across mathematics, question answering, and code generation demonstrate that MemReward, with only 20% labels, achieves 97.3% of Oracle performance on 3B and 96.6% on 1.5B, surpassing Oracle on out-of-domain tasks. Performance scales smoothly with label budget, reaching 99.4% of Oracle at 70% labels.

Related papers

• TeaRAG: A Token-Efficient Agentic Retrieval-Augmented Generation Framework [62.66056331998838]
  TeaRAG is a token-efficient agentic RAG framework capable of compressing both retrieval content and reasoning steps.<n>Our reward function evaluates the knowledge sufficiency by a knowledge matching mechanism, while penalizing excessive reasoning steps.
  arXiv  Detail & Related papers  (2025-11-07T16:08:34Z)
• RESTRAIN: From Spurious Votes to Signals -- Self-Driven RL with Self-Penalization [52.01526898310723]
  We introduce RESTRAIN, a self-penalizing RL framework that converts the absence of gold labels into a useful learning signal.<n>Instead of overcommitting to spurious majority votes, RESTRAIN exploits signals from the model's entire answer distribution.<n>On challenging reasoning benchmarks, RESTRAIN delivers large gains using only unlabeled data.
  arXiv  Detail & Related papers  (2025-10-02T16:24:01Z)
• BiRQ: Bi-Level Self-Labeling Random Quantization for Self-Supervised Speech Recognition [63.45645200463539]
  BiRQ is a bilevel SSL framework that combines the efficiency of BEST-RQ with the refinement benefits of HuBERT-style label enhancement.<n>We validate our method on various datasets, including 960-hour LibriSpeech, 150-hour AMI meetings and 5,000-hour YODAS.
  arXiv  Detail & Related papers  (2025-09-18T21:09:29Z)
• Leveraging Large Language Models for Effective Label-free Node Classification in Text-Attributed Graphs [10.538099379851198]
  Locle is an active self-training framework that does Label-free node Classification with LLMs cost-Effectively.<n>It iteratively identifies small sets of "critical" samples using GNNs and extracts informative pseudo-labels for them with both LLMs and GNNs.<n>It significantly outperforms state-of-the-art methods under the same query budget to LLMs in terms of label-free node classification.
  arXiv  Detail & Related papers  (2024-12-16T17:04:40Z)
• Learning from Label Proportions: Bootstrapping Supervised Learners via Belief Propagation [18.57840057487926]
  Learning from Label Proportions (LLP) is a learning problem where only aggregate level labels are available for groups of instances, called bags, during training. This setting arises in domains like advertising and medicine due to privacy considerations. We propose a novel algorithmic framework for this problem that iteratively performs two main steps.
  arXiv  Detail & Related papers  (2023-10-12T06:09:26Z)
• Is margin all you need? An extensive empirical study of active learning on tabular data [66.18464006872345]
  We analyze the performance of a variety of active learning algorithms on 69 real-world datasets from the OpenML-CC18 benchmark. Surprisingly, we find that the classical margin sampling technique matches or outperforms all others, including current state-of-art.
  arXiv  Detail & Related papers  (2022-10-07T21:18:24Z)
• Information Gain Propagation: a new way to Graph Active Learning with Soft Labels [26.20597165750861]
  Graph Neural Networks (GNNs) have achieved great success in various tasks, but their performance highly relies on a large number of labeled nodes. We propose GNN-based Active Learning (AL) methods to improve the labeling efficiency by selecting the most valuable nodes to label. Our method significantly outperforms the state-of-the-art GNN-based AL methods in terms of both accuracy and labeling cost.
  arXiv  Detail & Related papers  (2022-03-02T13:28:25Z)
• Accelerating BERT Inference for Sequence Labeling via Early-Exit [65.7292767360083]
  We extend the recent successful early-exit mechanism to accelerate the inference of PTMs for sequence labeling tasks. We also propose a token-level early-exit mechanism that allows partial tokens to exit early at different layers. Our approach can save up to 66%-75% inference cost with minimal performance degradation.
  arXiv  Detail & Related papers  (2021-05-28T14:39:26Z)
• SoQal: Selective Oracle Questioning in Active Learning [17.58391771585294]
  We propose SoQal, a questioning strategy that determines when a label should be requested from an oracle. We perform experiments on five publically-available datasets and illustrate SoQal's superiority relative to baseline approaches.
  arXiv  Detail & Related papers  (2020-04-22T09:53:55Z)
• SoQal: Selective Oracle Questioning for Consistency Based Active Learning of Cardiac Signals [17.58391771585294]
  Clinical settings are often characterized by abundant unlabelled data and limited labelled data. One way to mitigate this burden is via active learning (AL) which involves the (a) acquisition and (b) annotation of informative unlabelled instances. We show that BALC can outperform start-of-the-art acquisition functions such as BALD, and SoQal outperforms baseline methods even in the presence of a noisy oracle.
  arXiv  Detail & Related papers  (2020-04-20T18:20:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.