Value Memory Graph: A Graph-Structured World Model for Offline Reinforcement Learning

• URL: http://arxiv.org/abs/2206.04384v3
• Date: Tue, 2 May 2023 14:15:02 GMT
• Title: Value Memory Graph: A Graph-Structured World Model for Offline Reinforcement Learning
• Abstract summary: Reinforcement Learning (RL) methods are typically applied directly in environments to learn policies. We aim to build a simple and discrete world model that abstracts the original environment. Our experiments on the D4RL benchmark show that VMG can outperform state-of-the-art offline RL methods in several goal-oriented tasks.
• Score: 35.5986167075913
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reinforcement Learning (RL) methods are typically applied directly in environments to learn policies. In some complex environments with continuous state-action spaces, sparse rewards, and/or long temporal horizons, learning a good policy in the original environments can be difficult. Focusing on the offline RL setting, we aim to build a simple and discrete world model that abstracts the original environment. RL methods are applied to our world model instead of the environment data for simplified policy learning. Our world model, dubbed Value Memory Graph (VMG), is designed as a directed-graph-based Markov decision process (MDP) of which vertices and directed edges represent graph states and graph actions, separately. As state-action spaces of VMG are finite and relatively small compared to the original environment, we can directly apply the value iteration algorithm on VMG to estimate graph state values and figure out the best graph actions. VMG is trained from and built on the offline RL dataset. Together with an action translator that converts the abstract graph actions in VMG to real actions in the original environment, VMG controls agents to maximize episode returns. Our experiments on the D4RL benchmark show that VMG can outperform state-of-the-art offline RL methods in several goal-oriented tasks, especially when environments have sparse rewards and long temporal horizons. Code is available at https://github.com/TsuTikgiau/ValueMemoryGraph

Related papers

• Enhancing LLMs for Graph Tasks via Graph-aware LoRA Generation [80.56829418752513]
  Language models (LMs) generalize flexibly via a unified input-output interface, motivating recent attempts to adapt LMs to graph tasks.<n>We propose a novel weight-level information injection paradigm for adapting LMs to graph tasks.<n>We introduce GaRA, a Graph-aware LoRA generation model, which constructs low-rank weight updates conditioned on the original graph structures.
  arXiv  Detail & Related papers  (2026-06-21T10:40:33Z)
• Approximate Subgraph Matching with Neural Graph Representations and Reinforcement Learning [15.3743761404157]
  This paper proposes a Reinforcement Learning based Approximate Subgraph Matching (RL-ASM) algorithm.<n>Our model is built upon the branch-and-bound algorithm that selects one pair of nodes from the two input graphs at a time for potential matches.<n>Experiments on both synthetic and real-world datasets demonstrate that our RL-ASM outperforms existing methods in terms of effectiveness and efficiency.
  arXiv  Detail & Related papers  (2026-03-18T21:53:23Z)
• Incorporating Spatial Information into Goal-Conditioned Hierarchical Reinforcement Learning via Graph Representations [37.10671332775445]
  The integration of graphs with Goal-conditioned Reinforcement Learning (GCHRL) has recently gained attention.<n>Existing approaches typically rely on domain-specific knowledge to construct these graphs.<n>This paper proposes a solution by developing a graph encoder-decoder to evaluate unseen states.
  arXiv  Detail & Related papers  (2025-11-14T00:58:39Z)
• G1: Teaching LLMs to Reason on Graphs with Reinforcement Learning [58.73279333365234]
  Reinforcement Learning (RL) on synthetic graph-theoretic tasks can significantly scale graph reasoning abilities.<n>With RL on Erdos, G1 obtains substantial improvements in graph reasoning, where our finetuned 3B model even outperforms Qwen2.5-72B-Instruct (24x size)<n>Our findings offer an efficient, scalable path for building strong graph reasoners by finetuning LLMs with RL on graph-theoretic tasks.
  arXiv  Detail & Related papers  (2025-05-24T04:33:41Z)
• Compile Scene Graphs with Reinforcement Learning [69.36723767339001]
  Next token prediction is the fundamental principle for training large language models (LLMs) We introduce R1-SGG, a multimodal LLM (M-LLM) trained via supervised fine-tuning (SFT) on the scene graph dataset. We design a graph-centric reward function that integrates node-level rewards, edge-level rewards, and a format consistency reward.
  arXiv  Detail & Related papers  (2025-04-18T10:46:22Z)
• Dynamic and Textual Graph Generation Via Large-Scale LLM-based Agent Simulation [70.60461609393779]
  GraphAgent-Generator (GAG) is a novel simulation-based framework for dynamic graph generation. Our framework effectively replicates seven macro-level structural characteristics in established network science theories. It supports generating graphs with up to nearly 100,000 nodes or 10 million edges, with a minimum speed-up of 90.4%.
  arXiv  Detail & Related papers  (2024-10-13T12:57:08Z)
• Bridging Environments and Language with Rendering Functions and Vision-Language Models [7.704773649029078]
  Vision-language models (VLMs) have tremendous potential for grounding language. This paper introduces a novel decomposition of the problem of building language-conditioned agents (LCAs) We also explore several enhancements to the speed and quality of VLM-based LCAs.
  arXiv  Detail & Related papers  (2024-09-24T12:24:07Z)
• SAPG: Split and Aggregate Policy Gradients [37.433915947580076]
  We propose a new on-policy RL algorithm that can effectively leverage large-scale environments by splitting them into chunks and fusing them back together via importance sampling. Our algorithm, termed SAPG, shows significantly higher performance across a variety of challenging environments where vanilla PPO and other strong baselines fail to achieve high performance.
  arXiv  Detail & Related papers  (2024-07-29T17:59:50Z)
• GLBench: A Comprehensive Benchmark for Graph with Large Language Models [41.89444363336435]
  We introduce GLBench, the first comprehensive benchmark for evaluating GraphLLM methods in both supervised and zero-shot scenarios. GLBench provides a fair and thorough evaluation of different categories of GraphLLM methods, along with traditional baselines such as graph neural networks.
  arXiv  Detail & Related papers  (2024-07-10T08:20:47Z)
• Highway Graph to Accelerate Reinforcement Learning [18.849312069946993]
  We propose a novel graph structure, named highway graph, to model the state transition. By integrating the highway graph into RL, the RL training can be remarkably accelerated in the early stages. Deep neural network-based agent is trained using the highway graph, resulting in better generalization and lower storage costs.
  arXiv  Detail & Related papers  (2024-05-20T02:09:07Z)
• Action-Quantized Offline Reinforcement Learning for Robotic Skill Learning [68.16998247593209]
  offline reinforcement learning (RL) paradigm provides recipe to convert static behavior datasets into policies that can perform better than the policy that collected the data. In this paper, we propose an adaptive scheme for action quantization. We show that several state-of-the-art offline RL methods such as IQL, CQL, and BRAC improve in performance on benchmarks when combined with our proposed discretization scheme.
  arXiv  Detail & Related papers  (2023-10-18T06:07:10Z)
• Graph Decision Transformer [83.76329715043205]
  Graph Decision Transformer (GDT) is a novel offline reinforcement learning approach. GDT models the input sequence into a causal graph to capture potential dependencies between fundamentally different concepts. Our experiments show that GDT matches or surpasses the performance of state-of-the-art offline RL methods on image-based Atari and OpenAI Gym.
  arXiv  Detail & Related papers  (2023-03-07T09:10:34Z)
• Graph Backup: Data Efficient Backup Exploiting Markovian Transitions [24.765707880860543]
  A key to data-efficient RL is good value estimation, but current methods fail to fully utilise the structure of the trajectory data gathered from the environment. In this paper, we treat the transition data of the MDP as a graph, and define a novel backup operator, Graph Backup, which exploits this graph structure for better value estimation. Our method, when combined with popular value-based methods, provides improved performance over one-step and multi-step methods on a suite of data-efficient RL benchmarks.
  arXiv  Detail & Related papers  (2022-05-31T14:26:00Z)
• Self-supervised Graph-level Representation Learning with Local and Global Structure [71.45196938842608]
  We propose a unified framework called Local-instance and Global-semantic Learning (GraphLoG) for self-supervised whole-graph representation learning. Besides preserving the local similarities, GraphLoG introduces the hierarchical prototypes to capture the global semantic clusters. An efficient online expectation-maximization (EM) algorithm is further developed for learning the model.
  arXiv  Detail & Related papers  (2021-06-08T05:25:38Z)
• Heuristic Semi-Supervised Learning for Graph Generation Inspired by Electoral College [80.67842220664231]
  We propose a novel pre-processing technique, namely ELectoral COllege (ELCO), which automatically expands new nodes and edges to refine the label similarity within a dense subgraph. In all setups tested, our method boosts the average score of base models by a large margin of 4.7 points, as well as consistently outperforms the state-of-the-art.
  arXiv  Detail & Related papers  (2020-06-10T14:48:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.