Diffusion Models for Reinforcement Learning: Foundations, Taxonomy, and Development

• URL: http://arxiv.org/abs/2510.12253v1
• Date: Tue, 14 Oct 2025 08:03:46 GMT
• Title: Diffusion Models for Reinforcement Learning: Foundations, Taxonomy, and Development
• Authors: Changfu Xu, Jianxiong Guo, Yuzhu Liang, Haiyang Huang, Haodong Zou, Xi Zheng, Shui Yu, Xiaowen Chu, Jiannong Cao, Tian Wang,
• Abstract summary: Diffusion Models (DMs) offer key advantages for reinforcement learning (RL)<n>This survey delivers a comprehensive and up-to-date synthesis of diffusion-based RL.
• Score: 32.14985932997508
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Diffusion Models (DMs), as a leading class of generative models, offer key advantages for reinforcement learning (RL), including multi-modal expressiveness, stable training, and trajectory-level planning. This survey delivers a comprehensive and up-to-date synthesis of diffusion-based RL. We first provide an overview of RL, highlighting its challenges, and then introduce the fundamental concepts of DMs, investigating how they are integrated into RL frameworks to address key challenges in this research field. We establish a dual-axis taxonomy that organizes the field along two orthogonal dimensions: a function-oriented taxonomy that clarifies the roles DMs play within the RL pipeline, and a technique-oriented taxonomy that situates implementations across online versus offline learning regimes. We also provide a comprehensive examination of this progression from single-agent to multi-agent domains, thereby forming several frameworks for DM-RL integration and highlighting their practical utility. Furthermore, we outline several categories of successful applications of diffusion-based RL across diverse domains, discuss open research issues of current methodologies, and highlight key directions for future research to advance the field. Finally, we summarize the survey to identify promising future development directions. We are actively maintaining a GitHub repository (https://github.com/ChangfuXu/D4RL-FTD) for papers and other related resources to apply DMs for RL.

Related papers

• A Survey of Reinforcement Learning for Large Reasoning Models [98.58081012669369]
  Review of recent advances in Reinforcement Learning for reasoning with Large Language Models.<n>Further scaling of RL for LRMs now faces challenges not only in computational resources but also in algorithm design, training data, and infrastructure.
  arXiv  Detail & Related papers  (2025-09-10T17:59:43Z)
• The Landscape of Agentic Reinforcement Learning for LLMs: A Survey [104.31926740841128]
  The emergence of agentic reinforcement learning (Agentic RL) marks a paradigm shift from conventional reinforcement learning applied to large language models (LLM RL)<n>This survey formalizes this conceptual shift by contrasting the degenerate single-step Markov Decision Processes (MDPs) of LLM-RL with the temporally extended, partially observable Markov decision processes (POMDPs) that define Agentic RL.
  arXiv  Detail & Related papers  (2025-09-02T17:46:26Z)
• Inverse Reinforcement Learning Meets Large Language Model Post-Training: Basics, Advances, and Opportunities [62.05713042908654]
  This paper provides a review of advances in Large Language Models (LLMs) alignment through the lens of inverse reinforcement learning (IRL)<n>We highlight the necessity of constructing neural reward models from human data and discuss the formal and practical implications of this paradigm shift.
  arXiv  Detail & Related papers  (2025-07-17T14:22:24Z)
• Anomaly Detection and Generation with Diffusion Models: A Survey [51.61574868316922]
  Anomaly detection (AD) plays a pivotal role across diverse domains, including cybersecurity, finance, healthcare, and industrial manufacturing.<n>Recent advancements in deep learning, specifically diffusion models (DMs), have sparked significant interest.<n>This survey aims to guide researchers and practitioners in leveraging DMs for innovative AD solutions across diverse applications.
  arXiv  Detail & Related papers  (2025-06-11T03:29:18Z)
• WeThink: Toward General-purpose Vision-Language Reasoning via Reinforcement Learning [17.459985667824807]
  Building on the success of text-based reasoning models like DeepSeek-R1, extending these capabilities to multimodal reasoning holds great promise.<n>In this paper, we show how to achieve the general-purpose visual-language reasoning through reinforcement learning.
  arXiv  Detail & Related papers  (2025-06-09T16:20:54Z)
• MMaDA: Multimodal Large Diffusion Language Models [61.13527224215318]
  We introduce MMaDA, a novel class of multimodal diffusion foundation models.<n>It is designed to achieve superior performance across diverse domains such as textual reasoning, multimodal understanding, and text-to-image generation.
  arXiv  Detail & Related papers  (2025-05-21T17:59:05Z)
• Reinforced MLLM: A Survey on RL-Based Reasoning in Multimodal Large Language Models [22.796496516709514]
  This paper provides a systematic review of recent advances in reinforcement learning (RL)-based reasoning for Multimodal Large Language Models (MLLMs)<n>We highlight two main RL paradigms, value-model-free and value-model-based methods, and analyze how RL enhances reasoning abilities by optimizing reasoning trajectories and aligning multimodal information.<n>We provide an extensive overview of benchmark datasets, evaluation protocols, and current limitations, and propose future research directions to address challenges such as sparse rewards, inefficient cross-modal reasoning, and real-world deployment constraints.
  arXiv  Detail & Related papers  (2025-04-30T03:14:28Z)
• Semantic Role Labeling: A Systematical Survey [43.51170121441664]
  Semantic role labeling (SRL) is a central natural language processing (NLP) task aiming to understand the semantic roles within texts.<n>There is currently a lack of a comprehensive survey that thoroughly organizes and synthesizes the field.<n>This paper aims to review the entire research trajectory of the SRL community over the past two decades.
  arXiv  Detail & Related papers  (2025-02-09T12:45:49Z)
• Generative AI for Deep Reinforcement Learning: Framework, Analysis, and Use Cases [60.30995339585003]
  Deep reinforcement learning (DRL) has been widely applied across various fields and has achieved remarkable accomplishments.<n>DRL faces certain limitations, including low sample efficiency and poor generalization.<n>We present how to leverage generative AI (GAI) to address these issues and enhance the performance of DRL algorithms.
  arXiv  Detail & Related papers  (2024-05-31T01:25:40Z)
• Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methods [18.771658054884693]
  Large language models (LLMs) emerge as a promising avenue to augment reinforcement learning (RL) in aspects such as multi-task learning, sample efficiency, and high-level task planning. We propose a structured taxonomy to systematically categorize LLMs' functionalities in RL, including four roles: information processor, reward designer, decision-maker, and generator.
  arXiv  Detail & Related papers  (2024-03-30T08:28:08Z)
• Diffusion Models for Reinforcement Learning: A Survey [22.670096541841325]
  Diffusion models surpass previous generative models in sample quality and training stability. Recent works have shown the advantages of diffusion models in improving reinforcement learning (RL) solutions. This survey aims to provide an overview of this emerging field and hopes to inspire new avenues of research.
  arXiv  Detail & Related papers  (2023-11-02T13:23:39Z)
• Ensemble Reinforcement Learning: A Survey [43.17635633600716]
  Reinforcement Learning (RL) has emerged as a highly effective technique for addressing various scientific and applied problems. In response, ensemble reinforcement learning (ERL), a promising approach that combines the benefits of both RL and ensemble learning (EL), has gained widespread popularity. ERL leverages multiple models or training algorithms to comprehensively explore the problem space and possesses strong generalization capabilities.
  arXiv  Detail & Related papers  (2023-03-05T09:26:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.