DiffFP: Learning Behaviors from Scratch via Diffusion-based Fictitious Play

• URL: http://arxiv.org/abs/2511.13186v1
• Date: Mon, 17 Nov 2025 09:48:29 GMT
• Title: DiffFP: Learning Behaviors from Scratch via Diffusion-based Fictitious Play
• Authors: Akash Karthikeyan, Yash Vardhan Pant,
• Abstract summary: We propose DiffFP, a fictitious play framework that estimates the best response to unseen opponents in zero-sum games.<n>We validate our method on complex multi-agent environments, including racing and multi-particle zero-sum games.<n>Our approach achieves up to 3$times$ faster convergence and 30$times$ higher success rates on average against RL-based baselines.
• Score: 5.8808473430456525
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Self-play reinforcement learning has demonstrated significant success in learning complex strategic and interactive behaviors in competitive multi-agent games. However, achieving such behaviors in continuous decision spaces remains challenging. Ensuring adaptability and generalization in self-play settings is critical for achieving competitive performance in dynamic multi-agent environments. These challenges often cause methods to converge slowly or fail to converge at all to a Nash equilibrium, making agents vulnerable to strategic exploitation by unseen opponents. To address these challenges, we propose DiffFP, a fictitious play (FP) framework that estimates the best response to unseen opponents while learning a robust and multimodal behavioral policy. Specifically, we approximate the best response using a diffusion policy that leverages generative modeling to learn adaptive and diverse strategies. Through empirical evaluation, we demonstrate that the proposed FP framework converges towards $ε$-Nash equilibria in continuous- space zero-sum games. We validate our method on complex multi-agent environments, including racing and multi-particle zero-sum games. Simulation results show that the learned policies are robust against diverse opponents and outperform baseline reinforcement learning policies. Our approach achieves up to 3$\times$ faster convergence and 30$\times$ higher success rates on average against RL-based baselines, demonstrating its robustness to opponent strategies and stability across training iterations

Related papers

• Reinforcement Learning with Discrete Diffusion Policies for Combinatorial Action Spaces [57.466101098183884]
  Reinforcement learning (RL) struggles to scale to large, action spaces common in many real-world problems.<n>This paper introduces a novel framework for training discrete diffusion models as highly effective policies in complex settings.
  arXiv  Detail & Related papers  (2025-09-26T21:53:36Z)
• Curriculum Learning With Counterfactual Group Relative Policy Advantage For Multi-Agent Reinforcement Learning [15.539607264374242]
  Multi-agent reinforcement learning (MARL) has achieved strong performance in cooperative adversarial tasks.<n>We propose a dynamic curriculum learning framework that employs an self-adaptive difficulty adjustment mechanism.<n>Our method improves both training stability and final performance, achieving competitive results against state-of-the-art methods.
  arXiv  Detail & Related papers  (2025-06-09T08:38:18Z)
• Ranking Joint Policies in Dynamic Games using Evolutionary Dynamics [0.0]
  It has been shown that the dynamics of agents' interactions, even in simple two-player games, are incapable of reaching Nash equilibria.<n>Our goal is to identify agents' joint strategies that result in stable behavior, being resistant to changes, while also accounting for agents' payoffs.
  arXiv  Detail & Related papers  (2025-02-20T16:50:38Z)
• Mimicking To Dominate: Imitation Learning Strategies for Success in Multiagent Competitive Games [13.060023718506917]
  We develop a new multi-agent imitation learning model for predicting next moves of the opponents. We also present a new multi-agent reinforcement learning algorithm that combines our imitation learning model and policy training into one single training process. Experimental results show that our approach achieves superior performance compared to existing state-of-the-art multi-agent RL algorithms.
  arXiv  Detail & Related papers  (2023-08-20T07:30:13Z)
• Learning Diverse Risk Preferences in Population-based Self-play [23.07952140353786]
  Current self-play algorithms optimize the agent to maximize expected win-rates against its current or historical copies. We introduce diversity from the perspective that agents could have diverse risk preferences in the face of uncertainty. We show that our method achieves comparable or superior performance in competitive games.
  arXiv  Detail & Related papers  (2023-05-19T06:56:02Z)
• Finding mixed-strategy equilibria of continuous-action games without gradients using randomized policy networks [83.28949556413717]
  We study the problem of computing an approximate Nash equilibrium of continuous-action game without access to gradients. We model players' strategies using artificial neural networks. This paper is the first to solve general continuous-action games with unrestricted mixed strategies and without any gradient information.
  arXiv  Detail & Related papers  (2022-11-29T05:16:41Z)
• Nash Equilibria and Pitfalls of Adversarial Training in Adversarial Robustness Games [51.90475640044073]
  We study adversarial training as an alternating best-response strategy in a 2-player zero-sum game. On the other hand, a unique pure Nash equilibrium of the game exists and is provably robust.
  arXiv  Detail & Related papers  (2022-10-23T03:21:01Z)
• Conditional Imitation Learning for Multi-Agent Games [89.897635970366]
  We study the problem of conditional multi-agent imitation learning, where we have access to joint trajectory demonstrations at training time. We propose a novel approach to address the difficulties of scalability and data scarcity. Our model learns a low-rank subspace over ego and partner agent strategies, then infers and adapts to a new partner strategy by interpolating in the subspace.
  arXiv  Detail & Related papers  (2022-01-05T04:40:13Z)
• Disturbing Reinforcement Learning Agents with Corrupted Rewards [62.997667081978825]
  We analyze the effects of different attack strategies based on reward perturbations on reinforcement learning algorithms. We show that smoothly crafting adversarial rewards are able to mislead the learner, and that using low exploration probability values, the policy learned is more robust to corrupt rewards.
  arXiv  Detail & Related papers  (2021-02-12T15:53:48Z)
• Opponent Learning Awareness and Modelling in Multi-Objective Normal Form Games [5.0238343960165155]
  It is essential for an agent to learn about the behaviour of other agents in the system. We present the first study of the effects of such opponent modelling on multi-objective multi-agent interactions with non-linear utilities.
  arXiv  Detail & Related papers  (2020-11-14T12:35:32Z)
• Robust Reinforcement Learning using Adversarial Populations [118.73193330231163]
  Reinforcement Learning (RL) is an effective tool for controller design but can struggle with issues of robustness. We show that using a single adversary does not consistently yield robustness to dynamics variations under standard parametrizations of the adversary. We propose a population-based augmentation to the Robust RL formulation in which we randomly initialize a population of adversaries and sample from the population uniformly during training.
  arXiv  Detail & Related papers  (2020-08-04T20:57:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.