Learning to Simulate Self-Driven Particles System with Coordinated Policy Optimization

• URL: http://arxiv.org/abs/2110.13827v1
• Date: Tue, 26 Oct 2021 16:20:23 GMT
• Title: Learning to Simulate Self-Driven Particles System with Coordinated Policy Optimization
• Abstract summary: Self-Driven Particles (SDP) describe a category of multi-agent systems common in everyday life, such as flocking birds and traffic flows. Previous multi-agent reinforcement learning (MARL) methods define the agents to be teammates or enemies before hand. In this work we develop a novel MARL method called Coordinated Policy Optimization (CoPO), which incorporates social psychology principle to learn neural controller for SDP.
• Score: 36.113662645486755
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Self-Driven Particles (SDP) describe a category of multi-agent systems common in everyday life, such as flocking birds and traffic flows. In a SDP system, each agent pursues its own goal and constantly changes its cooperative or competitive behaviors with its nearby agents. Manually designing the controllers for such SDP system is time-consuming, while the resulting emergent behaviors are often not realistic nor generalizable. Thus the realistic simulation of SDP systems remains challenging. Reinforcement learning provides an appealing alternative for automating the development of the controller for SDP. However, previous multi-agent reinforcement learning (MARL) methods define the agents to be teammates or enemies before hand, which fail to capture the essence of SDP where the role of each agent varies to be cooperative or competitive even within one episode. To simulate SDP with MARL, a key challenge is to coordinate agents' behaviors while still maximizing individual objectives. Taking traffic simulation as the testing bed, in this work we develop a novel MARL method called Coordinated Policy Optimization (CoPO), which incorporates social psychology principle to learn neural controller for SDP. Experiments show that the proposed method can achieve superior performance compared to MARL baselines in various metrics. Noticeably the trained vehicles exhibit complex and diverse social behaviors that improve performance and safety of the population as a whole. Demo video and source code are available at: https://decisionforce.github.io/CoPO/

Related papers

• Beyond Cooperative Simulators: Generating Realistic User Personas for Robust Evaluation of LLM Agents [28.85410475505536]
  Persona Policies (PPol) is a control layer that induces realistic behavioral variation in user simulators.<n>PPol produces a diverse population of human-like personas for any task in the domain.<n>Agents trained with PPol are more robust to challenging, out-of-distribution behaviors, improving task success by +17% relative to training only on existing simulated interactions.
  arXiv  Detail & Related papers  (2026-05-13T02:16:51Z)
• CaP-X: A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation [48.85772216740915]
  "Code-as-Policy" considers how executable code can complement data-intensive Vision-Language-Action methods.<n>We present CaP-X, an open-access framework for studying Code-as-Policy agents in robot manipulation.
  arXiv  Detail & Related papers  (2026-03-23T18:08:10Z)
• REFINE-DP: Diffusion Policy Fine-tuning for Humanoid Loco-manipulation via Reinforcement Learning [23.094280494500214]
  We present REFINE-DP (REinforcement learning FINE-tuning of Diffusion Policy), a hierarchical framework that jointly optimize a DP high-level planner and an RL-based low-level loco-manipulation controller.<n> REFINE-DP achieves an over $90%$ success rate in simulation, even in out-of-distribution cases not seen in the pre-trained data.
  arXiv  Detail & Related papers  (2026-03-14T02:21:19Z)
• SPACeR: Self-Play Anchoring with Centralized Reference Models [50.55045557371374]
  Sim agent policies are realistic, human-like, fast, and scalable in multi-agent settings.<n>Recent progress in imitation learning with large diffusion-based or tokenized models has shown that behaviors can be captured directly from human driving data.<n>We propose SPACeR, a framework that leverages a pretrained tokenized autoregressive motion model as a central reference policy.
  arXiv  Detail & Related papers  (2025-10-20T19:53:02Z)
• Large Population Models [5.935007288459162]
  Large Population Models simulate entire populations with realistic behaviors and interactions at unprecedented scale.<n>This allows researchers to observe how agent behavior aggregates into system-level outcomes and test interventions before real-world implementation.<n>LPMs offer a complementary path in AI research illuminating collective intelligence and providing testing grounds for policies and social innovations before real-world deployment.
  arXiv  Detail & Related papers  (2025-07-14T04:11:54Z)
• SDPO: Segment-Level Direct Preference Optimization for Social Agents [56.970902914217156]
  Social agents powered by large language models (LLMs) can simulate human social behaviors but fall short in handling complex social dialogues. We propose Segment-Level Direct Preference Optimization (SDPO), which dynamically select key segments within interactions to optimize multi-turn agent behavior.
  arXiv  Detail & Related papers  (2025-01-03T14:09:46Z)
• Autonomous Vehicle Controllers From End-to-End Differentiable Simulation [60.05963742334746]
  We propose a differentiable simulator and design an analytic policy gradients (APG) approach to training AV controllers. Our proposed framework brings the differentiable simulator into an end-to-end training loop, where gradients of environment dynamics serve as a useful prior to help the agent learn a more grounded policy. We find significant improvements in performance and robustness to noise in the dynamics, as well as overall more intuitive human-like handling.
  arXiv  Detail & Related papers  (2024-09-12T11:50:06Z)
• Efficient Adaptation in Mixed-Motive Environments via Hierarchical Opponent Modeling and Planning [51.52387511006586]
  We propose Hierarchical Opponent modeling and Planning (HOP), a novel multi-agent decision-making algorithm. HOP is hierarchically composed of two modules: an opponent modeling module that infers others' goals and learns corresponding goal-conditioned policies. HOP exhibits superior few-shot adaptation capabilities when interacting with various unseen agents, and excels in self-play scenarios.
  arXiv  Detail & Related papers  (2024-06-12T08:48:06Z)
• PPS-QMIX: Periodically Parameter Sharing for Accelerating Convergence of Multi-Agent Reinforcement Learning [20.746383793882984]
  Training for multi-agent reinforcement learning(MARL) is a time-consuming process. One drawback is that strategy of each agent in MARL is independent but actually in cooperation. We propose three simple approaches called Average Sharing(A-PPS), Reward-Scalability Periodically and Partial Personalized Periodically.
  arXiv  Detail & Related papers  (2024-03-05T03:59:01Z)
• RPM: Generalizable Behaviors for Multi-Agent Reinforcement Learning [90.43925357575543]
  We propose ranked policy memory ( RPM) to collect diverse multi-agent trajectories for training MARL policies with good generalizability. RPM enables MARL agents to interact with unseen agents in multi-agent generalization evaluation scenarios and complete given tasks, and it significantly boosts the performance up to 402% on average.
  arXiv  Detail & Related papers  (2022-10-18T07:32:43Z)
• Centralized Model and Exploration Policy for Multi-Agent RL [13.661446184763117]
  Reinforcement learning in partially observable, fully cooperative multi-agent settings (Dec-POMDPs) can be used to address many real-world challenges. Current RL algorithms for Dec-POMDPs suffer from poor sample complexity. We propose a model-based algorithm, MARCO, in three cooperative communication tasks, where it improves sample efficiency by up to 20x.
  arXiv  Detail & Related papers  (2021-07-14T00:34:08Z)
• Where to go next: Learning a Subgoal Recommendation Policy for Navigation Among Pedestrians [40.58684597726312]
  Local trajectory optimization methods, such as model predictive control (MPC), can deal with those changes but require global guidance. This paper proposes to learn, via deep Reinforcement Learning (RL), an interaction-aware policy that provides long-term guidance to the local planner.
  arXiv  Detail & Related papers  (2021-02-25T18:41:58Z)
• TrafficSim: Learning to Simulate Realistic Multi-Agent Behaviors [74.67698916175614]
  We propose TrafficSim, a multi-agent behavior model for realistic traffic simulation. In particular, we leverage an implicit latent variable model to parameterize a joint actor policy. We show TrafficSim generates significantly more realistic and diverse traffic scenarios as compared to a diverse set of baselines.
  arXiv  Detail & Related papers  (2021-01-17T00:29:30Z)
• Model-based Reinforcement Learning for Decentralized Multiagent Rendezvous [66.6895109554163]
  Underlying the human ability to align goals with other agents is their ability to predict the intentions of others and actively update their own plans. We propose hierarchical predictive planning (HPP), a model-based reinforcement learning method for decentralized multiagent rendezvous.
  arXiv  Detail & Related papers  (2020-03-15T19:49:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.