Learning Collective Dynamics of Multi-Agent Systems using Event-based Vision

• URL: http://arxiv.org/abs/2411.07039v1
• Date: Mon, 11 Nov 2024 14:45:47 GMT
• Title: Learning Collective Dynamics of Multi-Agent Systems using Event-based Vision
• Authors: Minah Lee, Uday Kamal, Saibal Mukhopadhyay,
• Abstract summary: This paper proposes a novel problem: vision-based perception to learn and predict the collective dynamics of multi-agent systems. We focus on deep learning models to directly predict collective dynamics from visual data, captured as frames or events. We empirically demonstrate the effectiveness of event-based representation over traditional frame-based methods in predicting these collective behaviors.
• Score: 15.26086907502649
• License:
• Abstract: This paper proposes a novel problem: vision-based perception to learn and predict the collective dynamics of multi-agent systems, specifically focusing on interaction strength and convergence time. Multi-agent systems are defined as collections of more than ten interacting agents that exhibit complex group behaviors. Unlike prior studies that assume knowledge of agent positions, we focus on deep learning models to directly predict collective dynamics from visual data, captured as frames or events. Due to the lack of relevant datasets, we create a simulated dataset using a state-of-the-art flocking simulator, coupled with a vision-to-event conversion framework. We empirically demonstrate the effectiveness of event-based representation over traditional frame-based methods in predicting these collective behaviors. Based on our analysis, we present event-based vision for Multi-Agent dynamic Prediction (evMAP), a deep learning architecture designed for real-time, accurate understanding of interaction strength and collective behavior emergence in multi-agent systems.

Related papers

• Learning System Dynamics without Forgetting [60.08612207170659]
  Predicting trajectories of systems with unknown dynamics is crucial in various research fields, including physics and biology. We present a novel framework of Mode-switching Graph ODE (MS-GODE), which can continually learn varying dynamics. We construct a novel benchmark of biological dynamic systems, featuring diverse systems with disparate dynamics.
  arXiv  Detail & Related papers  (2024-06-30T14:55:18Z)
• Neural Clustering based Visual Representation Learning [61.72646814537163]
  Clustering is one of the most classic approaches in machine learning and data analysis. We propose feature extraction with clustering (FEC), which views feature extraction as a process of selecting representatives from data. FEC alternates between grouping pixels into individual clusters to abstract representatives and updating the deep features of pixels with current representatives.
  arXiv  Detail & Related papers  (2024-03-26T06:04:50Z)
• A VAE-based Framework for Learning Multi-Level Neural Granger-Causal Connectivity [15.295157876811066]
  This paper introduces a Variational Autoencoder based framework that jointly learns Granger-causal relationships amongst components in a collection of related-yet-heterogeneous dynamical systems. The performance of the proposed framework is evaluated on several synthetic data settings and benchmarked against existing approaches designed for individual system learning.
  arXiv  Detail & Related papers  (2024-02-25T16:11:32Z)
• ALP: Action-Aware Embodied Learning for Perception [60.64801970249279]
  We introduce Action-Aware Embodied Learning for Perception (ALP) ALP incorporates action information into representation learning through a combination of optimizing a reinforcement learning policy and an inverse dynamics prediction objective. We show that ALP outperforms existing baselines in several downstream perception tasks.
  arXiv  Detail & Related papers  (2023-06-16T21:51:04Z)
• Predictive Experience Replay for Continual Visual Control and Forecasting [62.06183102362871]
  We present a new continual learning approach for visual dynamics modeling and explore its efficacy in visual control and forecasting. We first propose the mixture world model that learns task-specific dynamics priors with a mixture of Gaussians, and then introduce a new training strategy to overcome catastrophic forgetting. Our model remarkably outperforms the naive combinations of existing continual learning and visual RL algorithms on DeepMind Control and Meta-World benchmarks with continual visual control tasks.
  arXiv  Detail & Related papers  (2023-03-12T05:08:03Z)
• Rethinking Trajectory Prediction via "Team Game" [118.59480535826094]
  We present a novel formulation for multi-agent trajectory prediction, which explicitly introduces the concept of interactive group consensus. On two multi-agent settings, i.e. team sports and pedestrians, the proposed framework consistently achieves superior performance compared to existing methods.
  arXiv  Detail & Related papers  (2022-10-17T07:16:44Z)
• Multi-Agent Imitation Learning with Copulas [102.27052968901894]
  Multi-agent imitation learning aims to train multiple agents to perform tasks from demonstrations by learning a mapping between observations and actions. In this paper, we propose to use copula, a powerful statistical tool for capturing dependence among random variables, to explicitly model the correlation and coordination in multi-agent systems. Our proposed model is able to separately learn marginals that capture the local behavioral patterns of each individual agent, as well as a copula function that solely and fully captures the dependence structure among agents.
  arXiv  Detail & Related papers  (2021-07-10T03:49:41Z)
• GATSBI: Generative Agent-centric Spatio-temporal Object Interaction [9.328991021103294]
  GAT SBI is a generative model that transforms a sequence of raw observations into a structured representation. We show GAT SBI achieves superior on scene decomposition and video prediction compared to its state-of-the-art counterparts.
  arXiv  Detail & Related papers  (2021-04-09T09:45:00Z)
• Deep learning reveals hidden interactions in complex systems [0.0]
  AgentNet is a model-free data-driven framework consisting of deep neural networks to reveal hidden interactions in complex systems. A demonstration with empirical data from a flock of birds showed that AgentNet could identify hidden interaction ranges exhibited by real birds.
  arXiv  Detail & Related papers  (2020-01-03T02:25:50Z)
• Improved Structural Discovery and Representation Learning of Multi-Agent Data [5.40729975786985]
  We present a dynamic alignment method which provides a robust ordering of structured multi-agent data. We demonstrate the value of this approach using a large amount of soccer tracking data from a professional league.
  arXiv  Detail & Related papers  (2019-12-30T22:49:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.