Poly-Autoregressive Prediction for Modeling Interactions

• URL: http://arxiv.org/abs/2502.08646v1
• Date: Wed, 12 Feb 2025 18:59:43 GMT
• Title: Poly-Autoregressive Prediction for Modeling Interactions
• Authors: Neerja Thakkar, Tara Sadjadpour, Jathushan Rajasegaran, Shiry Ginosar, Jitendra Malik,
• Abstract summary: We propose Poly-Autoregressive (PAR) modeling, which forecasts an ego agent's future behavior. We show that PAR can be applied to three different problems: human action forecasting in social situations, trajectory prediction for autonomous vehicles, and object pose forecasting during hand-object interaction.
• Score: 42.51313085280179
• License:
• Abstract: We introduce a simple framework for predicting the behavior of an agent in multi-agent settings. In contrast to autoregressive (AR) tasks, such as language processing, our focus is on scenarios with multiple agents whose interactions are shaped by physical constraints and internal motivations. To this end, we propose Poly-Autoregressive (PAR) modeling, which forecasts an ego agent's future behavior by reasoning about the ego agent's state history and the past and current states of other interacting agents. At its core, PAR represents the behavior of all agents as a sequence of tokens, each representing an agent's state at a specific timestep. With minimal data pre-processing changes, we show that PAR can be applied to three different problems: human action forecasting in social situations, trajectory prediction for autonomous vehicles, and object pose forecasting during hand-object interaction. Using a small proof-of-concept transformer backbone, PAR outperforms AR across these three scenarios. The project website can be found at https://neerja.me/PAR/.

Related papers

• Generative Agents: Interactive Simulacra of Human Behavior [86.1026716646289]
  We introduce generative agents--computational software agents that simulate believable human behavior. We describe an architecture that extends a large language model to store a complete record of the agent's experiences. We instantiate generative agents to populate an interactive sandbox environment inspired by The Sims.
  arXiv  Detail & Related papers  (2023-04-07T01:55:19Z)
• Deep Interactive Motion Prediction and Planning: Playing Games with Motion Prediction Models [162.21629604674388]
  This work presents a game-theoretic Model Predictive Controller (MPC) that uses a novel interactive multi-agent neural network policy as part of its predictive model. Fundamental to the success of our method is the design of a novel multi-agent policy network that can steer a vehicle given the state of the surrounding agents and the map information.
  arXiv  Detail & Related papers  (2022-04-05T17:58:18Z)
• Exploring Social Posterior Collapse in Variational Autoencoder for Interaction Modeling [26.01824780050843]
  Variational Autoencoder (VAE) has been widely applied in multi-agent interaction modeling. VAE is prone to ignoring historical social context when predicting the future trajectory of an agent. We propose a novel sparse graph attention message-passing layer, which helps us detect social posterior collapse.
  arXiv  Detail & Related papers  (2021-12-01T06:20:58Z)
• You Mostly Walk Alone: Analyzing Feature Attribution in Trajectory Prediction [52.442129609979794]
  Recent deep learning approaches for trajectory prediction show promising performance. It remains unclear which features such black-box models actually learn to use for making predictions. This paper proposes a procedure that quantifies the contributions of different cues to model performance.
  arXiv  Detail & Related papers  (2021-10-11T14:24:15Z)
• Scene Transformer: A unified multi-task model for behavior prediction and planning [42.758178896204036]
  We formulate a model for predicting the behavior of all agents jointly in real-world driving environments. Inspired by recent language modeling approaches, we use a masking strategy as the query to our model. We evaluate our approach on autonomous driving datasets for behavior prediction, and achieve state-of-the-art performance.
  arXiv  Detail & Related papers  (2021-06-15T20:20:44Z)
• Time-series Imputation of Temporally-occluded Multiagent Trajectories [18.862173210927658]
  We study the problem of multiagent time-series imputation, where available past and future observations of subsets of agents are used to estimate missing observations for other agents. Our approach, called the Graph Imputer, uses forward- and backward-information in combination with graph networks and variational autoencoders. We evaluate our approach on a dataset of football matches, using a projective camera module to train and evaluate our model for the off-screen player state estimation setting.
  arXiv  Detail & Related papers  (2021-06-08T09:58:43Z)
• AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting [25.151713845738335]
  We propose a new Transformer, AgentFormer, that jointly models the time and social dimensions. Based on AgentFormer, we propose a multi-agent trajectory prediction model that can attend to features of any agent at any previous timestep. Our method significantly improves the state of the art on well-established pedestrian and autonomous driving datasets.
  arXiv  Detail & Related papers  (2021-03-25T17:59:01Z)
• Instance-Aware Predictive Navigation in Multi-Agent Environments [93.15055834395304]
  We propose an Instance-Aware Predictive Control (IPC) approach, which forecasts interactions between agents as well as future scene structures. We adopt a novel multi-instance event prediction module to estimate the possible interaction among agents in the ego-centric view. We design a sequential action sampling strategy to better leverage predicted states on both scene-level and instance-level.
  arXiv  Detail & Related papers  (2021-01-14T22:21:25Z)
• End-to-end Contextual Perception and Prediction with Interaction Transformer [79.14001602890417]
  We tackle the problem of detecting objects in 3D and forecasting their future motion in the context of self-driving. To capture their spatial-temporal dependencies, we propose a recurrent neural network with a novel Transformer architecture. Our model can be trained end-to-end, and runs in real-time.
  arXiv  Detail & Related papers  (2020-08-13T14:30:12Z)
• Robust Trajectory Forecasting for Multiple Intelligent Agents in Dynamic Scene [11.91073327154494]
  We present a novel method for robust trajectory forecasting of multiple agents in dynamic scenes. The proposed method outperforms the state-of-the-art prediction methods in terms of prediction accuracy.
  arXiv  Detail & Related papers  (2020-05-27T02:32:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.