Where Will They Go? Modelling Multimodal Pedestrian Manoeuvres from Ego-centric Videos

• URL: http://arxiv.org/abs/2606.18824v1
• Date: Wed, 17 Jun 2026 08:55:01 GMT
• Title: Where Will They Go? Modelling Multimodal Pedestrian Manoeuvres from Ego-centric Videos
• Abstract summary: We propose MMPM, a mode-aware framework that models future trajectory distributions into semantically meaningful modes.<n> MMPM consists of two modules: behavior-aware Pedestrian Interaction Module (PIM) and CVAE-based Mode-aware Trajectory Predictor (MTP)<n> Experiments on PIE and JAAD datasets show that our method surpasses state-of-the-art baselines.
• Score: 27.591588122564648
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Pedestrian trajectory prediction from an ego-centric camera is challenging since it depends on complex interactions with vehicles and scene context, as well as the intention of the pedestrian. By modelling correlation and intent from the historical and future trajectories of the pedestrian, it will usually result in a multimodal (i.e. multiple modes) distribution. Existing stochastic predictors often sample multiple futures from a single unimodal distribution, which can yield sub-optimal 'mixed-mode' trajectories that lie between distinct motion patterns and become implausible in real scenes. In this paper, we propose MMPM, a mode-aware framework that separately models future trajectory distributions into semantically meaningful modes based on the pedestrian's crossing behavior. MMPM consists of two modules: behavior-aware Pedestrian Interaction Module (PIM) that jointly captures pedestrian-vehicle and pedestrian-environment interactions by introducing gaze, head and hand gesture, and a CVAE-based Mode-aware Trajectory Predictor (MTP) module to model the future trajectory distributions on two modes, crossing and non-crossing the road, separately. A query-based decoder further enforces mode consistency during decoding. Experiments on PIE and JAAD datasets show that our method surpasses state-of-the-art baselines. Our proposed MTP is model-agnostic, which can be integrated into existing frameworks such as BiTrap-NP and SGNet-ED to further improve future trajectory prediction performance. We additionally introduce a data-driven validation protocol that matches predictions to spatio-temporally consistent ground-truth trajectories, demonstrating improved frame-wise displacement errors over previous work.

Related papers

• Trajeglish: Traffic Modeling as Next-Token Prediction [67.28197954427638]
  A longstanding challenge for self-driving development is simulating dynamic driving scenarios seeded from recorded driving logs. We apply tools from discrete sequence modeling to model how vehicles, pedestrians and cyclists interact in driving scenarios. Our model tops the Sim Agents Benchmark, surpassing prior work along the realism meta metric by 3.3% and along the interaction metric by 9.9%.
  arXiv  Detail & Related papers  (2023-12-07T18:53:27Z)
• JFP: Joint Future Prediction with Interactive Multi-Agent Modeling for Autonomous Driving [12.460224193998362]
  We propose an end-to-end trainable model that learns directly the interaction between pairs of agents in a structured, graphical model formulation. Our approach improves significantly on the trajectory overlap metrics while obtaining on-par or better performance on single-agent trajectory metrics.
  arXiv  Detail & Related papers  (2022-12-16T20:59:21Z)
• STGlow: A Flow-based Generative Framework with Dual Graphormer for Pedestrian Trajectory Prediction [22.553356096143734]
  We propose a novel generative flow based framework with dual graphormer for pedestrian trajectory prediction (STGlow) Our method can more precisely model the underlying data distribution by optimizing the exact log-likelihood of motion behaviors. Experimental results on several benchmarks demonstrate that our method achieves much better performance compared to previous state-of-the-art approaches.
  arXiv  Detail & Related papers  (2022-11-21T07:29:24Z)
• PedFormer: Pedestrian Behavior Prediction via Cross-Modal Attention Modulation and Gated Multitask Learning [10.812772606528172]
  We propose a novel framework that relies on different data modalities to predict future trajectories and crossing actions of pedestrians from an ego-centric perspective. We show that our model improves state-of-the-art in trajectory and action prediction by up to 22% and 13% respectively on various metrics.
  arXiv  Detail & Related papers  (2022-10-14T15:12:00Z)
• Hierarchical Latent Structure for Multi-Modal Vehicle Trajectory Forecasting [0.0]
  We introduce a hierarchical latent structure into a VAE-based trajectory forecasting model. Our model is capable of generating clear multi-modal trajectory distributions and outperforms the state-of-the-art (SOTA) models in terms of prediction accuracy.
  arXiv  Detail & Related papers  (2022-07-11T04:52:28Z)
• Stochastic Trajectory Prediction via Motion Indeterminacy Diffusion [88.45326906116165]
  We present a new framework to formulate the trajectory prediction task as a reverse process of motion indeterminacy diffusion (MID) We encode the history behavior information and the social interactions as a state embedding and devise a Transformer-based diffusion model to capture the temporal dependencies of trajectories. Experiments on the human trajectory prediction benchmarks including the Stanford Drone and ETH/UCY datasets demonstrate the superiority of our method.
  arXiv  Detail & Related papers  (2022-03-25T16:59:08Z)
• You Mostly Walk Alone: Analyzing Feature Attribution in Trajectory Prediction [52.442129609979794]
  Recent deep learning approaches for trajectory prediction show promising performance. It remains unclear which features such black-box models actually learn to use for making predictions. This paper proposes a procedure that quantifies the contributions of different cues to model performance.
  arXiv  Detail & Related papers  (2021-10-11T14:24:15Z)
• SGCN:Sparse Graph Convolution Network for Pedestrian Trajectory Prediction [64.16212996247943]
  We present a Sparse Graph Convolution Network(SGCN) for pedestrian trajectory prediction. Specifically, the SGCN explicitly models the sparse directed interaction with a sparse directed spatial graph to capture adaptive interaction pedestrians. visualizations indicate that our method can capture adaptive interactions between pedestrians and their effective motion tendencies.
  arXiv  Detail & Related papers  (2021-04-04T03:17:42Z)
• Multimodal Trajectory Prediction via Topological Invariance for Navigation at Uncontrolled Intersections [45.508973373913946]
  We focus on decentralized navigation among multiple non-communicating rational agents at street intersections without traffic signs or signals. Our key insight is that the geometric structure of the intersection and the incentive of agents to move efficiently and avoid collisions (rationality) reduces the space of likely behaviors. We design Multiple Topologies Prediction (MTP), a data-driven trajectory-prediction mechanism that reconstructs trajectory representations of high-likelihood modes in multiagent intersection scenes.
  arXiv  Detail & Related papers  (2020-11-08T02:56:42Z)
• Haar Wavelet based Block Autoregressive Flows for Trajectories [129.37479472754083]
  Prediction of trajectories such as that of pedestrians is crucial to the performance of autonomous agents. We introduce a novel Haar wavelet based block autoregressive model leveraging split couplings. We illustrate the advantages of our approach for generating diverse and accurate trajectories on two real-world datasets.
  arXiv  Detail & Related papers  (2020-09-21T13:57:10Z)
• SMART: Simultaneous Multi-Agent Recurrent Trajectory Prediction [72.37440317774556]
  We propose advances that address two key challenges in future trajectory prediction. multimodality in both training data and predictions and constant time inference regardless of number of agents.
  arXiv  Detail & Related papers  (2020-07-26T08:17:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.