Multi-Transmotion: Pre-trained Model for Human Motion Prediction

• URL: http://arxiv.org/abs/2411.02673v1
• Date: Mon, 04 Nov 2024 23:15:21 GMT
• Title: Multi-Transmotion: Pre-trained Model for Human Motion Prediction
• Authors: Yang Gao, Po-Chien Luan, Alexandre Alahi,
• Abstract summary: Multi-Transmotion is an innovative transformer-based model designed for cross-modality pre-training. Our methodology demonstrates competitive performance across various datasets on several downstream tasks.
• Score: 68.87010221355223
• License:
• Abstract: The ability of intelligent systems to predict human behaviors is crucial, particularly in fields such as autonomous vehicle navigation and social robotics. However, the complexity of human motion have prevented the development of a standardized dataset for human motion prediction, thereby hindering the establishment of pre-trained models. In this paper, we address these limitations by integrating multiple datasets, encompassing both trajectory and 3D pose keypoints, to propose a pre-trained model for human motion prediction. We merge seven distinct datasets across varying modalities and standardize their formats. To facilitate multimodal pre-training, we introduce Multi-Transmotion, an innovative transformer-based model designed for cross-modality pre-training. Additionally, we present a novel masking strategy to capture rich representations. Our methodology demonstrates competitive performance across various datasets on several downstream tasks, including trajectory prediction in the NBA and JTA datasets, as well as pose prediction in the AMASS and 3DPW datasets. The code is publicly available: https://github.com/vita-epfl/multi-transmotion

Related papers

• SmartPretrain: Model-Agnostic and Dataset-Agnostic Representation Learning for Motion Prediction [37.461695201579914]
  We propose SmartPretrain, a general and scalable framework for motion prediction. Our approach integrates contrastive and reconstructive SSL, leveraging the strengths of both generative and discriminative paradigms. SmartPretrain consistently improves the performance of state-of-the-art prediction models across datasets, data splits and main metrics.
  arXiv  Detail & Related papers  (2024-10-11T09:52:26Z)
• Humanoid Locomotion as Next Token Prediction [84.21335675130021]
  Our model is a causal transformer trained via autoregressive prediction of sensorimotor trajectories. We show that our model enables a full-sized humanoid to walk in San Francisco zero-shot. Our model can transfer to the real world even when trained on only 27 hours of walking data, and can generalize commands not seen during training like walking backward.
  arXiv  Detail & Related papers  (2024-02-29T18:57:37Z)
• Social-Transmotion: Promptable Human Trajectory Prediction [65.80068316170613]
  Social-Transmotion is a generic Transformer-based model that exploits diverse and numerous visual cues to predict human behavior. Our approach is validated on multiple datasets, including JTA, JRDB, Pedestrians and Cyclists in Road Traffic, and ETH-UCY.
  arXiv  Detail & Related papers  (2023-12-26T18:56:49Z)
• The MI-Motion Dataset and Benchmark for 3D Multi-Person Motion Prediction [13.177817435234449]
  3D multi-person motion prediction is a challenging task that involves modeling individual behaviors and interactions between people. We introduce the Multi-Person Interaction Motion (MI-Motion) dataset, which includes skeleton sequences of multiple individuals collected by motion capture systems. The dataset contains 167k frames of interacting people's skeleton poses and is categorized into 5 different activity scenes.
  arXiv  Detail & Related papers  (2023-06-23T15:38:22Z)
• Conditioned Human Trajectory Prediction using Iterative Attention Blocks [70.36888514074022]
  We present a simple yet effective pedestrian trajectory prediction model aimed at pedestrians positions prediction in urban-like environments. Our model is a neural-based architecture that can run several layers of attention blocks and transformers in an iterative sequential fashion. We show that without explicit introduction of social masks, dynamical models, social pooling layers, or complicated graph-like structures, it is possible to produce on par results with SoTA models.
  arXiv  Detail & Related papers  (2022-06-29T07:49:48Z)
• TRiPOD: Human Trajectory and Pose Dynamics Forecasting in the Wild [77.59069361196404]
  TRiPOD is a novel method for predicting body dynamics based on graph attentional networks. To incorporate a real-world challenge, we learn an indicator representing whether an estimated body joint is visible/invisible at each frame. Our evaluation shows that TRiPOD outperforms all prior work and state-of-the-art specifically designed for each of the trajectory and pose forecasting tasks.
  arXiv  Detail & Related papers  (2021-04-08T20:01:00Z)
• SMART: Simultaneous Multi-Agent Recurrent Trajectory Prediction [72.37440317774556]
  We propose advances that address two key challenges in future trajectory prediction. multimodality in both training data and predictions and constant time inference regardless of number of agents.
  arXiv  Detail & Related papers  (2020-07-26T08:17:10Z)
• AC-VRNN: Attentive Conditional-VRNN for Multi-Future Trajectory Prediction [30.61190086847564]
  We propose a generative architecture for multi-future trajectory predictions based on Conditional Variational Recurrent Neural Networks (C-VRNNs) Human interactions are modeled with a graph-based attention mechanism enabling an online attentive hidden state refinement of the recurrent estimation.
  arXiv  Detail & Related papers  (2020-05-17T17:21:23Z)
• Trajectron++: Dynamically-Feasible Trajectory Forecasting With Heterogeneous Data [37.176411554794214]
  Reasoning about human motion is an important prerequisite to safe and socially-aware robotic navigation. We present Trajectron++, a modular, graph-structured recurrent model that forecasts the trajectories of a general number of diverse agents. We demonstrate its performance on several challenging real-world trajectory forecasting datasets.
  arXiv  Detail & Related papers  (2020-01-09T16:47:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.