Long-term Human Motion Prediction with Scene Context

• URL: http://arxiv.org/abs/2007.03672v3
• Date: Fri, 31 Jul 2020 17:23:11 GMT
• Title: Long-term Human Motion Prediction with Scene Context
• Authors: Zhe Cao, Hang Gao, Karttikeya Mangalam, Qi-Zhi Cai, Minh Vo, Jitendra Malik
• Abstract summary: We propose a novel three-stage framework for predicting human motion. Our method first samples multiple human motion goals, then plans 3D human paths towards each goal, and finally predicts 3D human pose sequences following each path.
• Score: 60.096118270451974
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Human movement is goal-directed and influenced by the spatial layout of the objects in the scene. To plan future human motion, it is crucial to perceive the environment -- imagine how hard it is to navigate a new room with lights off. Existing works on predicting human motion do not pay attention to the scene context and thus struggle in long-term prediction. In this work, we propose a novel three-stage framework that exploits scene context to tackle this task. Given a single scene image and 2D pose histories, our method first samples multiple human motion goals, then plans 3D human paths towards each goal, and finally predicts 3D human pose sequences following each path. For stable training and rigorous evaluation, we contribute a diverse synthetic dataset with clean annotations. In both synthetic and real datasets, our method shows consistent quantitative and qualitative improvements over existing methods.

Related papers

• Multimodal Sense-Informed Prediction of 3D Human Motions [16.71099574742631]
  This work introduces a novel multi-modal sense-informed motion prediction approach, which conditions high-fidelity generation on two modal information. The gaze information is regarded as the human intention, and combined with both motion and scene features, we construct a ternary intention-aware attention to supervise the generation. On two real-world benchmarks, the proposed method achieves state-of-the-art performance both in 3D human pose and trajectory prediction.
  arXiv  Detail & Related papers  (2024-05-05T12:38:10Z)
• Scene-aware Human Motion Forecasting via Mutual Distance Prediction [13.067687949642641]
  We propose to model the human-scene interaction with the mutual distance between the human body and the scene. Such mutual distances constrain both the local and global human motion, resulting in a whole-body motion constrained prediction. We develop a pipeline with two sequential steps: predicting the future mutual distances first, followed by forecasting future human motion.
  arXiv  Detail & Related papers  (2023-10-01T08:32:46Z)
• Staged Contact-Aware Global Human Motion Forecasting [7.930326095134298]
  Scene-aware global human motion forecasting is critical for manifold applications, including virtual reality, robotics, and sports. We propose a STAGed contact-aware global human motion forecasting STAG, a novel three-stage pipeline for predicting global human motion in a 3D environment. STAG achieves a 1.8% and 16.2% overall improvement in pose and trajectory prediction, respectively, on the scene-aware GTA-IM dataset.
  arXiv  Detail & Related papers  (2023-09-16T10:47:48Z)
• Task-Oriented Human-Object Interactions Generation with Implicit Neural Representations [61.659439423703155]
  TOHO: Task-Oriented Human-Object Interactions Generation with Implicit Neural Representations. Our method generates continuous motions that are parameterized only by the temporal coordinate. This work takes a step further toward general human-scene interaction simulation.
  arXiv  Detail & Related papers  (2023-03-23T09:31:56Z)
• HUMANISE: Language-conditioned Human Motion Generation in 3D Scenes [54.61610144668777]
  We present a novel scene-and-language conditioned generative model that can produce 3D human motions in 3D scenes. Our experiments demonstrate that our model generates diverse and semantically consistent human motions in 3D scenes.
  arXiv  Detail & Related papers  (2022-10-18T10:14:11Z)
• Contact-aware Human Motion Forecasting [87.04827994793823]
  We tackle the task of scene-aware 3D human motion forecasting, which consists of predicting future human poses given a 3D scene and a past human motion. Our approach outperforms the state-of-the-art human motion forecasting and human synthesis methods on both synthetic and real datasets.
  arXiv  Detail & Related papers  (2022-10-08T07:53:19Z)
• Towards Diverse and Natural Scene-aware 3D Human Motion Synthesis [117.15586710830489]
  We focus on the problem of synthesizing diverse scene-aware human motions under the guidance of target action sequences. Based on this factorized scheme, a hierarchical framework is proposed, with each sub-module responsible for modeling one aspect. Experiment results show that the proposed framework remarkably outperforms previous methods in terms of diversity and naturalness.
  arXiv  Detail & Related papers  (2022-05-25T18:20:01Z)
• Scene-aware Generative Network for Human Motion Synthesis [125.21079898942347]
  We propose a new framework, with the interaction between the scene and the human motion taken into account. Considering the uncertainty of human motion, we formulate this task as a generative task. We derive a GAN based learning approach, with discriminators to enforce the compatibility between the human motion and the contextual scene.
  arXiv  Detail & Related papers  (2021-05-31T09:05:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.