Expressive Forecasting of 3D Whole-body Human Motions

• URL: http://arxiv.org/abs/2312.11972v2
• Date: Thu, 4 Apr 2024 16:41:22 GMT
• Title: Expressive Forecasting of 3D Whole-body Human Motions
• Authors: Pengxiang Ding, Qiongjie Cui, Min Zhang, Mengyuan Liu, Haofan Wang, Donglin Wang,
• Abstract summary: We are the first to formulate a whole-body human pose forecasting framework. Our model involves two key constituents: cross-context alignment (XCA) and cross-context interaction (XCI) We conduct extensive experiments on a newly-introduced large-scale benchmark and achieve state-of-theart performance.
• Score: 38.93700642077312
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Human motion forecasting, with the goal of estimating future human behavior over a period of time, is a fundamental task in many real-world applications. However, existing works typically concentrate on predicting the major joints of the human body without considering the delicate movements of the human hands. In practical applications, hand gesture plays an important role in human communication with the real world, and expresses the primary intention of human beings. In this work, we are the first to formulate a whole-body human pose forecasting task, which jointly predicts the future body and hand activities. Correspondingly, we propose a novel Encoding-Alignment-Interaction (EAI) framework that aims to predict both coarse (body joints) and fine-grained (gestures) activities collaboratively, enabling expressive and cross-facilitated forecasting of 3D whole-body human motions. Specifically, our model involves two key constituents: cross-context alignment (XCA) and cross-context interaction (XCI). Considering the heterogeneous information within the whole-body, XCA aims to align the latent features of various human components, while XCI focuses on effectively capturing the context interaction among the human components. We conduct extensive experiments on a newly-introduced large-scale benchmark and achieve state-of-the-art performance. The code is public for research purposes at https://github.com/Dingpx/EAI.

Related papers

• THOR: Text to Human-Object Interaction Diffusion via Relation Intervention [51.02435289160616]
  We propose a novel Text-guided Human-Object Interaction diffusion model with Relation Intervention (THOR) In each diffusion step, we initiate text-guided human and object motion and then leverage human-object relations to intervene in object motion. We construct Text-BEHAVE, a Text2HOI dataset that seamlessly integrates textual descriptions with the currently largest publicly available 3D HOI dataset.
  arXiv  Detail & Related papers  (2024-03-17T13:17:25Z)
• Inter-X: Towards Versatile Human-Human Interaction Analysis [100.254438708001]
  We propose Inter-X, a dataset with accurate body movements and diverse interaction patterns. The dataset includes 11K interaction sequences and more than 8.1M frames. We also equip Inter-X with versatile annotations of more than 34K fine-grained human part-level textual descriptions.
  arXiv  Detail & Related papers  (2023-12-26T13:36:05Z)
• InterDiff: Generating 3D Human-Object Interactions with Physics-Informed Diffusion [29.25063155767897]
  This paper addresses a novel task of anticipating 3D human-object interactions (HOIs) Our task is significantly more challenging, as it requires modeling dynamic objects with various shapes, capturing whole-body motion, and ensuring physically valid interactions. Experiments on multiple human-object interaction datasets demonstrate the effectiveness of our method for this task, capable of producing realistic, vivid, and remarkably long-term 3D HOI predictions.
  arXiv  Detail & Related papers  (2023-08-31T17:59:08Z)
• Task-Oriented Human-Object Interactions Generation with Implicit Neural Representations [61.659439423703155]
  TOHO: Task-Oriented Human-Object Interactions Generation with Implicit Neural Representations. Our method generates continuous motions that are parameterized only by the temporal coordinate. This work takes a step further toward general human-scene interaction simulation.
  arXiv  Detail & Related papers  (2023-03-23T09:31:56Z)
• Full-Body Articulated Human-Object Interaction [61.01135739641217]
  CHAIRS is a large-scale motion-captured f-AHOI dataset consisting of 16.2 hours of versatile interactions. CHAIRS provides 3D meshes of both humans and articulated objects during the entire interactive process. By learning the geometrical relationships in HOI, we devise the very first model that leverage human pose estimation.
  arXiv  Detail & Related papers  (2022-12-20T19:50:54Z)
• TRiPOD: Human Trajectory and Pose Dynamics Forecasting in the Wild [77.59069361196404]
  TRiPOD is a novel method for predicting body dynamics based on graph attentional networks. To incorporate a real-world challenge, we learn an indicator representing whether an estimated body joint is visible/invisible at each frame. Our evaluation shows that TRiPOD outperforms all prior work and state-of-the-art specifically designed for each of the trajectory and pose forecasting tasks.
  arXiv  Detail & Related papers  (2021-04-08T20:01:00Z)
• Socially and Contextually Aware Human Motion and Pose Forecasting [48.083060946226]
  We propose a novel framework to tackle both tasks of human motion (or skeleton pose) and body skeleton pose forecasting. We consider incorporating both scene and social contexts, as critical clues for this prediction task. Our proposed framework achieves a superior performance compared to several baselines on two social datasets.
  arXiv  Detail & Related papers  (2020-07-14T06:12:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.