HOIMotion: Forecasting Human Motion During Human-Object Interactions Using Egocentric 3D Object Bounding Boxes

• URL: http://arxiv.org/abs/2407.02633v1
• Date: Tue, 2 Jul 2024 19:58:35 GMT
• Title: HOIMotion: Forecasting Human Motion During Human-Object Interactions Using Egocentric 3D Object Bounding Boxes
• Authors: Zhiming Hu, Zheming Yin, Daniel Haeufle, Syn Schmitt, Andreas Bulling,
• Abstract summary: We present HOIMotion, a novel approach for human motion forecasting during human-object interactions. Our method integrates information about past body poses and egocentric 3D object bounding boxes. We show that HOIMotion consistently outperforms state-of-the-art methods by a large margin.
• Score: 10.237077867790612
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present HOIMotion - a novel approach for human motion forecasting during human-object interactions that integrates information about past body poses and egocentric 3D object bounding boxes. Human motion forecasting is important in many augmented reality applications but most existing methods have only used past body poses to predict future motion. HOIMotion first uses an encoder-residual graph convolutional network (GCN) and multi-layer perceptrons to extract features from body poses and egocentric 3D object bounding boxes, respectively. Our method then fuses pose and object features into a novel pose-object graph and uses a residual-decoder GCN to forecast future body motion. We extensively evaluate our method on the Aria digital twin (ADT) and MoGaze datasets and show that HOIMotion consistently outperforms state-of-the-art methods by a large margin of up to 8.7% on ADT and 7.2% on MoGaze in terms of mean per joint position error. Complementing these evaluations, we report a human study (N=20) that shows that the improvements achieved by our method result in forecasted poses being perceived as both more precise and more realistic than those of existing methods. Taken together, these results reveal the significant information content available in egocentric 3D object bounding boxes for human motion forecasting and the effectiveness of our method in exploiting this information.

Related papers

• Diff-IP2D: Diffusion-Based Hand-Object Interaction Prediction on Egocentric Videos [22.81433371521832]
  We propose Diff-IP2D to forecast future hand trajectories and object affordances concurrently in an iterative non-autoregressive manner. Our method significantly outperforms the state-of-the-art baselines on both the off-the-shelf metrics and our newly proposed evaluation protocol.
  arXiv  Detail & Related papers  (2024-05-07T14:51:05Z)
• HMP: Hand Motion Priors for Pose and Shape Estimation from Video [52.39020275278984]
  We develop a generative motion prior specific for hands, trained on the AMASS dataset which features diverse and high-quality hand motions. Our integration of a robust motion prior significantly enhances performance, especially in occluded scenarios. We demonstrate our method's efficacy via qualitative and quantitative evaluations on the HO3D and DexYCB datasets.
  arXiv  Detail & Related papers  (2023-12-27T22:35:33Z)
• Human Action Recognition in Egocentric Perspective Using 2D Object and Hands Pose [2.0305676256390934]
  Egocentric action recognition is essential for healthcare and assistive technology that relies on egocentric cameras. This study explores the feasibility of using 2D hand and object pose information for egocentric action recognition.
  arXiv  Detail & Related papers  (2023-06-08T12:15:16Z)
• UnrealEgo: A New Dataset for Robust Egocentric 3D Human Motion Capture [70.59984501516084]
  UnrealEgo is a new large-scale naturalistic dataset for egocentric 3D human pose estimation. It is based on an advanced concept of eyeglasses equipped with two fisheye cameras that can be used in unconstrained environments. We propose a new benchmark method with a simple but effective idea of devising a 2D keypoint estimation module for stereo inputs to improve 3D human pose estimation.
  arXiv  Detail & Related papers  (2022-08-02T17:59:54Z)
• Learn to Predict How Humans Manipulate Large-sized Objects from Interactive Motions [82.90906153293585]
  We propose a graph neural network, HO-GCN, to fuse motion data and dynamic descriptors for the prediction task. We show the proposed network that consumes dynamic descriptors can achieve state-of-the-art prediction results and help the network better generalize to unseen objects.
  arXiv  Detail & Related papers  (2022-06-25T09:55:39Z)
• Estimating 3D Motion and Forces of Human-Object Interactions from Internet Videos [49.52070710518688]
  We introduce a method to reconstruct the 3D motion of a person interacting with an object from a single RGB video. Our method estimates the 3D poses of the person together with the object pose, the contact positions and the contact forces on the human body.
  arXiv  Detail & Related papers  (2021-11-02T13:40:18Z)
• Improving Robustness and Accuracy via Relative Information Encoding in 3D Human Pose Estimation [59.94032196768748]
  We propose a relative information encoding method that yields positional and temporal enhanced representations. Our method outperforms state-of-the-art methods on two public datasets.
  arXiv  Detail & Related papers  (2021-07-29T14:12:19Z)
• TRiPOD: Human Trajectory and Pose Dynamics Forecasting in the Wild [77.59069361196404]
  TRiPOD is a novel method for predicting body dynamics based on graph attentional networks. To incorporate a real-world challenge, we learn an indicator representing whether an estimated body joint is visible/invisible at each frame. Our evaluation shows that TRiPOD outperforms all prior work and state-of-the-art specifically designed for each of the trajectory and pose forecasting tasks.
  arXiv  Detail & Related papers  (2021-04-08T20:01:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.