BEHAVE: Dataset and Method for Tracking Human Object Interactions

• URL: http://arxiv.org/abs/2204.06950v1
• Date: Thu, 14 Apr 2022 13:21:19 GMT
• Title: BEHAVE: Dataset and Method for Tracking Human Object Interactions
• Authors: Bharat Lal Bhatnagar, Xianghui Xie, Ilya A. Petrov, Cristian Sminchisescu, Christian Theobalt, Gerard Pons-Moll
• Abstract summary: We present the first full body human- object interaction dataset with multi-view RGBD frames and corresponding 3D SMPL and object fits along with the annotated contacts between them. We use this data to learn a model that can jointly track humans and objects in natural environments with an easy-to-use portable multi-camera setup.
• Score: 105.77368488612704
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Modelling interactions between humans and objects in natural environments is central to many applications including gaming, virtual and mixed reality, as well as human behavior analysis and human-robot collaboration. This challenging operation scenario requires generalization to vast number of objects, scenes, and human actions. Unfortunately, there exist no such dataset. Moreover, this data needs to be acquired in diverse natural environments, which rules out 4D scanners and marker based capture systems. We present BEHAVE dataset, the first full body human- object interaction dataset with multi-view RGBD frames and corresponding 3D SMPL and object fits along with the annotated contacts between them. We record around 15k frames at 5 locations with 8 subjects performing a wide range of interactions with 20 common objects. We use this data to learn a model that can jointly track humans and objects in natural environments with an easy-to-use portable multi-camera setup. Our key insight is to predict correspondences from the human and the object to a statistical body model to obtain human-object contacts during interactions. Our approach can record and track not just the humans and objects but also their interactions, modeled as surface contacts, in 3D. Our code and data can be found at: http://virtualhumans.mpi-inf.mpg.de/behave

Related papers

• HOI-M3:Capture Multiple Humans and Objects Interaction within Contextual Environment [43.6454394625555]
  HOI-M3 is a novel large-scale dataset for modeling the interactions of Multiple huMans and Multiple objects. It provides accurate 3D tracking for both humans and objects from dense RGB and object-mounted IMU inputs.
  arXiv  Detail & Related papers  (2024-03-30T09:24:25Z)
• ParaHome: Parameterizing Everyday Home Activities Towards 3D Generative Modeling of Human-Object Interactions [11.32229757116179]
  We introduce the ParaHome system, designed to capture dynamic 3D movements of humans and objects within a common home environment. By leveraging the ParaHome system, we collect a novel large-scale dataset of human-object interaction.
  arXiv  Detail & Related papers  (2024-01-18T18:59:58Z)
• Template Free Reconstruction of Human-object Interaction with Procedural Interaction Generation [38.08445005326031]
  We propose ProciGen to procedurally generate datasets with both, plausible interaction and diverse object variation. We generate 1M+ human-object interaction pairs in 3D and leverage this large-scale data to train our HDM (Procedural Diffusion Model) Our HDM is an image-conditioned diffusion model that learns both realistic interaction and highly accurate human and object shapes.
  arXiv  Detail & Related papers  (2023-12-12T08:32:55Z)
• Revisit Human-Scene Interaction via Space Occupancy [55.67657438543008]
  Human-scene Interaction (HSI) generation is a challenging task and crucial for various downstream tasks. In this work, we argue that interaction with a scene is essentially interacting with the space occupancy of the scene from an abstract physical perspective. By treating pure motion sequences as records of humans interacting with invisible scene occupancy, we can aggregate motion-only data into a large-scale paired human-occupancy interaction database.
  arXiv  Detail & Related papers  (2023-12-05T12:03:00Z)
• Object Motion Guided Human Motion Synthesis [22.08240141115053]
  We study the problem of full-body human motion synthesis for the manipulation of large-sized objects. We propose Object MOtion guided human MOtion synthesis (OMOMO), a conditional diffusion framework. We develop a novel system that captures full-body human manipulation motions by simply attaching a smartphone to the object being manipulated.
  arXiv  Detail & Related papers  (2023-09-28T08:22:00Z)
• Full-Body Articulated Human-Object Interaction [61.01135739641217]
  CHAIRS is a large-scale motion-captured f-AHOI dataset consisting of 16.2 hours of versatile interactions. CHAIRS provides 3D meshes of both humans and articulated objects during the entire interactive process. By learning the geometrical relationships in HOI, we devise the very first model that leverage human pose estimation.
  arXiv  Detail & Related papers  (2022-12-20T19:50:54Z)
• Learn to Predict How Humans Manipulate Large-sized Objects from Interactive Motions [82.90906153293585]
  We propose a graph neural network, HO-GCN, to fuse motion data and dynamic descriptors for the prediction task. We show the proposed network that consumes dynamic descriptors can achieve state-of-the-art prediction results and help the network better generalize to unseen objects.
  arXiv  Detail & Related papers  (2022-06-25T09:55:39Z)
• Interaction Replica: Tracking Human-Object Interaction and Scene Changes From Human Motion [48.982957332374866]
  Modeling changes caused by humans is essential for building digital twins. Our method combines visual localization of humans in the scene with contact-based reasoning about human-scene interactions from IMU data. Our code, data and model are available on our project page at http://virtualhumans.mpi-inf.mpg.de/ireplica/.
  arXiv  Detail & Related papers  (2022-05-05T17:58:06Z)
• Estimating 3D Motion and Forces of Human-Object Interactions from Internet Videos [49.52070710518688]
  We introduce a method to reconstruct the 3D motion of a person interacting with an object from a single RGB video. Our method estimates the 3D poses of the person together with the object pose, the contact positions and the contact forces on the human body.
  arXiv  Detail & Related papers  (2021-11-02T13:40:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.