Hoi! -- A Multimodal Dataset for Force-Grounded, Cross-View Articulated Manipulation

• URL: http://arxiv.org/abs/2512.04884v2
• Date: Tue, 10 Feb 2026 14:42:26 GMT
• Title: Hoi! -- A Multimodal Dataset for Force-Grounded, Cross-View Articulated Manipulation
• Abstract summary: We present a dataset for force-grounded, cross-view articulated manipulation.<n>The dataset contains 3048 sequences across 381 articulated objects in 38 environments.
• Score: 53.99392085883293
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present a dataset for force-grounded, cross-view articulated manipulation that couples what is seen with what is done and what is felt during real human interaction. The dataset contains 3048 sequences across 381 articulated objects in 38 environments. Each object is operated under four embodiments - (i) human hand, (ii) human hand with a wrist-mounted camera, (iii) handheld UMI gripper, and (iv) a custom Hoi! gripper - where the tool embodiment provides synchronized end-effector forces and tactile sensing. Our dataset offers a holistic view of interaction understanding from video, enabling researchers to evaluate how well methods transfer between human and robotic viewpoints, but also investigate underexplored modalities such as force sensing and prediction. Further information can be found on the Website.

Related papers

• RoboTacDex: A Dexterous Visual-Tactile-Action Dataset for Humanoid Manipulation [29.49648908136145]
  RoboTacDex is a large, multi-modal, and diverse dataset of dexterous manipulation behaviors performed with a humanoid robot.<n>Built on the publicly accessible humanoid robot Unitree G1, RoboTacDex consists of 6k trajectories covering 19 tasks, 23 skills, and interactions with 22 objects.
  arXiv  Detail & Related papers  (2026-06-30T15:39:29Z)
• Do as I Do: Dexterous Manipulation Data from Everyday Human Videos [69.98851738909168]
  We present DO AS I DO, an algorithm to reconstruct and retarget monocular RGB human videos to multi-fingered dexterous robotic hands.<n>The algorithm then retargets these hand-object interaction estimates into a sequence of actions executable in the real world.<n>Overall, DO AS I DO outperforms previous state of the art in estimating hand-object interactions and extracting dexterous manipulation trajectories from RGB videos.
  arXiv  Detail & Related papers  (2026-06-17T17:57:34Z)
• Articulated 3D Scene Graphs for Open-World Mobile Manipulation [55.97942733699124]
  We present MoMa-SG, a framework for building semantic-kinematic 3D scene graphs of articulated scenes.<n>We estimate articulation models using a novel unified twist estimation formulation.<n>We also introduce the novel Arti4D-Semantic dataset.
  arXiv  Detail & Related papers  (2026-02-18T10:40:35Z)
• DexCanvas: Bridging Human Demonstrations and Robot Learning for Dexterous Manipulation [25.208854363099352]
  This dataset contains 7,000 hours of dexterous hand-object interactions seeded from 70 hours of real human demonstrations.<n>Each entry combines synchronized multi-view RGB-D, high-precision mocap with MANO hand parameters, and per-frame contact points with physically consistent force profiles.<n>Our real-to-sim pipeline uses reinforcement learning to train policies that control an actuated MANO hand in physics simulation.
  arXiv  Detail & Related papers  (2025-10-17T16:08:14Z)
• Grasp Like Humans: Learning Generalizable Multi-Fingered Grasping from Human Proprioceptive Sensorimotor Integration [26.351720551267846]
  Tactile and kinesthetic perceptions are crucial for human dexterous manipulation, enabling reliable grasping of objects via sensorimotor integration.<n>We propose a novel glove-mediated tactile-kinematic perception-prediction framework for grasp skill transfer from human intuitive and natural operation to robotic execution based on imitation learning.
  arXiv  Detail & Related papers  (2025-09-10T07:44:12Z)
• Benchmarks and Challenges in Pose Estimation for Egocentric Hand Interactions with Objects [89.95728475983263]
  holistic 3Dunderstanding of such interactions from egocentric views is important for tasks in robotics, AR/VR, action recognition and motion generation. We design the HANDS23 challenge based on the AssemblyHands and ARCTIC datasets with carefully designed training and testing splits. Based on the results of the top submitted methods and more recent baselines on the leaderboards, we perform a thorough analysis on 3D hand(-object) reconstruction tasks.
  arXiv  Detail & Related papers  (2024-03-25T05:12:21Z)
• HOH: Markerless Multimodal Human-Object-Human Handover Dataset with Large Object Count [4.441651788685931]
  The HOH (Human-Object-Human) Handover dataset is a large object count dataset with 136 objects. HOH represents natural human-human handover interactions, overcoming challenges with markered datasets. To date, HOH is the largest handover dataset in number of objects, participants, pairs with role reversal accounted for, and total interactions captured.
  arXiv  Detail & Related papers  (2023-10-01T16:48:48Z)
• Full-Body Articulated Human-Object Interaction [61.01135739641217]
  CHAIRS is a large-scale motion-captured f-AHOI dataset consisting of 16.2 hours of versatile interactions. CHAIRS provides 3D meshes of both humans and articulated objects during the entire interactive process. By learning the geometrical relationships in HOI, we devise the very first model that leverage human pose estimation.
  arXiv  Detail & Related papers  (2022-12-20T19:50:54Z)
• Video-based Pose-Estimation Data as Source for Transfer Learning in Human Activity Recognition [71.91734471596433]
  Human Activity Recognition (HAR) using on-body devices identifies specific human actions in unconstrained environments. Previous works demonstrated that transfer learning is a good strategy for addressing scenarios with scarce data. This paper proposes using datasets intended for human-pose estimation as a source for transfer learning.
  arXiv  Detail & Related papers  (2022-12-02T18:19:36Z)
• HANDS: A Multimodal Dataset for Modeling Towards Human Grasp Intent Inference in Prosthetic Hands [3.7886097009023376]
  Advanced prosthetic hands of the future are anticipated to benefit from improved shared control between a robotic hand and its human user. multimodal sensor data may include various environment sensors including vision, as well as human physiology and behavior sensors. A fusion methodology for environmental state and human intent estimation can combine these sources of evidence in order to help prosthetic hand motion planning and control.
  arXiv  Detail & Related papers  (2021-03-08T15:51:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.