Human Hands as Probes for Interactive Object Understanding

• URL: http://arxiv.org/abs/2112.09120v1
• Date: Thu, 16 Dec 2021 18:58:03 GMT
• Title: Human Hands as Probes for Interactive Object Understanding
• Abstract summary: We demonstrate that observation of what human hands interact with and how can provide both the relevant data and the necessary supervision. We apply these basic principles on the EPIC-KITCHENS dataset, and successfully learn state-sensitive features.
• Score: 8.330625683887957
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Interactive object understanding, or what we can do to objects and how is a long-standing goal of computer vision. In this paper, we tackle this problem through observation of human hands in in-the-wild egocentric videos. We demonstrate that observation of what human hands interact with and how can provide both the relevant data and the necessary supervision. Attending to hands, readily localizes and stabilizes active objects for learning and reveals places where interactions with objects occur. Analyzing the hands shows what we can do to objects and how. We apply these basic principles on the EPIC-KITCHENS dataset, and successfully learn state-sensitive features, and object affordances (regions of interaction and afforded grasps), purely by observing hands in egocentric videos.

Related papers

• Vision-Language Models for Egocentric Video: From Hand-Object Interaction to Embodied AI [0.5156484100374058]
  Egocentric video captures activities from the wearer's perspective, providing a direct view of human attention, hand--object interaction, and goal-directed behavior.<n>Current models recognize visible objects more reliably than evolving interactions, actions, and user intent, especially over long activities.<n>We identify temporally grounded reasoning, interaction-aware supervision, efficient long-video processing, multimodal fusion, graph-enhanced representations, cross-domain generalization, privacy, and trustworthy evaluation as key priorities for deployable embodied intelligence.
  arXiv  Detail & Related papers  (2026-08-19T08:21:58Z)
• Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues? [75.43540911934191]
  Hand-object interaction (HOI) recognition requires capturing both hand manipulations and object transformations.<n>We propose a new learning paradigm that combines hand-object masked training and an HOI-dynamics-aware decoder.<n>We show that our training strategy exploits hand- and object-centric information more effectively than existing models.
  arXiv  Detail & Related papers  (2026-07-09T14:09:29Z)
• Emergent Active Perception and Dexterity of Simulated Humanoids from Visual Reinforcement Learning [69.71072181304066]
  We introduce Perceptive Dexterous Control (PDC), a framework for vision-driven whole-body control with simulated humanoids.<n>PDC operates solely on egocentric vision for task specification, enabling object search, target placement, and skill selection through visual cues.<n>We show that training from scratch with reinforcement learning can produce emergent behaviors such as active search.
  arXiv  Detail & Related papers  (2025-05-18T07:33:31Z)
• Visual-Geometric Collaborative Guidance for Affordance Learning [63.038406948791454]
  We propose a visual-geometric collaborative guided affordance learning network that incorporates visual and geometric cues. Our method outperforms the representative models regarding objective metrics and visual quality.
  arXiv  Detail & Related papers  (2024-10-15T07:35:51Z)
• EgoChoir: Capturing 3D Human-Object Interaction Regions from Egocentric Views [51.53089073920215]
  Understanding egocentric human-object interaction (HOI) is a fundamental aspect of human-centric perception. Existing methods primarily leverage observations of HOI to capture interaction regions from an exocentric view. We present EgoChoir, which links object structures with interaction contexts inherent in appearance and head motion to reveal object affordance.
  arXiv  Detail & Related papers  (2024-05-22T14:03:48Z)
• Benchmarks and Challenges in Pose Estimation for Egocentric Hand Interactions with Objects [89.95728475983263]
  holistic 3Dunderstanding of such interactions from egocentric views is important for tasks in robotics, AR/VR, action recognition and motion generation. We design the HANDS23 challenge based on the AssemblyHands and ARCTIC datasets with carefully designed training and testing splits. Based on the results of the top submitted methods and more recent baselines on the leaderboards, we perform a thorough analysis on 3D hand(-object) reconstruction tasks.
  arXiv  Detail & Related papers  (2024-03-25T05:12:21Z)
• AffordPose: A Large-scale Dataset of Hand-Object Interactions with Affordance-driven Hand Pose [16.65196181081623]
  We present AffordPose, a large-scale dataset of hand-object interactions with affordance-driven hand pose. We collect a total of 26.7K hand-object interactions, each including the 3D object shape, the part-level affordance label, and the manually adjusted hand poses. The comprehensive data analysis shows the common characteristics and diversity of hand-object interactions per affordance.
  arXiv  Detail & Related papers  (2023-09-16T10:25:28Z)
• Weakly Supervised Human-Object Interaction Detection in Video via Contrastive Spatiotemporal Regions [81.88294320397826]
  A system does not know what human-object interactions are present in a video as or the actual location of the human and object. We introduce a dataset comprising over 6.5k videos with human-object interaction that have been curated from sentence captions. We demonstrate improved performance over weakly supervised baselines adapted to our annotations on our video dataset.
  arXiv  Detail & Related papers  (2021-10-07T15:30:18Z)
• Capturing the objects of vision with neural networks [0.0]
  Human visual perception carves a scene at its physical joints, decomposing the world into objects. Deep neural network (DNN) models of visual object recognition, by contrast, remain largely tethered to the sensory input. We review related work in both fields and examine how these fields can help each other.
  arXiv  Detail & Related papers  (2021-09-07T21:49:53Z)
• Careful with That! Observation of Human Movements to Estimate Objects Properties [106.925705883949]
  We focus on the features of human motor actions that communicate insights on the weight of an object. Our final goal is to enable a robot to autonomously infer the degree of care required in object handling.
  arXiv  Detail & Related papers  (2021-03-02T08:14:56Z)
• Object Properties Inferring from and Transfer for Human Interaction Motions [51.896592493436984]
  In this paper, we present a fine-grained action recognition method that learns to infer object properties from human interaction motion alone. We collect a large number of videos and 3D skeletal motions of the performing actors using an inertial motion capture device. In particular, we learn to identify the interacting object, by estimating its weight, or its fragility or delicacy.
  arXiv  Detail & Related papers  (2020-08-20T14:36:34Z)
• Learning About Objects by Learning to Interact with Them [29.51363040054068]
  Humans often learn about their world with little to no external supervision. We present a computational framework to discover objects and learn their physical properties. Our agent, when placed within the near photo-realistic and physics-enabled AI2-THOR environment, interacts with its world and learns about objects.
  arXiv  Detail & Related papers  (2020-06-16T16:47:50Z)
• Hand-Priming in Object Localization for Assistive Egocentric Vision [45.38703542455732]
  Egocentric vision holds great promises for increasing access to visual information and improving the quality of life for people with visual impairments. We propose localization models that leverage the presence of the hand as the contextual information for priming the center area of the object of interest. Using egocentric datasets from sighted and blind individuals, we show that the hand-priming achieves higher precision than other approaches.
  arXiv  Detail & Related papers  (2020-02-28T05:32:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.