Hand-Priming in Object Localization for Assistive Egocentric Vision

• URL: http://arxiv.org/abs/2002.12557v1
• Date: Fri, 28 Feb 2020 05:32:36 GMT
• Title: Hand-Priming in Object Localization for Assistive Egocentric Vision
• Abstract summary: Egocentric vision holds great promises for increasing access to visual information and improving the quality of life for people with visual impairments. We propose localization models that leverage the presence of the hand as the contextual information for priming the center area of the object of interest. Using egocentric datasets from sighted and blind individuals, we show that the hand-priming achieves higher precision than other approaches.
• Score: 45.38703542455732
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Egocentric vision holds great promises for increasing access to visual information and improving the quality of life for people with visual impairments, with object recognition being one of the daily challenges for this population. While we strive to improve recognition performance, it remains difficult to identify which object is of interest to the user; the object may not even be included in the frame due to challenges in camera aiming without visual feedback. Also, gaze information, commonly used to infer the area of interest in egocentric vision, is often not dependable. However, blind users often tend to include their hand either interacting with the object that they wish to recognize or simply placing it in proximity for better camera aiming. We propose localization models that leverage the presence of the hand as the contextual information for priming the center area of the object of interest. In our approach, hand segmentation is fed to either the entire localization network or its last convolutional layers. Using egocentric datasets from sighted and blind individuals, we show that the hand-priming achieves higher precision than other approaches, such as fine-tuning, multi-class, and multi-task learning, which also encode hand-object interactions in localization.

Related papers

• Focus Where It Counts: A Salience-Driven Vision-Language Model for Low Vision Assistance [18.85523453781126]
  We propose a salience-driven captioning framework that prioritizes scene elements according to their importance for human-centered assistance.<n>We build salience-aware datasets verified by low vision participants, propose Salience-LLaVA to describe objects in the order of importance, introduce SCMI to evaluate ordering accuracy, and deploy the system on assistive glasses to demonstrate real-world practicality.
  arXiv  Detail & Related papers  (2026-08-28T11:35:00Z)
• Vision-Language Models for Egocentric Video: From Hand-Object Interaction to Embodied AI [0.5156484100374058]
  Egocentric video captures activities from the wearer's perspective, providing a direct view of human attention, hand--object interaction, and goal-directed behavior.<n>Current models recognize visible objects more reliably than evolving interactions, actions, and user intent, especially over long activities.<n>We identify temporally grounded reasoning, interaction-aware supervision, efficient long-video processing, multimodal fusion, graph-enhanced representations, cross-domain generalization, privacy, and trustworthy evaluation as key priorities for deployable embodied intelligence.
  arXiv  Detail & Related papers  (2026-08-19T08:21:58Z)
• Do Egocentric Video-Language Models Capture Both Hand- and Object-Centric Cues? [75.43540911934191]
  Hand-object interaction (HOI) recognition requires capturing both hand manipulations and object transformations.<n>We propose a new learning paradigm that combines hand-object masked training and an HOI-dynamics-aware decoder.<n>We show that our training strategy exploits hand- and object-centric information more effectively than existing models.
  arXiv  Detail & Related papers  (2026-07-09T14:09:29Z)
• Hand-Aware Egocentric Motion Reconstruction with Sequence-Level Context [17.735273173582716]
  We present HaMoS, the first hand-aware, sequence-level diffusion framework that directly conditions on both head trajectory and intermittently visible hand cues.<n>We also demonstrate that sequence-level contexts such as body shape and field-of-view are crucial for accurate motion reconstruction.
  arXiv  Detail & Related papers  (2025-12-22T11:26:41Z)
• NaviSense: A Multimodal Assistive Mobile application for Object Retrieval by Persons with Visual Impairment [21.405966774051326]
  'NaviSense' is a mobile assistive system that combines conversational AI, vision-language models, augmented reality (AR), and LiDAR.<n>Users specify objects via natural language and receive continuous spatial feedback to navigate toward the target.<n>NaviSense significantly reduced object retrieval time and was preferred over existing tools.
  arXiv  Detail & Related papers  (2025-09-23T05:45:11Z)
• A Light and Smart Wearable Platform with Multimodal Foundation Model for Enhanced Spatial Reasoning in People with Blindness and Low Vision [9.057330310306696]
  People with blindness and low vision (pBLV) face significant challenges, struggling to navigate environments and locate objects due to limited visual cues.<n>Current multi-modal large language (MLLM) models for low vision people lack the spatial reasoning capabilities needed to effectively assist in these tasks.<n>We propose a novel spatial enhanced multi-modal large language model based approach for visually impaired individuals.
  arXiv  Detail & Related papers  (2025-05-16T05:32:25Z)
• Benchmarks and Challenges in Pose Estimation for Egocentric Hand Interactions with Objects [89.95728475983263]
  holistic 3Dunderstanding of such interactions from egocentric views is important for tasks in robotics, AR/VR, action recognition and motion generation. We design the HANDS23 challenge based on the AssemblyHands and ARCTIC datasets with carefully designed training and testing splits. Based on the results of the top submitted methods and more recent baselines on the leaderboards, we perform a thorough analysis on 3D hand(-object) reconstruction tasks.
  arXiv  Detail & Related papers  (2024-03-25T05:12:21Z)
• InterTracker: Discovering and Tracking General Objects Interacting with Hands in the Wild [40.489171608114574]
  Existing methods rely on frame-based detectors to locate interacting objects. We propose to leverage hand-object interaction to track interactive objects. Our proposed method outperforms the state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-08-06T09:09:17Z)
• Spotlight Attention: Robust Object-Centric Learning With a Spatial Locality Prior [88.9319150230121]
  Object-centric vision aims to construct an explicit representation of the objects in a scene. We incorporate a spatial-locality prior into state-of-the-art object-centric vision models. We obtain significant improvements in segmenting objects in both synthetic and real-world datasets.
  arXiv  Detail & Related papers  (2023-05-31T04:35:50Z)
• Bi-directional Object-context Prioritization Learning for Saliency Ranking [60.62461793691836]
  Existing approaches focus on learning either object-object or object-scene relations. We observe that spatial attention works concurrently with object-based attention in the human visual recognition system. We propose a novel bi-directional method to unify spatial attention and object-based attention for saliency ranking.
  arXiv  Detail & Related papers  (2022-03-17T16:16:03Z)
• Human Hands as Probes for Interactive Object Understanding [8.330625683887957]
  We demonstrate that observation of what human hands interact with and how can provide both the relevant data and the necessary supervision. We apply these basic principles on the EPIC-KITCHENS dataset, and successfully learn state-sensitive features.
  arXiv  Detail & Related papers  (2021-12-16T18:58:03Z)
• Recurrent Attention Models with Object-centric Capsule Representation for Multi-object Recognition [4.143091738981101]
  We show that an object-centric hidden representation in an encoder-decoder model with iterative glimpse attention yields effective integration of attention and recognition. Our work takes a step toward a general architecture for how to integrate recurrent object-centric representation into the planning of attentional glimpses.
  arXiv  Detail & Related papers  (2021-10-11T01:41:21Z)
• Weakly Supervised Human-Object Interaction Detection in Video via Contrastive Spatiotemporal Regions [81.88294320397826]
  A system does not know what human-object interactions are present in a video as or the actual location of the human and object. We introduce a dataset comprising over 6.5k videos with human-object interaction that have been curated from sentence captions. We demonstrate improved performance over weakly supervised baselines adapted to our annotations on our video dataset.
  arXiv  Detail & Related papers  (2021-10-07T15:30:18Z)
• Capturing the objects of vision with neural networks [0.0]
  Human visual perception carves a scene at its physical joints, decomposing the world into objects. Deep neural network (DNN) models of visual object recognition, by contrast, remain largely tethered to the sensory input. We review related work in both fields and examine how these fields can help each other.
  arXiv  Detail & Related papers  (2021-09-07T21:49:53Z)
• Spatio-Temporal Graph for Video Captioning with Knowledge Distillation [50.034189314258356]
  We propose a graph model for video captioning that exploits object interactions in space and time. Our model builds interpretable links and is able to provide explicit visual grounding. To avoid correlations caused by the variable number of objects, we propose an object-aware knowledge distillation mechanism.
  arXiv  Detail & Related papers  (2020-03-31T03:58:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.