ShelfHelp: Empowering Humans to Perform Vision-Independent Manipulation Tasks with a Socially Assistive Robotic Cane

• URL: http://arxiv.org/abs/2405.20501v1
• Date: Thu, 30 May 2024 21:42:54 GMT
• Title: ShelfHelp: Empowering Humans to Perform Vision-Independent Manipulation Tasks with a Socially Assistive Robotic Cane
• Authors: Shivendra Agrawal, Suresh Nayak, Ashutosh Naik, Bradley Hayes,
• Abstract summary: We present a proof-of-concept socially assistive robotic system we call ShelfHelp. ShelfHelp includes a novel visual product locator algorithm and a novel planner that autonomously issues verbal manipulation guidance commands. We show the system's success in locating and providing effective manipulation guidance to retrieve desired products with novice users.
• Score: 1.6536887609019442
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The ability to shop independently, especially in grocery stores, is important for maintaining a high quality of life. This can be particularly challenging for people with visual impairments (PVI). Stores carry thousands of products, with approximately 30,000 new products introduced each year in the US market alone, presenting a challenge even for modern computer vision solutions. Through this work, we present a proof-of-concept socially assistive robotic system we call ShelfHelp, and propose novel technical solutions for enhancing instrumented canes traditionally meant for navigation tasks with additional capability within the domain of shopping. ShelfHelp includes a novel visual product locator algorithm designed for use in grocery stores and a novel planner that autonomously issues verbal manipulation guidance commands to guide the user during product retrieval. Through a human subjects study, we show the system's success in locating and providing effective manipulation guidance to retrieve desired products with novice users. We compare two autonomous verbal guidance modes achieving comparable performance to a human assistance baseline and present encouraging findings that validate our system's efficiency and effectiveness and through positive subjective metrics including competence, intelligence, and ease of use.

Related papers

• Casper: Inferring Diverse Intents for Assistive Teleoperation with Vision Language Models [50.19518681574399]
  A central challenge in real-world assistive teleoperation is for the robot to infer a wide range of human intentions from user control inputs.<n>We introduce Casper, an assistive teleoperation system that leverages commonsense knowledge embedded in pre-trained visual language models.<n>We show that Casper improves task performance, reduces human cognitive load, and achieves higher user satisfaction than direct teleoperation and assistive teleoperation baselines.
  arXiv  Detail & Related papers  (2025-06-17T17:06:43Z)
• AI-based Wearable Vision Assistance System for the Visually Impaired: Integrating Real-Time Object Recognition and Contextual Understanding Using Large Vision-Language Models [0.0]
  This paper introduces a novel wearable vision assistance system with artificial intelligence (AI) technology to deliver real-time feedback to a user through a sound beep mechanism. The system provides detailed descriptions of objects in the user's environment using a large vision language model (LVLM)
  arXiv  Detail & Related papers  (2024-12-28T07:26:39Z)
• Smart Help: Strategic Opponent Modeling for Proactive and Adaptive Robot Assistance in Households [30.33911147366425]
  Smart Help aims to provide proactive yet adaptive support to human agents with diverse disabilities. We introduce an innovative opponent modeling module that provides a nuanced understanding of the main agent's capabilities and goals. Our findings illustrate the potential of AI-imbued assistive robots in improving the well-being of vulnerable groups.
  arXiv  Detail & Related papers  (2024-04-13T13:03:59Z)
• Self-Explainable Affordance Learning with Embodied Caption [63.88435741872204]
  We introduce Self-Explainable Affordance learning (SEA) with embodied caption. SEA enables robots to articulate their intentions and bridge the gap between explainable vision-language caption and visual affordance learning. We propose a novel model to effectively combine affordance grounding with self-explanation in a simple but efficient manner.
  arXiv  Detail & Related papers  (2024-04-08T15:22:38Z)
• Can Foundation Models Watch, Talk and Guide You Step by Step to Make a Cake? [62.59699229202307]
  Despite advances in AI, it remains a significant challenge to develop interactive task guidance systems. We created a new multimodal benchmark dataset, Watch, Talk and Guide (WTaG) based on natural interaction between a human user and a human instructor. We leveraged several foundation models to study to what extent these models can be quickly adapted to perceptually enabled task guidance.
  arXiv  Detail & Related papers  (2023-11-01T15:13:49Z)
• NOPA: Neurally-guided Online Probabilistic Assistance for Building Socially Intelligent Home Assistants [79.27554831580309]
  We study how to build socially intelligent robots to assist people in their homes. We focus on assistance with online goal inference, where robots must simultaneously infer humans' goals.
  arXiv  Detail & Related papers  (2023-01-12T18:59:34Z)
• Human Decision Makings on Curriculum Reinforcement Learning with Difficulty Adjustment [52.07473934146584]
  We guide the curriculum reinforcement learning results towards a preferred performance level that is neither too hard nor too easy via learning from the human decision process. Our system is highly parallelizable, making it possible for a human to train large-scale reinforcement learning applications. It shows reinforcement learning performance can successfully adjust in sync with the human desired difficulty level.
  arXiv  Detail & Related papers  (2022-08-04T23:53:51Z)
• ASHA: Assistive Teleoperation via Human-in-the-Loop Reinforcement Learning [91.58711082348293]
  Reinforcement learning from online user feedback on the system's performance presents a natural solution to this problem. This approach tends to require a large amount of human-in-the-loop training data, especially when feedback is sparse. We propose a hierarchical solution that learns efficiently from sparse user feedback.
  arXiv  Detail & Related papers  (2022-02-05T02:01:19Z)
• Towards a Multi-purpose Robotic Nursing Assistant [0.0]
  Multi-purpose Intelligent Nurse Aid (MINA) robotic system is capable of providing walking assistance to the patients and perform teleoperation tasks with an easy-to-use and intuitive Graphical User Interface (GUI) This paper presents preliminary results from the walking assistant task that improves upon the current state-of-the-art methods and shows the developed GUI for teleoperation.
  arXiv  Detail & Related papers  (2021-06-07T15:00:12Z)
• Non-invasive Cognitive-level Human Interfacing for the Robotic Restoration of Reaching & Grasping [5.985098076571228]
  We present a robotic system for human augmentation, capable of actuating the user's arm and fingers for them. We combine wearable eye tracking, the visual context of the environment and the structural grammar of human actions to create a cognitive-level assistive robotic setup.
  arXiv  Detail & Related papers  (2021-02-25T16:32:04Z)
• Customized Handling of Unintended Interface Operation in Assistive Robots [7.657378889055477]
  We present an assistance system that reasons about a human's intended actions during robot teleoperation in order to provide appropriate corrections for unintended behavior. We model the human's physical interaction with a control interface during robot teleoperation and distinguish between intended and measured physical actions explicitly.
  arXiv  Detail & Related papers  (2020-07-04T13:23:22Z)
• AvE: Assistance via Empowerment [77.08882807208461]
  We propose a new paradigm for assistance by instead increasing the human's ability to control their environment. This task-agnostic objective preserves the person's autonomy and ability to achieve any eventual state.
  arXiv  Detail & Related papers  (2020-06-26T04:40:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.