NaviSense: A Multimodal Assistive Mobile application for Object Retrieval by Persons with Visual Impairment

• URL: http://arxiv.org/abs/2509.18672v1
• Date: Tue, 23 Sep 2025 05:45:11 GMT
• Title: NaviSense: A Multimodal Assistive Mobile application for Object Retrieval by Persons with Visual Impairment
• Authors: Ajay Narayanan Sridhar, Fuli Qiao, Nelson Daniel Troncoso Aldas, Yanpei Shi, Mehrdad Mahdavi, Laurent Itti, Vijaykrishnan Narayanan,
• Abstract summary: 'NaviSense' is a mobile assistive system that combines conversational AI, vision-language models, augmented reality (AR), and LiDAR.<n>Users specify objects via natural language and receive continuous spatial feedback to navigate toward the target.<n>NaviSense significantly reduced object retrieval time and was preferred over existing tools.
• Score: 21.405966774051326
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: People with visual impairments often face significant challenges in locating and retrieving objects in their surroundings. Existing assistive technologies present a trade-off: systems that offer precise guidance typically require pre-scanning or support only fixed object categories, while those with open-world object recognition lack spatial feedback for reaching the object. To address this gap, we introduce 'NaviSense', a mobile assistive system that combines conversational AI, vision-language models, augmented reality (AR), and LiDAR to support open-world object detection with real-time audio-haptic guidance. Users specify objects via natural language and receive continuous spatial feedback to navigate toward the target without needing prior setup. Designed with insights from a formative study and evaluated with 12 blind and low-vision participants, NaviSense significantly reduced object retrieval time and was preferred over existing tools, demonstrating the value of integrating open-world perception with precise, accessible guidance.

Related papers

• OpenFrontier: General Navigation with Visual-Language Grounded Frontiers [54.661157616245966]
  Open-world navigation requires robots to make decisions in complex everyday environments.<n>Recent advances in vision--language navigation (VLN) and vision--language--action (VLA) models enable end-to-end policies conditioned on natural language.<n>We propose OpenFrontier, a training-free navigation framework that seamlessly integrates diverse vision--language prior models.
  arXiv  Detail & Related papers  (2026-03-05T17:02:22Z)
• MR.NAVI: Mixed-Reality Navigation Assistant for the Visually Impaired [42.45301319345154]
  We present MR. NAVI, a mixed reality system that enhances spatial awareness for visually impaired users.<n>Our system combines computer vision algorithms for object detection and depth estimation with natural language processing to provide contextual scene descriptions.
  arXiv  Detail & Related papers  (2025-05-28T14:02:56Z)
• AI-based Wearable Vision Assistance System for the Visually Impaired: Integrating Real-Time Object Recognition and Contextual Understanding Using Large Vision-Language Models [0.0]
  This paper introduces a novel wearable vision assistance system with artificial intelligence (AI) technology to deliver real-time feedback to a user through a sound beep mechanism.<n>The system provides detailed descriptions of objects in the user's environment using a large vision language model (LVLM)
  arXiv  Detail & Related papers  (2024-12-28T07:26:39Z)
• Interacted Object Grounding in Spatio-Temporal Human-Object Interactions [70.8859442754261]
  We introduce a new open-world benchmark: Grounding Interacted Objects (GIO)<n>An object grounding task is proposed expecting vision systems to discover interacted objects.<n>We propose a 4D question-answering framework (4D-QA) to discover interacted objects from diverse videos.
  arXiv  Detail & Related papers  (2024-12-27T09:08:46Z)
• ObjectFinder: An Open-Vocabulary Assistive System for Interactive Object Search by Blind People [42.050924675417654]
  We present ObjectFinder, an open-vocabulary wearable system for interactive object search by blind people.<n>ObjectFinder allows users to query target objects using flexible wording.<n>It provides egocentric localization information in real-time, including distance and direction.
  arXiv  Detail & Related papers  (2024-12-04T08:38:45Z)
• ESC: Exploration with Soft Commonsense Constraints for Zero-shot Object Navigation [75.13546386761153]
  We present a novel zero-shot object navigation method, Exploration with Soft Commonsense constraints (ESC) ESC transfers commonsense knowledge in pre-trained models to open-world object navigation without any navigation experience. Experiments on MP3D, HM3D, and RoboTHOR benchmarks show that our ESC method improves significantly over baselines.
  arXiv  Detail & Related papers  (2023-01-30T18:37:32Z)
• Detect and Approach: Close-Range Navigation Support for People with Blindness and Low Vision [13.478275180547925]
  People with blindness and low vision (pBLV) experience significant challenges when locating final destinations or targeting specific objects in unfamiliar environments. We develop a novel wearable navigation solution to provide real-time guidance for a user to approach a target object of interest efficiently and effectively in unfamiliar environments.
  arXiv  Detail & Related papers  (2022-08-17T18:38:20Z)
• Object Manipulation via Visual Target Localization [64.05939029132394]
  Training agents to manipulate objects, poses many challenges. We propose an approach that explores the environment in search for target objects, computes their 3D coordinates once they are located, and then continues to estimate their 3D locations even when the objects are not visible. Our evaluations show a massive 3x improvement in success rate over a model that has access to the same sensory suite.
  arXiv  Detail & Related papers  (2022-03-15T17:59:01Z)
• Deep Learning for Embodied Vision Navigation: A Survey [108.13766213265069]
  "Embodied visual navigation" problem requires an agent to navigate in a 3D environment mainly rely on its first-person observation. This paper attempts to establish an outline of the current works in the field of embodied visual navigation by providing a comprehensive literature survey.
  arXiv  Detail & Related papers  (2021-07-07T12:09:04Z)
• Diagnosing Vision-and-Language Navigation: What Really Matters [61.72935815656582]
  Vision-and-language navigation (VLN) is a multimodal task where an agent follows natural language instructions and navigates in visual environments. Recent studies witness a slow-down in the performance improvements in both indoor and outdoor VLN tasks. In this work, we conduct a series of diagnostic experiments to unveil agents' focus during navigation.
  arXiv  Detail & Related papers  (2021-03-30T17:59:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.