Understanding the Use of a Large Language Model-Powered Guide to Make Virtual Reality Accessible for Blind and Low Vision People

• URL: http://arxiv.org/abs/2603.09964v1
• Date: Tue, 10 Mar 2026 17:56:57 GMT
• Title: Understanding the Use of a Large Language Model-Powered Guide to Make Virtual Reality Accessible for Blind and Low Vision People
• Abstract summary: We developed a large language model (LLM)-powered guide for social virtual reality (VR)<n>We studied its use with 16 blind and low vision (BLV) participants in virtual environments with confederates posing as other users.<n>We found that when alone, participants treated the guide as a tool, but treated it companionably around others.
• Score: 17.62268629374374
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: As social virtual reality (VR) grows more popular, addressing accessibility for blind and low vision (BLV) users is increasingly critical. Researchers have proposed an AI "sighted guide" to help users navigate VR and answer their questions, but it has not been studied with users. To address this gap, we developed a large language model (LLM)-powered guide and studied its use with 16 BLV participants in virtual environments with confederates posing as other users. We found that when alone, participants treated the guide as a tool, but treated it companionably around others, giving it nicknames, rationalizing its mistakes with its appearance, and encouraging confederate-guide interaction. Our work furthers understanding of guides as a versatile method for VR accessibility and presents design recommendations for future guides.

Related papers

• Probing the Gaps in ChatGPT Live Video Chat for Real-World Assistance for People who are Blind or Visually Impaired [10.648018999640758]
  We present findings from an exploratory study with eight blind or visually impaired (BVI) participants.<n>Our findings indicate that current live video AI effectively provides guidance and answers for static visual scenes but falls short in delivering essential live descriptions required in dynamic situations.<n>We discuss implications for assistive video AI agents, including incorporating additional sensing capabilities for real-world use.
  arXiv  Detail & Related papers  (2025-08-05T16:59:02Z)
• Exploring Context-aware and LLM-driven Locomotion for Immersive Virtual Reality [8.469329222500726]
  We propose a novel locomotion technique powered by large language models (LLMs)<n>We evaluate three locomotion methods: controller-based teleportation, voice-based steering, and our language model-driven approach.<n>Our findings indicate that the LLM-driven locomotion possesses comparable usability, presence, and cybersickness scores to established methods.
  arXiv  Detail & Related papers  (2025-04-24T07:48:09Z)
• DWIM: Towards Tool-aware Visual Reasoning via Discrepancy-aware Workflow Generation & Instruct-Masking Tuning [57.285435980459205]
  compositional visual reasoning approaches have shown promise as more effective strategies than end-to-end VR methods.<n>We propose DWIM: Discrepancy-aware training generation, which assesses tool usage and extracts more viable for training.<n>Instruct-Masking fine-tuning, which guides the model to only clone effective actions, enabling the generation of more practical solutions.
  arXiv  Detail & Related papers  (2025-03-25T01:57:59Z)
• Generating Contextually-Relevant Navigation Instructions for Blind and Low Vision People [9.503205949175966]
  Navigating unfamiliar environments presents significant challenges for blind and low-vision (BLV) individuals. We construct a dataset of images and goals across different scenarios such as searching through kitchens or navigating outdoors.
  arXiv  Detail & Related papers  (2024-07-11T06:40:36Z)
• Learning High-Quality Navigation and Zooming on Omnidirectional Images in Virtual Reality [37.564863636844905]
  We present a novel system, called OmniVR, designed to enhance visual clarity during VR navigation. Our system enables users to effortlessly locate and zoom in on the objects of interest in VR.
  arXiv  Detail & Related papers  (2024-05-01T07:08:24Z)
• Human-oriented Representation Learning for Robotic Manipulation [64.59499047836637]
  Humans inherently possess generalizable visual representations that empower them to efficiently explore and interact with the environments in manipulation tasks. We formalize this idea through the lens of human-oriented multi-task fine-tuning on top of pre-trained visual encoders. Our Task Fusion Decoder consistently improves the representation of three state-of-the-art visual encoders for downstream manipulation policy-learning.
  arXiv  Detail & Related papers  (2023-10-04T17:59:38Z)
• Force-Aware Interface via Electromyography for Natural VR/AR Interaction [69.1332992637271]
  We design a learning-based neural interface for natural and intuitive force inputs in VR/AR. We show that our interface can decode finger-wise forces in real-time with 3.3% mean error, and generalize to new users with little calibration. We envision our findings to push forward research towards more realistic physicality in future VR/AR.
  arXiv  Detail & Related papers  (2022-10-03T20:51:25Z)
• Learning Effect of Lay People in Gesture-Based Locomotion in Virtual Reality [81.5101473684021]
  Some of the most promising methods are gesture-based and do not require additional handheld hardware. Recent work focused mostly on user preference and performance of the different locomotion techniques. This work is investigated whether and how quickly users can adapt to a hand gesture-based locomotion system in VR.
  arXiv  Detail & Related papers  (2022-06-16T10:44:16Z)
• A Systematic Review on Interactive Virtual Reality Laboratory [1.3999481573773072]
  This study aims to comprehend the work done in quality education from a distance using VR. Adopting virtual reality in education can help students learn more effectively. This highlights the importance of a significant expansion of VR use in learning.
  arXiv  Detail & Related papers  (2022-03-26T07:16:01Z)
• Wireless Edge-Empowered Metaverse: A Learning-Based Incentive Mechanism for Virtual Reality [102.4151387131726]
  We propose a learning-based Incentive Mechanism framework for VR services in the Metaverse. First, we propose the quality of perception as the metric for VR users in the virtual world. Second, for quick trading of VR services between VR users (i.e., buyers) and VR SPs (i.e., sellers), we design a double Dutch auction mechanism. Third, for auction communication reduction, we design a deep reinforcement learning-based auctioneer to accelerate this auction process.
  arXiv  Detail & Related papers  (2021-11-07T13:02:52Z)
• Reinforcement Learning with Videos: Combining Offline Observations with Interaction [151.73346150068866]
  Reinforcement learning is a powerful framework for robots to acquire skills from experience. Videos of humans are a readily available source of broad and interesting experiences. We propose a framework for reinforcement learning with videos.
  arXiv  Detail & Related papers  (2020-11-12T17:15:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.