Insight: Enhancing Mobile Accessibility for Blind and Visually Impaired Users with LLMs

• URL: http://arxiv.org/abs/2605.09803v1
• Date: Sun, 10 May 2026 22:51:44 GMT
• Title: Insight: Enhancing Mobile Accessibility for Blind and Visually Impaired Users with LLMs
• Abstract summary: This paper introduces Insight, an Android accessibility service that provides natural language interaction and real-time summarization of the screen.<n>Results show Insight reduced mental effort and task time, and was preferred because of its dialogue interface, but users felt the need for interruption management.
• Score: 0.6717411448500911
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This research paper addresses the limitations of current mobile accessibility services like TalkBack, which provide manual gesture-based sequential feedback to BVI users. Motivated by the promise of large language models (LLMs), this paper introduces Insight, an Android accessibility service that provides natural language interaction and real-time summarization of the screen. The paper performs a within-subject experimental study with users to compare Insight and TalkBack on usability factors. Results show Insight reduced mental effort and task time, and was preferred because of its dialogue interface, but users felt the need for interruption management. Results show LLM-based interfaces can significantly improve mobile accessibility, and describe the potential of hybrid solutions combining gesture and dialogue modalities towards more inclusive design.

Related papers

• Personal Visual Context Learning in Large Multimodal Models [53.33351452841322]
  We present Personal Visual Context Learning (Personal ), the prompt-time capability of using user-specific visual context to resolve personalized queries.<n>Our analysis of LMMs identifies a profound context utilization gap, revealing that the mechanisms for leveraging visual evidence, as well as aggregating multiple visual observations, remain critically understudied.<n>Motivated by these findings, we propose the Agentic Context Bank, a strong-time baseline that structures a user's visual context into a self-refining memory bank and employs query-adaptive evidence selection.
  arXiv  Detail & Related papers  (2026-05-11T17:59:17Z)
• GhostUI: Unveiling Hidden Interactions in Mobile UI [12.023496228003337]
  GhostUI is a new dataset designed to enable the detection of hidden interactions in mobile applications.<n>GhostUI provides before-and-after screenshots, simplified view hierarchies, gesture metadata, and task descriptions.
  arXiv  Detail & Related papers  (2026-01-27T06:40:29Z)
• Agentic Conversational Search with Contextualized Reasoning via Reinforcement Learning [66.52010873968383]
  We introduce a conversational agent that interleaves search and reasoning across turns, enabling exploratory and adaptive behaviors learned through reinforcement learning (RL) training.<n>The experimental results across four widely used conversational benchmarks demonstrate the effectiveness of our methods.
  arXiv  Detail & Related papers  (2026-01-19T14:55:54Z)
• Generative Interfaces for Language Models [70.25765232527762]
  We propose a paradigm in which large language models (LLMs) respond to user queries by proactively generating user interfaces (UIs)<n>Our framework leverages structured interface-specific representations and iterative refinements to translate user queries into task-specific UIs.<n>Results show that generative interfaces consistently outperform conversational ones, with up to a 72% improvement in human preference.
  arXiv  Detail & Related papers  (2025-08-26T17:43:20Z)
• User Feedback in Human-LLM Dialogues: A Lens to Understand Users But Noisy as a Learning Signal [59.120335322495436]
  We analyze user feedback in the user-LLM conversation logs, providing insights into when and why such feedback occurs.<n>Second, we study harvesting learning signals from such implicit user feedback.
  arXiv  Detail & Related papers  (2025-07-30T23:33:29Z)
• Designing Effective LLM-Assisted Interfaces for Curriculum Development [0.0]
  Large Language Models (LLMs) have the potential to transform the way a dynamic curriculum can be delivered.<n>This paper introduces two novel User Interface (UI) designs, UI Predefined and UI Open.<n>By reducing the reliance on intricate prompt engineering, these UIs improve usability, streamline interaction, and lower workload.
  arXiv  Detail & Related papers  (2025-06-13T13:21:53Z)
• Exploring Mobile Touch Interaction with Large Language Models [26.599610206222142]
  We propose to control Large Language Models via touch gestures performed directly on the text.<n>Results demonstrate that touch-based control of LLMs is both feasible and user-friendly.<n>This work lays the foundation for further research into gesture-based interaction with LLMs on touch devices.
  arXiv  Detail & Related papers  (2025-02-11T15:17:00Z)
• Constraining Participation: Affordances of Feedback Features in Interfaces to Large Language Models [49.266685603250416]
  Large language models (LLMs) are now accessible to anyone with a computer, a web browser, and an internet connection via browser-based interfaces.<n>This article examines how interactive feedback features in ChatGPT's interface afford user participation in LLMs.
  arXiv  Detail & Related papers  (2024-08-27T13:50:37Z)
• Large Language User Interfaces: Voice Interactive User Interfaces powered by LLMs [5.06113628525842]
  We present a framework that can serve as an intermediary between a user and their user interface (UI) We employ a system that stands upon textual semantic mappings of UI components, in the form of annotations. Our engine can classify the most appropriate application, extract relevant parameters, and subsequently execute precise predictions of the user's expected actions.
  arXiv  Detail & Related papers  (2024-02-07T21:08:49Z)
• Enabling Conversational Interaction with Mobile UI using Large Language Models [15.907868408556885]
  To perform diverse UI tasks with natural language, developers typically need to create separate datasets and models for each specific task. This paper investigates the feasibility of enabling versatile conversational interactions with mobile UIs using a single language model.
  arXiv  Detail & Related papers  (2022-09-18T20:58:39Z)
• Mobile App Tasks with Iterative Feedback (MoTIF): Addressing Task Feasibility in Interactive Visual Environments [54.405920619915655]
  We introduce Mobile app Tasks with Iterative Feedback (MoTIF), a dataset with natural language commands for the greatest number of interactive environments to date. MoTIF is the first to contain natural language requests for interactive environments that are not satisfiable. We perform initial feasibility classification experiments and only reach an F1 score of 37.3, verifying the need for richer vision-language representations.
  arXiv  Detail & Related papers  (2021-04-17T14:48:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.