VisionAssist: An Open-Source Smartphone Assistant for AI-Based Visual Accessibility
- URL: http://arxiv.org/abs/2607.16750v1
- Date: Sat, 18 Jul 2026 10:28:57 GMT
- Title: VisionAssist: An Open-Source Smartphone Assistant for AI-Based Visual Accessibility
- Abstract summary: textbfVisionAssist is an open-source mobile application designed to improve independence by providing AI-powered visual assistance through a smartphone.<n>It generates spoken descriptions of captured images, allowing users to identify visual content such as food labels, documents, and everyday objects.<n>The application supports hands-free interaction through voice commands and delivers all feedback using text-to-speech synthesis.
- Score: 0.5371337604556311
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: People with low vision often face challenges in performing everyday tasks that require interpreting visual information. We present \textbf{VisionAssist}, an open-source mobile application designed to improve independence by providing AI-powered visual assistance through a smartphone. The application integrates three complementary functionalities within a single interface. First, it enables users to locate specific objects by analyzing the live camera feed. Second, it generates spoken descriptions of captured images, allowing users to identify visual content such as food labels, documents, and everyday objects. Third, it integrates with the smartphone's contacts and calendar to facilitate emergency calls and provide voice-based reminders. The application supports hands-free interaction through voice commands and delivers all feedback using text-to-speech synthesis, making it fully accessible to users with visual impairments. By combining multiple assistive services into a unified platform and releasing the project as open-source software, the proposed solution aims to encourage community contributions and accelerate the development of accessible technologies. The source code is publicly available at: https://github.com/AOzlemC/LowVisionProject.git
Related papers
- Sanvaad: A Multimodal Accessibility Framework for ISL Recognition and Voice-Based Interaction [0.0]
Sanvaad is a lightweight multimodal accessibility framework designed to support real time, two-way communication.<n>For deaf users, Sanvaad includes an ISL recognition module built on MediaPipe landmarks.<n>For visually impaired users, the framework provides a screen free voice interface that integrates multilingual speech recognition, text summarization, and text-to-speech generation.
arXiv Detail & Related papers (2025-12-06T16:16:21Z) - An Artificial Intelligence-based Assistant for the Visually Impaired [2.7825760447670955]
This paper describes an artificial intelligence-based assistant application, AIDEN, developed during 2023 and 2024.<n>Visually impaired individuals face challenges in identifying objects, reading text, and navigating unfamiliar environments.<n>This application leverages state-of-the-art machine learning algorithms to identify and describe objects, read text, and answer questions about the environment.
arXiv Detail & Related papers (2025-11-08T17:23:51Z) - Vinci: A Real-time Embodied Smart Assistant based on Egocentric Vision-Language Model [49.90916095152366]
We introduce Vinci, a real-time embodied smart assistant built upon an egocentric vision-language model.<n>vinci operates in an "always on" mode, continuously observing the environment to deliver seamless interaction and assistance.<n>We release the complete implementation for the development of the device in conjunction with a demo web platform to test uploaded videos.
arXiv Detail & Related papers (2024-12-30T16:57:05Z) - AI-based Wearable Vision Assistance System for the Visually Impaired: Integrating Real-Time Object Recognition and Contextual Understanding Using Large Vision-Language Models [0.0]
This paper introduces a novel wearable vision assistance system with artificial intelligence (AI) technology to deliver real-time feedback to a user through a sound beep mechanism.<n>The system provides detailed descriptions of objects in the user's environment using a large vision language model (LVLM)
arXiv Detail & Related papers (2024-12-28T07:26:39Z) - OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and Understanding [112.87441334765693]
OMG-LLaVA is a new framework combining powerful pixel-level vision understanding with reasoning abilities.
It can accept various visual and text prompts for flexible user interaction.
OMG-LLaVA achieves image-level, object-level, and pixel-level reasoning and understanding in a single model.
arXiv Detail & Related papers (2024-06-27T17:59:01Z) - Tell Me What's Next: Textual Foresight for Generic UI Representations [65.10591722192609]
We propose Textual Foresight, a novel pretraining objective for learning UI screen representations.
Textual Foresight generates global text descriptions of future UI states given a current UI and local action taken.
We train with our newly constructed mobile app dataset, OpenApp, which results in the first public dataset for app UI representation learning.
arXiv Detail & Related papers (2024-06-12T02:43:19Z) - Improve accessibility for Low Vision and Blind people using Machine Learning and Computer Vision [0.0]
This project explores how machine learning and computer vision could be utilized to improve accessibility for people with visual impairments.
This project will concentrate on building a mobile application that helps blind people to orient in space by receiving audio and haptic feedback.
arXiv Detail & Related papers (2024-03-24T21:19:17Z) - VisionGPT: Vision-Language Understanding Agent Using Generalized Multimodal Framework [47.58359136198136]
We introduce VisionGPT to consolidate and automate the integration of state-of-the-art foundation models.
VisionGPT builds upon a generalized multimodal framework that distinguishes itself through three key features.
This paper outlines the architecture and capabilities of VisionGPT, demonstrating its potential to revolutionize the field of computer vision.
arXiv Detail & Related papers (2024-03-14T01:39:40Z) - VLG: General Video Recognition with Web Textual Knowledge [47.3660792813967]
We focus on the general video recognition (GVR) problem of solving different recognition tasks within a unified framework.
By leveraging semantic knowledge from noisy text descriptions crawled from the Internet, we present a unified visual-linguistic framework (VLG)
Our VLG is first pre-trained on video and language datasets to learn a shared feature space, and then devises a flexible bi-modal attention head to collaborate high-level semantic concepts under different settings.
arXiv Detail & Related papers (2022-12-03T15:46:49Z) - AudioVisual Video Summarization [103.47766795086206]
In video summarization, existing approaches just exploit the visual information while neglecting the audio information.
We propose to jointly exploit the audio and visual information for the video summarization task, and develop an AudioVisual Recurrent Network (AVRN) to achieve this.
arXiv Detail & Related papers (2021-05-17T08:36:10Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.