ScreenAudit: Detecting Screen Reader Accessibility Errors in Mobile Apps Using Large Language Models

• URL: http://arxiv.org/abs/2504.02110v1
• Date: Wed, 02 Apr 2025 20:18:45 GMT
• Title: ScreenAudit: Detecting Screen Reader Accessibility Errors in Mobile Apps Using Large Language Models
• Abstract summary: ScreenAudit is a system designed to traverse mobile app screens, extract metadata and transcripts, and identify screen reader accessibility errors.<n>Our findings indicate that ScreenAudit achieves an average coverage of 69.2%, compared to only 31.3% with a widely-used accessibility checker.
• Score: 16.056030925238684
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Many mobile apps are inaccessible, thereby excluding people from their potential benefits. Existing rule-based accessibility checkers aim to mitigate these failures by identifying errors early during development but are constrained in the types of errors they can detect. We present ScreenAudit, an LLM-powered system designed to traverse mobile app screens, extract metadata and transcripts, and identify screen reader accessibility errors overlooked by existing checkers. We recruited six accessibility experts including one screen reader user to evaluate ScreenAudit's reports across 14 unique app screens. Our findings indicate that ScreenAudit achieves an average coverage of 69.2%, compared to only 31.3% with a widely-used accessibility checker. Expert feedback indicated that ScreenAudit delivered higher-quality feedback and addressed more aspects of screen reader accessibility compared to existing checkers, and that ScreenAudit would benefit app developers in real-world settings.

Related papers

• SPIEval: Evaluating Large Language Models as Mobile Assistants over Scattered Personal Information [61.41881137371855]
  We introduce SPIEval, a benchmark for large language models (LLMs)<n> SPIEval comprises 250 tasks spanning 4,335 personal records distributed across 10 apps and supports multi-turn interaction through 21 tools.<n>The best-performing model, GPT-5.5 (xhigh), achieves only 57.3% accuracy, while the weakest achieves just 16.4%.
  arXiv  Detail & Related papers  (2026-08-11T09:14:53Z)
• Spotting Setting-Related UI Display Bugs in Android Apps [19.918077756884852]
  Android provides a wide range of system settings that allow users to control the runtime behaviors of apps, such as screen rotation and UI display.<n> setting-related bugs occur when developers do not fully align their apps with the extensive range of system settings that users can define.<n>These bugs can commonly affect apps' UI, causing setting-related UI display (SUD) bugs that negatively impact user experience.
  arXiv  Detail & Related papers  (2026-07-05T05:17:42Z)
• FIKA-Bench: From Fine-grained Recognition to Fine-Grained Knowledge Acquisition [54.31138496553705]
  Fine-grained recognition in everyday life is often not a closed-book classification problem.<n>Existing benchmarks primarily evaluate visually recognition, leaving this active external knowledge acquisition ability underexplored.<n>We study fine-grained knowledge acquisition, where a system must seek, verify, and use external evidence to answer open-ended fine-grained recognition questions.
  arXiv  Detail & Related papers  (2026-05-13T08:49:51Z)
• Broken Access: On the Challenges of Screen Reader Assisted Two-Factor and Passwordless Authentication [6.774193583753525]
  Currently deployed authentication mainly concentrates on sighted users, overlooking the needs of the blind and visually impaired community.<n>We model web authentication for such users as screen reader assisted authentication and introduce an evaluation framework called AWARE.<n>We analyzed these screen reader assisted authentication interactions with authentication methods in three settings.
  arXiv  Detail & Related papers  (2026-03-08T21:42:13Z)
• PaperAudit-Bench: Benchmarking Error Detection in Research Papers for Critical Automated Peer Review [54.141490756509306]
  We introduce PaperAudit-Bench, which consists of two components: PaperAudit-Dataset, an error dataset, and PaperAudit-Review, an automated review framework.<n>Experiments on PaperAudit-Bench reveal large variability in error detectability across models and detection depths.<n>We show that the dataset supports training lightweight LLM detectors via SFT and RL, enabling effective error detection at reduced computational cost.
  arXiv  Detail & Related papers  (2026-01-07T04:26:12Z)
• TaskAudit: Detecting Functiona11ity Errors in Mobile Apps via Agentic Task Execution [17.208420259998178]
  TaskAudit is an accessibility evaluation system that focuses on detecting functiona11ity errors through simulated interactions.<n> Evaluation on real-world apps shows that our strategy detects 48 functiona11ity errors from 54 app screens, compared to between 4 and 20 with existing checkers.
  arXiv  Detail & Related papers  (2025-10-14T20:28:49Z)
• Screencast-Based Analysis of User-Perceived GUI Responsiveness [53.53923672866705]
  tool is a technique that measures GUI responsiveness directly from mobile screencasts.<n>It uses computer vision to detect user interactions and analyzes frame-level visual changes to compute two key metrics.<n>tool has been deployed in an industrial testing pipeline and analyzes thousands of screencasts daily.
  arXiv  Detail & Related papers  (2025-08-02T12:13:50Z)
• Advancing Mobile UI Testing by Learning Screen Usage Semantics [0.42303492200814446]
  This research seeks to enhance automated UI testing techniques by learning the screen usage semantics of mobile apps.<n>It also improves the usability of a mobile app's interface by identifying and mitigating UI design issues.
  arXiv  Detail & Related papers  (2025-05-15T01:40:43Z)
• Are Users More Willing to Use Formally Verified Password Managers? [47.205801464292485]
  We design and implement two experiments to understand how formal verification impacts users. We focus on the application domain of password managers since it has been documented that the lack of trust in password managers might lead to lower adoption. We conclude that formal verification is seen as desirable by users and identify three actional recommendations to improve formal verification communication efforts.
  arXiv  Detail & Related papers  (2025-04-02T20:57:49Z)
• Forensics-Bench: A Comprehensive Forgery Detection Benchmark Suite for Large Vision Language Models [53.55128042938329]
  Forensics-Bench is a new forgery detection evaluation benchmark suite. It comprises 63,292 meticulously curated multi-choice visual questions, covering 112 unique forgery detection types. We conduct thorough evaluations on 22 open-sourced LVLMs and 3 proprietary models GPT-4o, Gemini 1.5 Pro, and Claude 3.5 Sonnet.
  arXiv  Detail & Related papers  (2025-03-19T09:21:44Z)
• On the (In)Security of Non-resettable Device Identifiers in Custom Android Systems [4.319029420721902]
  We present IDRadar, a scalable and accurate approach for identifying vulnerable properties and settings on custom Android ROMs.<n>We have identified 8,192 system properties and 3,620 settings that store non-resettable identifiers, with 3,477 properties and 1,336 settings lacking adequate access control.<n>Our large-scale analysis can identify a large number of security issues which are two orders of magnitude greater than existing techniques.
  arXiv  Detail & Related papers  (2025-02-21T07:52:46Z)
• OpenFactCheck: Building, Benchmarking Customized Fact-Checking Systems and Evaluating the Factuality of Claims and LLMs [59.836774258359945]
  OpenFactCheck is a framework for building customized automatic fact-checking systems.<n>It allows users to easily customize an automatic fact-checker and verify the factual correctness of documents and claims.<n>CheckerEVAL is a solution for gauging the reliability of automatic fact-checkers' verification results using human-annotated datasets.
  arXiv  Detail & Related papers  (2024-05-09T07:15:19Z)
• MotorEase: Automated Detection of Motor Impairment Accessibility Issues in Mobile App UIs [8.057618278428494]
  MotorEase is capable of identifying accessibility issues in mobile app UIs that impact motor-impaired users. It adapts computer vision and text processing techniques to enable a semantic understanding of app UI screens. It is able to identify violations with an average accuracy of 90%, and a false positive rate of less than 9%.
  arXiv  Detail & Related papers  (2024-03-20T15:53:07Z)
• GenAudit: Fixing Factual Errors in Language Model Outputs with Evidence [64.95492752484171]
  We present GenAudit -- a tool intended to assist fact-checking LLM responses for document-grounded tasks.<n>GenAudit suggests edits to the LLM response by revising or removing claims that are not supported by the reference document, and also presents evidence from the reference for facts that do appear to have support.<n> Comprehensive evaluation by human raters shows that GenAudit can detect errors in 8 different LLM outputs when summarizing documents from diverse domains.
  arXiv  Detail & Related papers  (2024-02-19T21:45:55Z)
• Towards Automated Accessibility Report Generation for Mobile Apps [14.908672785900832]
  We propose a system to generate whole app accessibility reports. It combines varied data collection methods (e.g., app crawling, manual recording) with an existing accessibility scanner.
  arXiv  Detail & Related papers  (2023-09-29T19:05:11Z)
• Understanding Mobile App Reviews to Guide Misuse Audits [17.71313286969027]
  We leverage app reviews to identify exploitable apps and their functionalities that enable misuse. Stories by abusers and victims mostly focus on past misuses, whereas stories by third parties mostly identify stories indicating the potential for misuse. In total, we confirmed 156 exploitable apps facilitating the misuse.
  arXiv  Detail & Related papers  (2023-03-19T23:43:36Z)
• Aurora Guard: Reliable Face Anti-Spoofing via Mobile Lighting System [103.5604680001633]
  Anti-spoofing against high-resolution rendering replay of paper photos or digital videos remains an open problem. We propose a simple yet effective face anti-spoofing system, termed Aurora Guard (AG)
  arXiv  Detail & Related papers  (2021-02-01T09:17:18Z)
• Emerging App Issue Identification via Online Joint Sentiment-Topic Tracing [66.57888248681303]
  We propose a novel emerging issue detection approach named MERIT. Based on the AOBST model, we infer the topics negatively reflected in user reviews for one app version. Experiments on popular apps from Google Play and Apple's App Store demonstrate the effectiveness of MERIT.
  arXiv  Detail & Related papers  (2020-08-23T06:34:05Z)
• Decentralized Privacy-Preserving Proximity Tracing [50.27258414960402]
  DP3T provides a technological foundation to help slow the spread of SARS-CoV-2. System aims to minimise privacy and security risks for individuals and communities.
  arXiv  Detail & Related papers  (2020-05-25T12:32:02Z)
• Unblind Your Apps: Predicting Natural-Language Labels for Mobile GUI Components by Deep Learning [21.56849865328527]
  More than 77% apps have issues of missing labels, according to our analysis of 10,408 Android apps. We develop a deep-learning based model, called LabelDroid, to automatically predict the labels of image-based buttons. The experimental results show that our model can make accurate predictions and the generated labels are of higher quality than that from real Android developers.
  arXiv  Detail & Related papers  (2020-03-01T02:31:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.