A Systematic Survey on Image Description Techniques for STEM Domains

• URL: http://arxiv.org/abs/2607.21611v1
• Date: Fri, 22 May 2026 10:06:11 GMT
• Title: A Systematic Survey on Image Description Techniques for STEM Domains
• Abstract summary: This systematic survey examines 20 peer-reviewed studies on AI-based techniques for describing STEM visuals.<n>The analysis reveals a shift from static, one-shot alt text toward interactive and multimodal systems that integrate conversational interfaces, keyboard navigation, and audio or haptic feedback.
• Score: 0.46603287532620746
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The proliferation of visual data in Science, Technology, Engineering, and Mathematics (STEM) fields presents accessibility barrier for individuals with blindness or visual impairments. While recent advances in Artificial Intelligence (AI) offer new opportunities to generate textual descriptions of STEM images, the research landscape is fragmented and its impact on real users remains limited. This systematic survey examines 20 peer-reviewed studies on AI-based techniques for describing STEM visuals, with a specific focus on accessibility and human-computer interaction. Following the PRISMA methodology and a ROBIS-based risk-of-bias assessment, the review analyzes (i) the types of STEM visuals targeted, (ii) the AI and machine learning architectures employed, (iii) the datasets and evaluation metrics adopted, and (iv) the interaction modalities through which descriptions are delivered. The analysis reveals a shift from static, one-shot alt text toward interactive and multimodal systems that integrate conversational interfaces, keyboard navigation, and audio or haptic feedback. However, critical challenges persist, including factual inaccuracies and hallucinations, the scarcity of accessibility-first datasets co-designed with blind and low-vision users, and a heavy reliance on automatic text-overlap metrics that poorly capture perceived usefulness and trust. The survey concludes by outlining key research directions for HCI, emphasizing user-controlled verbosity, explainable and verifiable AI pipelines, and the integration of accessible description tools into mainstream STEM authoring and learning environments.

Related papers

• Designing Gaze Analytics for ELA Instruction: A User-Centered Dashboard with Conversational AI Support [3.741199946315248]
  Eye-tracking offers rich insights into student cognition and engagement.<n>However, it remains underutilized in classroom-facing educational technology.<n>We present the iterative design and evaluation of a gaze-based learning analytics dashboard for English Language Arts.
  arXiv  Detail & Related papers  (2025-09-03T22:01:14Z)
• Hyperspectral Imaging [49.45523645429475]
  Hyperspectral imaging (HSI) is an advanced sensing modality that simultaneously captures spatial and spectral information.<n>This Primer presents a comprehensive overview of HSI, from the underlying physical principles and sensor architectures to key steps in data acquisition, calibration, and correction.
  arXiv  Detail & Related papers  (2025-08-11T15:47:24Z)
• A Survey on (M)LLM-Based GUI Agents [62.57899977018417]
  Graphical User Interface (GUI) Agents have emerged as a transformative paradigm in human-computer interaction.<n>Recent advances in large language models and multimodal learning have revolutionized GUI automation across desktop, mobile, and web platforms.<n>This survey identifies key technical challenges, including accurate element localization, effective knowledge retrieval, long-horizon planning, and safety-aware execution control.
  arXiv  Detail & Related papers  (2025-03-27T17:58:31Z)
• Vision Technologies with Applications in Traffic Surveillance Systems: A Holistic Survey [18.05238725149463]
  Traffic Surveillance Systems (TSS) have become increasingly crucial in modern intelligent transportation systems.<n>This paper presents a systematic review of vision technologies in TSS.<n>We examine both low-level and high-level perception tasks.
  arXiv  Detail & Related papers  (2024-11-30T04:17:56Z)
• Automated Assessment of Multimodal Answer Sheets in the STEM domain [0.3958317527488535]
  This research endeavors to develop efficient and reliable grading methods through the implementation of automated,assessment techniques using Artificial Intelligence (AI) Our,contributions lie in two key areas: firstly, the development of a,robust system for evaluating textual answers in STEM, leveraging,sample answers for precise comparison and grading, enabled by, advanced algorithms and natural language processing techniques. By bridging the gap between,visual representation and semantic meaning, our approach ensures accurate evaluation while minimizing manual intervention.
  arXiv  Detail & Related papers  (2024-09-24T05:10:13Z)
• How Human-Centered Explainable AI Interface Are Designed and Evaluated: A Systematic Survey [48.97104365617498]
  The emerging area of em Explainable Interfaces (EIs) focuses on the user interface and user experience design aspects of XAI. This paper presents a systematic survey of 53 publications to identify current trends in human-XAI interaction and promising directions for EI design and development.
  arXiv  Detail & Related papers  (2024-03-21T15:44:56Z)
• Bias and Fairness in Large Language Models: A Survey [73.87651986156006]
  We present a comprehensive survey of bias evaluation and mitigation techniques for large language models (LLMs) We first consolidate, formalize, and expand notions of social bias and fairness in natural language processing. We then unify the literature by proposing three intuitive, two for bias evaluation, and one for mitigation.
  arXiv  Detail & Related papers  (2023-09-02T00:32:55Z)
• Re-mine, Learn and Reason: Exploring the Cross-modal Semantic Correlations for Language-guided HOI detection [57.13665112065285]
  Human-Object Interaction (HOI) detection is a challenging computer vision task. We present a framework that enhances HOI detection by incorporating structured text knowledge.
  arXiv  Detail & Related papers  (2023-07-25T14:20:52Z)
• Vision+X: A Survey on Multimodal Learning in the Light of Data [64.03266872103835]
  multimodal machine learning that incorporates data from various sources has become an increasingly popular research area. We analyze the commonness and uniqueness of each data format mainly ranging from vision, audio, text, and motions. We investigate the existing literature on multimodal learning from both the representation learning and downstream application levels.
  arXiv  Detail & Related papers  (2022-10-05T13:14:57Z)
• Beyond Visuals : Examining the Experiences of Geoscience Professionals With Vision Disabilities in Accessing Data Visualizations [0.0]
  This study seeks to understand the experiences of professionals who are blind/vision impaired in one such STEM discipline (geosciences) in accessing data visualizations. A reflexive thematic analysis revealed the negative impact of visualizations in influencing their career path, lack of data exploration tools for research, barriers in accessing works of peers and mismatched pace of visualization and accessibility research.
  arXiv  Detail & Related papers  (2022-07-27T00:07:44Z)
• Empathetic Conversational Systems: A Review of Current Advances, Gaps, and Opportunities [2.741266294612776]
  A growing number of studies have recognized the benefits of empathy and started to incorporate empathy in conversational systems. This paper examines this rapidly growing field using five review dimensions.
  arXiv  Detail & Related papers  (2022-05-09T05:19:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.