Querying Multimodal Scientific Papers with AI: Practices and Preferences Across Blind, Low-Vision, and Sighted Scientists

• URL: http://arxiv.org/abs/2607.18514v1
• Date: Mon, 20 Jul 2026 21:14:26 GMT
• Title: Querying Multimodal Scientific Papers with AI: Practices and Preferences Across Blind, Low-Vision, and Sighted Scientists
• Abstract summary: Blind or low-vision (BLV) scientists have traditionally relied on static alternative text to access figures in papers.<n>The rise of artificial intelligence (AI) has made interactive question-answering (QA) a feasible paradigm for visual exploration.<n>We interview five BLV and five sighted scientists across different STEM fields to understand how they use two AI tools, ChatGPT and Gemini, to query multimodal scientific documents.
• Score: 19.90873686121111
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Visual diagrams, figures, and tables are central to scientific papers, and convey information beyond what is captured in text. While blind or low-vision (BLV) scientists have traditionally relied on static alternative text to access figures in papers, the rise of artificial intelligence (AI) has made interactive question-answering (QA) a feasible paradigm for visual exploration; yet little is known about how scientists use visual QA in practice or how to improve its accessibility. In this work, we interview five BLV and five sighted scientists across different STEM fields to understand how they use two AI tools, ChatGPT and Gemini, to query multimodal scientific documents. Our findings characterize how scientists review multimodal content, including existing practices (along with accessibility workarounds) for engaging with visuals, and feedback on the suitability of AI-generated responses to multimodal queries. We further find that vague or incomplete image descriptions, as well as incorrect AI outputs more broadly, can cause both BLV and sighted scientists to abandon AI workflows. To support future research, we additionally contribute a dataset of 115 queries and responses from our participants' interactions with the AI tools for papers in their field. We close by discussing implications for AI-powered scientific QA systems, emphasizing considerations for access across abilities and domains.

Related papers

• Explainable AI for Blind and Low-Vision Users: Navigating Trust, Modality, and Interpretability in the Agentic Era [0.9741676285072485]
  For blind and low-vision (BLV) users, the lack of accessible explanations creates a fundamental barrier to the independent use of AI-driven assistive technologies.<n>This paper investigates the unique XAI requirements of the BLV community through a comprehensive analysis of user interviews and contemporary research.<n> Empirical evidence suggests that while BLV users highly value conversational explanations, they frequently experience "self-blame" for AI failures.
  arXiv  Detail & Related papers  (2026-03-31T19:39:52Z)
• Research Paradigm of Materials Science Tetrahedra with Artificial Intelligence [7.779425241736052]
  We propose two new research paradigms to stimulate data-driven and AI-augmented research.<n>One focuses on AI for materials science by considering the Matter-Data-PotentialAgent diagram.<n>The other demonstrates AI research by discussing Data-Architecture-Inference-Inference relationships.
  arXiv  Detail & Related papers  (2026-03-14T04:30:03Z)
• Accelerating Scientific Research with Gemini: Case Studies and Common Techniques [105.15622072347811]
  Large language models (LLMs) have opened new avenues for accelerating scientific research.<n>We present a collection of case studies demonstrating how researchers have successfully collaborated with advanced AI models.
  arXiv  Detail & Related papers  (2026-02-03T18:56:17Z)
• AI4X Roadmap: Artificial Intelligence for the advancement of scientific pursuit and its future directions [65.44445343399126]
  We look at AI-enabled science across biology, chemistry, climate science, mathematics, materials science, physics, self-driving laboratories and unconventional computing.<n>Several shared themes emerge: the need for diverse and trustworthy data, transferable electronic-structure and interatomic models, AI systems integrated into end-to-end scientific synthesis.<n>Across domains, we highlight how large foundation models, active learning and self-driving laboratories can close loops between prediction and validation.
  arXiv  Detail & Related papers  (2025-11-26T02:10:28Z)
• Advancing AI Research Assistants with Expert-Involved Learning [84.30323604785646]
  Large language models (LLMs) and large multimodal models (LMMs) promise to accelerate biomedical discovery, yet their reliability remains unclear.<n>We introduce ARIEL (AI Research Assistant for Expert-in-the-Loop Learning), an open-source evaluation and optimization framework.<n>We find that state-of-the-art models generate fluent but incomplete summaries, whereas LMMs struggle with detailed visual reasoning.
  arXiv  Detail & Related papers  (2025-05-03T14:21:48Z)
• AI-in-the-loop: The future of biomedical visual analytics applications in the era of AI [3.0942901747200975]
  How will massive developments of AI in data analytics shape future data visualizations and visual analytics?<n>What are opportunities, open challenges and threats in the context of an increasingly powerful AI?<n>We highlight the potential of AI to transform biomedical visualization as a research field.
  arXiv  Detail & Related papers  (2024-12-20T13:27:24Z)
• SPIQA: A Dataset for Multimodal Question Answering on Scientific Papers [43.18330795060871]
  SPIQA is a dataset specifically designed to interpret complex figures and tables within the context of scientific research articles.<n>We employ automatic and manual curation to create the dataset.<n> SPIQA comprises 270K questions divided into training, validation, and three different evaluation splits.
  arXiv  Detail & Related papers  (2024-07-12T16:37:59Z)
• Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AI [116.8199519880327]
  Embodied Artificial Intelligence (Embodied AI) is crucial for achieving Artificial General Intelligence (AGI)<n>In this survey, we give a comprehensive exploration of the latest advancements in Embodied AI.
  arXiv  Detail & Related papers  (2024-07-09T14:14:47Z)
• MMSci: A Dataset for Graduate-Level Multi-Discipline Multimodal Scientific Understanding [59.41495657570397]
  We present a comprehensive dataset compiled from Nature Communications articles covering 72 scientific fields.<n>We evaluated 19 proprietary and open-source models on two benchmark tasks, figure captioning and multiple-choice, and conducted human expert annotation.<n>Fine-tuning Qwen2-VL-7B with our task-specific data achieved better performance than GPT-4o and even human experts in multiple-choice evaluations.
  arXiv  Detail & Related papers  (2024-07-06T00:40:53Z)
• Virtual Reality for Understanding Artificial-Intelligence-driven Scientific Discovery with an Application in Quantum Optics [1.0858565995100633]
  We show how transferring part of the analysis process into an immersive Virtual Reality environment can assist researchers in developing an understanding of AI-generated solutions. We demonstrate the usefulness of VR in finding interpretable configurations of abstract graphs, representing Quantum Optics experiments.
  arXiv  Detail & Related papers  (2024-02-20T17:48:01Z)
• Multimodal Lecture Presentations Dataset: Understanding Multimodality in Educational Slides [57.86931911522967]
  We test the capabilities of machine learning models in multimodal understanding of educational content. Our dataset contains aligned slides and spoken language, for 180+ hours of video and 9000+ slides, with 10 lecturers from various subjects. We introduce PolyViLT, a multimodal transformer trained with a multi-instance learning loss that is more effective than current approaches.
  arXiv  Detail & Related papers  (2022-08-17T05:30:18Z)
• Empowering Things with Intelligence: A Survey of the Progress, Challenges, and Opportunities in Artificial Intelligence of Things [98.10037444792444]
  We show how AI can empower the IoT to make it faster, smarter, greener, and safer. First, we present progress in AI research for IoT from four perspectives: perceiving, learning, reasoning, and behaving. Finally, we summarize some promising applications of AIoT that are likely to profoundly reshape our world.
  arXiv  Detail & Related papers  (2020-11-17T13:14:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.