Say It My Way: Exploring Control in Conversational Visual Question Answering with Blind Users

• URL: http://arxiv.org/abs/2602.16930v1
• Date: Wed, 18 Feb 2026 22:44:43 GMT
• Title: Say It My Way: Exploring Control in Conversational Visual Question Answering with Blind Users
• Abstract summary: We show how users can customize visual question answering (VQA) tools with a real-world conversational VQA system.<n>The system lacked verbosity controls, was limited in estimating distance in space and time, relied on inaccessible image framing, and offered little to no camera guidance.<n>We discuss how customization techniques such as prompt engineering can help participants work around these limitations.
• Score: 8.54162000547885
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Prompting and steering techniques are well established in general-purpose generative AI, yet assistive visual question answering (VQA) tools for blind users still follow rigid interaction patterns with limited opportunities for customization. User control can be helpful when system responses are misaligned with their goals and contexts, a gap that becomes especially consequential for blind users that may rely on these systems for access. We invite 11 blind users to customize their interactions with a real-world conversational VQA system. Drawing on 418 interactions, reflections, and post-study interviews, we analyze prompting-based techniques participants adopted, including those introduced in the study and those developed independently in real-world settings. VQA interactions were often lengthy: participants averaged 3 turns, sometimes up to 21, with input text typically tenfold shorter than the responses they heard. Built on state-of-the-art LLMs, the system lacked verbosity controls, was limited in estimating distance in space and time, relied on inaccessible image framing, and offered little to no camera guidance. We discuss how customization techniques such as prompt engineering can help participants work around these limitations. Alongside a new publicly available dataset, we offer insights for interaction design at both query and system levels.

Related papers

• Socrates went Nuclear: Comparing Interaction Strategies for AI systems in a Learning Context using Brain Sensing [66.03354630509942]
  This paper reports on a study where we compare three designs for user-AI interaction in a learning context.<n>Fifty study participants were tasked with learning about nuclear safety protocols.
  arXiv  Detail & Related papers  (2026-09-01T02:24:35Z)
• HARP: The Human--AI Research Platform [0.9332987715848716]
  Large language models (LLMs) have shifted human--computer interaction from traditional'' interface journeys toward more conversational exchanges.<n>HarP places participants in controlled mock scenarios with live, AI agents.<n>HarP enables systematic testing of how AI design choices affect users.
  arXiv  Detail & Related papers  (2026-07-22T22:38:16Z)
• Sketch-based Access Control: A Multimodal Interface for Translating User Preferences into Intent-Aligned Policies [9.431045721676204]
  We present Sketch-based Access Control (SBAC), a sketch-based, AI-assisted access control authoring system.<n>SBAC combines the expressive power of sketching with the interpretive capabilities of multimodal large language models (MLLMs) to support the interpretation and validation of policy specifications.
  arXiv  Detail & Related papers  (2026-05-11T05:39:40Z)
• From Use to Oversight: How Mental Models Influence User Behavior and Output in AI Writing Assistants [70.1699066541501]
  We examine two types of mental models -- functional or related to what the system does, and structural or related to how the system works.<n>We find that while participants in the structural mental model condition demonstrate a better understanding of the system, this can have a backfiring effect.
  arXiv  Detail & Related papers  (2026-04-06T20:50:44Z)
• HumAIne-Chatbot: Real-Time Personalized Conversational AI via Reinforcement Learning [0.4931504898146351]
  textbfHumAIne-chatbot is an AI-driven conversational agent that personalizes responses through a novel user profiling framework.<n>During live interactions, an online reinforcement learning agent refines per-user models by combining implicit signals.<n>Results show consistent improvements in user satisfaction, personalization accuracy, and task achievement when personalization features were enabled.
  arXiv  Detail & Related papers  (2025-09-04T15:16:38Z)
• User Feedback in Human-LLM Dialogues: A Lens to Understand Users But Noisy as a Learning Signal [59.120335322495436]
  We analyze user feedback in the user-LLM conversation logs, providing insights into when and why such feedback occurs.<n>Second, we study harvesting learning signals from such implicit user feedback.
  arXiv  Detail & Related papers  (2025-07-30T23:33:29Z)
• Creating General User Models from Computer Use [53.59999173952482]
  This paper presents an architecture for a general user model (GUM) that learns about you by observing any interaction you have with your computer.<n>The GUM takes as input any unstructured observation of a user (e.g., device screenshots) and constructs confidence-weighted propositions that capture user knowledge and preferences.
  arXiv  Detail & Related papers  (2025-05-16T04:00:31Z)
• Trapped by Expectations: Functional Fixedness in LLM-Enabled Chat Search [9.166043188084414]
  We investigated the impact of functional fixedness on large language models (LLM)-enabled chat search.<n>We found pre-chat expectations shape language use and prompting behavior.<n>With appropriate system support, this may promote broader exploration of LLM capabilities.
  arXiv  Detail & Related papers  (2025-04-02T19:14:01Z)
• Evaluating Multimodal Language Models as Visual Assistants for Visually Impaired Users [42.132487737233845]
  This paper explores the effectiveness of Multimodal Large Language models (MLLMs) as assistive technologies for visually impaired individuals.<n>We conduct a user survey to identify adoption patterns and key challenges users face with such technologies.
  arXiv  Detail & Related papers  (2025-03-28T16:54:25Z)
• AI-based Wearable Vision Assistance System for the Visually Impaired: Integrating Real-Time Object Recognition and Contextual Understanding Using Large Vision-Language Models [0.0]
  This paper introduces a novel wearable vision assistance system with artificial intelligence (AI) technology to deliver real-time feedback to a user through a sound beep mechanism.<n>The system provides detailed descriptions of objects in the user's environment using a large vision language model (LVLM)
  arXiv  Detail & Related papers  (2024-12-28T07:26:39Z)
• Detecting Agreement in Multi-party Conversational AI [2.755416211770089]
  We present our work on a multi-party conversational system, which invites two users to play a trivia quiz game. The system detects users' agreement or disagreement on a final answer and responds accordingly. Our annotated transcripts and the code for the proposed system have been released open-source on GitHub.
  arXiv  Detail & Related papers  (2023-11-06T11:04:39Z)
• Are Akpans Trick or Treat: Unveiling Helpful Biases in Assistant Systems [55.09907990139756]
  Information-seeking AI assistant systems aim to answer users' queries about knowledge in a timely manner.<n>In this paper, we study computational measurements of helpfulness.<n> Experiments with state-of-the-art dialogue systems reveal that existing systems tend to be more helpful for questions regarding concepts from highly-developed countries.
  arXiv  Detail & Related papers  (2022-05-25T07:58:38Z)
• ASHA: Assistive Teleoperation via Human-in-the-Loop Reinforcement Learning [91.58711082348293]
  Reinforcement learning from online user feedback on the system's performance presents a natural solution to this problem. This approach tends to require a large amount of human-in-the-loop training data, especially when feedback is sparse. We propose a hierarchical solution that learns efficiently from sparse user feedback.
  arXiv  Detail & Related papers  (2022-02-05T02:01:19Z)
• Improving Conversational Question Answering Systems after Deployment using Feedback-Weighted Learning [69.42679922160684]
  We propose feedback-weighted learning based on importance sampling to improve upon an initial supervised system using binary user feedback. Our work opens the prospect to exploit interactions with real users and improve conversational systems after deployment.
  arXiv  Detail & Related papers  (2020-11-01T19:50:34Z)
• ConvAI3: Generating Clarifying Questions for Open-Domain Dialogue Systems (ClariQ) [64.60303062063663]
  This document presents a detailed description of the challenge on clarifying questions for dialogue systems (ClariQ) The challenge is organized as part of the Conversational AI challenge series (ConvAI3) at Search Oriented Conversational AI (SCAI) EMNLP workshop in 2020.
  arXiv  Detail & Related papers  (2020-09-23T19:48:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.