From Use to Oversight: How Mental Models Influence User Behavior and Output in AI Writing Assistants

• URL: http://arxiv.org/abs/2604.05166v1
• Date: Mon, 06 Apr 2026 20:50:44 GMT
• Title: From Use to Oversight: How Mental Models Influence User Behavior and Output in AI Writing Assistants
• Abstract summary: We examine two types of mental models -- functional or related to what the system does, and structural or related to how the system works.<n>We find that while participants in the structural mental model condition demonstrate a better understanding of the system, this can have a backfiring effect.
• Score: 70.1699066541501
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: AI-based writing assistants are ubiquitous, yet little is known about how users' mental models shape their use. We examine two types of mental models -- functional or related to what the system does, and structural or related to how the system works -- and how they affect control behavior -- how users request, accept, or edit AI suggestions as they write -- and writing outcomes. We primed participants ($N = 48$) with different system descriptions to induce these mental models before asking them to complete a cover letter writing task using a writing assistant that occasionally offered preconfigured ungrammatical suggestions to test whether the mental models affected participants' critical oversight. We find that while participants in the structural mental model condition demonstrate a better understanding of the system, this can have a backfiring effect: while these participants judged the system as more usable, they also produced letters with more grammatical errors, highlighting a complex relationship between system understanding, trust, and control in contexts that require user oversight of error-prone AI outputs.

Related papers

• Mind2Dialogue: Training Human-Aware Language Models by Simulating User Mental States [72.77158276658636]
  Mind2Dialogue is a framework for simulating users' mental states and turning them into privileged supervision for human-aware training.<n>We first propose a psychology-guided simulator that preserves personal characteristics while updating mental states to generate coherent conversations.<n>Our privileged distillation then trains models on the Oracle's well-informed responses to assist users without direct access to their mental states at deployment.
  arXiv  Detail & Related papers  (2026-09-14T17:55:58Z)
• Who Controls the Conversation? User Perspectives On Generative AI (LLM) System Prompts [7.340845393655051]
  This paper argues that system prompts warrant explicit, user-centred design attention.<n>It asks: what do system prompts contain, how do end-users perceive them, and what do these perceptions offer for design and governance practice?<n>Our results reveal user perspectives on: the benefits and risks of system prompts; the values they prefer to be associated with prompt-design; their levels of comfort with different types of prompts; and degrees of transparency and user control regarding prompt content.
  arXiv  Detail & Related papers  (2026-02-16T17:59:30Z)
• Automated Feedback Generation for Undergraduate Mathematics: Development and Evaluation of an AI Teaching Assistant [0.0]
  We present a system that processes free-form natural language input, handles a wide range of edge cases, and comments on the technical correctness of submitted proofs.<n>We show that by the metrics we evaluate, the quality of the feedback generated is comparable to that produced by human experts.<n>A version of our tool is deployed on the Imperial mathematics homework platform Lambda.
  arXiv  Detail & Related papers  (2026-01-06T23:02:22Z)
• Annotating Errors in English Learners' Written Language Production: Advancing Automated Written Feedback Systems [25.047364950581265]
  We introduce an annotation framework that models each error's error type and generalizability.<n>We collect a dataset of annotated learner errors and corresponding human-written feedback comments.<n>We evaluate keyword-guided, keyword-free, and template-guided methods of generating feedback.
  arXiv  Detail & Related papers  (2025-08-09T04:06:18Z)
• Interactive Reasoning: Visualizing and Controlling Chain-of-Thought Reasoning in Large Language Models [54.85405423240165]
  We introduce Interactive Reasoning, an interaction design that visualizes chain-of-thought outputs as a hierarchy of topics.<n>We implement interactive reasoning in Hippo, a prototype for AI-assisted decision making in the face of uncertain trade-offs.
  arXiv  Detail & Related papers  (2025-06-30T10:00:43Z)
• Understanding Learner-LLM Chatbot Interactions and the Impact of Prompting Guidelines [9.834055425277874]
  This study investigates learner-AI interactions through an educational experiment in which participants receive structured guidance on effective prompting.<n>To assess user behavior and prompting efficacy, we analyze a dataset of 642 interactions from 107 users.<n>Our findings provide a deeper understanding of how users engage with Large Language Models and the role of structured prompting guidance in enhancing AI-assisted communication.
  arXiv  Detail & Related papers  (2025-04-10T15:20:43Z)
• Towards Large-Scale Interpretable Knowledge Graph Reasoning for Dialogue Systems [109.16553492049441]
  We propose a novel method to incorporate the knowledge reasoning capability into dialogue systems in a more scalable and generalizable manner. To the best of our knowledge, this is the first work to have transformer models generate responses by reasoning over differentiable knowledge graphs.
  arXiv  Detail & Related papers  (2022-03-20T17:51:49Z)
• AES Systems Are Both Overstable And Oversensitive: Explaining Why And Proposing Defenses [66.49753193098356]
  We investigate the reason behind the surprising adversarial brittleness of scoring models. Our results indicate that autoscoring models, despite getting trained as "end-to-end" models, behave like bag-of-words models. We propose detection-based protection models that can detect oversensitivity and overstability causing samples with high accuracies.
  arXiv  Detail & Related papers  (2021-09-24T03:49:38Z)
• Learning User-Interpretable Descriptions of Black-Box AI System Capabilities [9.608555640607731]
  This paper presents an approach for learning user-interpretable symbolic descriptions of the limits and capabilities of a black-box AI system. It uses a hierarchical active querying paradigm to generate questions and to learn a user-interpretable model of the AI system based on its responses.
  arXiv  Detail & Related papers  (2021-07-28T23:33:31Z)
• A general framework for scientifically inspired explanations in AI [76.48625630211943]
  We instantiate the concept of structure of scientific explanation as the theoretical underpinning for a general framework in which explanations for AI systems can be implemented. This framework aims to provide the tools to build a "mental-model" of any AI system so that the interaction with the user can provide information on demand and be closer to the nature of human-made explanations.
  arXiv  Detail & Related papers  (2020-03-02T10:32:21Z)
• A Neural Topical Expansion Framework for Unstructured Persona-oriented Dialogue Generation [52.743311026230714]
  Persona Exploration and Exploitation (PEE) is able to extend the predefined user persona description with semantically correlated content. PEE consists of two main modules: persona exploration and persona exploitation. Our approach outperforms state-of-the-art baselines in terms of both automatic and human evaluations.
  arXiv  Detail & Related papers  (2020-02-06T08:24:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.