Who Controls the Conversation? User Perspectives On Generative AI (LLM) System Prompts

• URL: http://arxiv.org/abs/2603.00089v1
• Date: Mon, 16 Feb 2026 17:59:30 GMT
• Title: Who Controls the Conversation? User Perspectives On Generative AI (LLM) System Prompts
• Abstract summary: This paper argues that system prompts warrant explicit, user-centred design attention.<n>It asks: what do system prompts contain, how do end-users perceive them, and what do these perceptions offer for design and governance practice?<n>Our results reveal user perspectives on: the benefits and risks of system prompts; the values they prefer to be associated with prompt-design; their levels of comfort with different types of prompts; and degrees of transparency and user control regarding prompt content.
• Score: 7.340845393655051
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: System prompts - instructions that shape the behaviour of generative AI systems - strongly influence system outputs and users' experiences. They define the model's guidelines, `personality', and guardrails, taking precedence over user inputs. Despite their influence, transparency is limited: system prompts are generally not made public and most platforms instruct models to conceal them, leaving users disconnected from and unaware of a key mechanism guiding and governing their AI interactions. This paper argues that system prompts warrant explicit, user-centred design attention and, focusing on large language models (LLMs), asks: what do system prompts contain, how do end-users perceive them, and what do these perceptions offer for design and governance practice? Our results reveal user perspectives on: the benefits and risks of system prompts; the values they prefer to be associated with prompt-design; their levels of comfort with different types of prompts; and degrees of transparency and user control regarding prompt content. From these findings emerge considerations for how designers can better align system prompt mechanisms with user expectations and preferences over these mechanisms that directly shape how generative AI systems behave.

Related papers

• From Use to Oversight: How Mental Models Influence User Behavior and Output in AI Writing Assistants [70.1699066541501]
  We examine two types of mental models -- functional or related to what the system does, and structural or related to how the system works.<n>We find that while participants in the structural mental model condition demonstrate a better understanding of the system, this can have a backfiring effect.
  arXiv  Detail & Related papers  (2026-04-06T20:50:44Z)
• Visual or Textual: Effects of Explanation Format and Personal Characteristics on the Perception of Explanations in an Educational Recommender System [1.3125039212103755]
  We examine how explanation format and PCs jointly influence perceived control, transparency, trust, and satisfaction in an educational recommender system (ERS)<n>Our results show that a well-designed visual, simple, interactive, selective, easy to understand visualization that clearly and intuitively communicates how user preferences are linked to recommendations fosters perceived control, transparency, appropriate trust, and satisfaction in the ERS for most users, independent of their PCs.
  arXiv  Detail & Related papers  (2026-03-26T16:37:13Z)
• Just Ask: Curious Code Agents Reveal System Prompts in Frontier LLMs [65.6660735371212]
  We present textbftextscJustAsk, a framework that autonomously discovers effective extraction strategies through interaction alone.<n>It formulates extraction as an online exploration problem, using Upper Confidence Bound--based strategy selection and a hierarchical skill space spanning atomic probes and high-level orchestration.<n>Our results expose system prompts as a critical yet largely unprotected attack surface in modern agent systems.
  arXiv  Detail & Related papers  (2026-01-29T03:53:25Z)
• Interactive Reasoning: Visualizing and Controlling Chain-of-Thought Reasoning in Large Language Models [54.85405423240165]
  We introduce Interactive Reasoning, an interaction design that visualizes chain-of-thought outputs as a hierarchy of topics.<n>We implement interactive reasoning in Hippo, a prototype for AI-assisted decision making in the face of uncertain trade-offs.
  arXiv  Detail & Related papers  (2025-06-30T10:00:43Z)
• Position is Power: System Prompts as a Mechanism of Bias in Large Language Models (LLMs) [7.71667852309443]
  System prompts in Large Language Models (LLMs) are predefined directives that guide model behaviour.<n>LLMs deployers increasingly use them to ensure consistent responses across contexts.<n>As system prompts become more complex, they can directly or indirectly introduce unaccounted for side effects.
  arXiv  Detail & Related papers  (2025-05-27T12:19:08Z)
• Creating General User Models from Computer Use [53.59999173952482]
  This paper presents an architecture for a general user model (GUM) that learns about you by observing any interaction you have with your computer.<n>The GUM takes as input any unstructured observation of a user (e.g., device screenshots) and constructs confidence-weighted propositions that capture user knowledge and preferences.
  arXiv  Detail & Related papers  (2025-05-16T04:00:31Z)
• Exploring the Impact of Personality Traits on Conversational Recommender Systems: A Simulation with Large Language Models [70.180385882195]
  This paper introduces a personality-aware user simulation for Conversational Recommender Systems (CRSs)<n>The user agent induces customizable personality traits and preferences, while the system agent possesses the persuasion capability to simulate realistic interaction in CRSs.<n> Experimental results demonstrate that state-of-the-art LLMs can effectively generate diverse user responses aligned with specified personality traits.
  arXiv  Detail & Related papers  (2025-04-09T13:21:17Z)
• AI Automatons: AI Systems Intended to Imitate Humans [54.19152688545896]
  There is a growing proliferation of AI systems designed to mimic people's behavior, work, abilities, likenesses, or humanness.<n>The research, design, deployment, and availability of such AI systems have prompted growing concerns about a wide range of possible legal, ethical, and other social impacts.
  arXiv  Detail & Related papers  (2025-03-04T03:55:38Z)
• Dynamic Prompt Middleware: Contextual Prompt Refinement Controls for Comprehension Tasks [15.808343159772066]
  This study investigates user needs for control over AI-generated explanations in comprehension tasks.<n>We implement two prompt approaches: Dynamic Prompt Refinement Control (Dynamic PRC) and Static Prompt Refinement Control (Static PRC)<n>We evaluate the impact of these approaches on user control of AI responses for crafting better explanations.
  arXiv  Detail & Related papers  (2024-12-03T10:27:04Z)
• The Illusion of Competence: Evaluating the Effect of Explanations on Users' Mental Models of Visual Question Answering Systems [6.307898834231964]
  We examine how users perceive the limitations of an AI system when it encounters a task that it cannot perform perfectly. We employ a visual question answer and explanation task where we control the AI system's limitations by manipulating the visual inputs. Our goal is to determine whether participants can perceive the limitations of the system.
  arXiv  Detail & Related papers  (2024-06-27T13:44:03Z)
• System-2 Recommenders: Disentangling Utility and Engagement in Recommendation Systems via Temporal Point-Processes [80.97898201876592]
  We propose a generative model in which past content interactions impact the arrival rates of users based on a self-exciting Hawkes process. We show analytically that given samples it is possible to disentangle System-1 and System-2 and allow content optimization based on user utility.
  arXiv  Detail & Related papers  (2024-05-29T18:19:37Z)
• User-Controllable Recommendation via Counterfactual Retrospective and Prospective Explanations [96.45414741693119]
  We present a user-controllable recommender system that seamlessly integrates explainability and controllability. By providing both retrospective and prospective explanations through counterfactual reasoning, users can customize their control over the system.
  arXiv  Detail & Related papers  (2023-08-02T01:13:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.