Who Does Your AI Work For? Designing Conversational Agents as Digital Fiduciaries

• URL: http://arxiv.org/abs/2605.28908v1
• Date: Wed, 27 May 2026 17:07:02 GMT
• Title: Who Does Your AI Work For? Designing Conversational Agents as Digital Fiduciaries
• Abstract summary: Much of the literature on AI systems has focused on aligning users' goals with the agents that act on their behalf.<n>This provocation argues that conversational agents should be held to a similar standard and introduces fiduciary design as a guiding principle.
• Score: 2.538209532048867
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Conversational agents are increasingly integrated into the most private and intimate aspects of users' lives, from discussions of mental health to financial decisions. As a result, these systems have access to reams of sensitive user data. Much of the literature on AI systems has focused on aligning users' goals with the agents that act on their behalf. While this work is vitally important, it may overlook the need to establish a new normative baseline. Conversational AI agents, designed to feel and interact anthropomorphically with human users, must be held to a standard of care commensurate with their capabilities and access. When a client hires a personal lawyer, undergoes surgery, or receives advice from an investment manager, the expert they consult often has a fiduciary duty to act in their client's best interests. This provocation argues that conversational agents should be held to a similar standard and introduces fiduciary design as a guiding principle. In this respect, conversational AI trust and accountability could be unified into a single design and legal paradigm.

Related papers

• Trust by Design: Trust Calibration Through Non-Advisory Socratic Dialogue in Conversational Agents [0.0]
  We present CASELy, a conversational agent designed to limit its own authority through non-advisory Socratic dialogue.<n>The agent asks reflective questions grounded exclusively in user input and refuses to provide advice, recommendations, or interpretations.
  arXiv  Detail & Related papers  (2026-09-13T22:19:18Z)
• Designing AI for Real Users -- Accessibility Gaps in Retail AI Front-End [0.0]
  We argue that many AI front-ends implicitly assume an 'ideal user body and mind'<n>This becomes visible and ethically consequential when examined through the experiences of differently abled users.<n>We propose front-end assurance as a practical complement to AI governance.
  arXiv  Detail & Related papers  (2026-03-30T09:04:17Z)
• Embracing Trustworthy Brain-Agent Collaboration as Paradigm Extension for Intelligent Assistive Technologies [51.93721053301417]
  This paper argues that the field is poised for a paradigm extension from Brain-Computer Interfaces to Brain-Agent Collaboration.<n>We emphasize reframing agents as active and collaborative partners for intelligent assistance rather than passive brain signal data processors.
  arXiv  Detail & Related papers  (2025-10-25T00:25:45Z)
• Navigating Privacy and Trust: AI Assistants as Social Support for Older Adults [1.6257069916752018]
  Older adults must navigate trade-offs between usability, data privacy, and personal agency.<n>This paper advocates for participatory design approaches that position older adults as active decision makers in shaping AI assistant functionality.
  arXiv  Detail & Related papers  (2025-05-05T19:00:14Z)
• A Desideratum for Conversational Agents: Capabilities, Challenges, and Future Directions [51.96890647837277]
  Large Language Models (LLMs) have propelled conversational AI from traditional dialogue systems into sophisticated agents capable of autonomous actions, contextual awareness, and multi-turn interactions with users.<n>This survey paper presents a desideratum for next-generation Conversational Agents - what has been achieved, what challenges persist, and what must be done for more scalable systems that approach human-level intelligence.
  arXiv  Detail & Related papers  (2025-04-07T21:01:25Z)
• Interactive Dialogue Agents via Reinforcement Learning on Hindsight Regenerations [58.65755268815283]
  Many real dialogues are interactive, meaning an agent's utterances will influence their conversational partner, elicit information, or change their opinion. We use this fact to rewrite and augment existing suboptimal data, and train via offline reinforcement learning (RL) an agent that outperforms both prompting and learning from unaltered human demonstrations. Our results in a user study with real humans show that our approach greatly outperforms existing state-of-the-art dialogue agents.
  arXiv  Detail & Related papers  (2024-11-07T21:37:51Z)
• Designing AI Personalities: Enhancing Human-Agent Interaction Through Thoughtful Persona Design [7.610735476681428]
  This workshop aims to establish a research community focused on AI agent persona design for various contexts. We will explore critical aspects of persona design, such as voice, embodiment, and demographics, and their impact on user satisfaction and engagement. Topics include the design of conversational interfaces, the influence of agent personas on user experience, and approaches for creating contextually appropriate AI agents.
  arXiv  Detail & Related papers  (2024-10-30T06:58:59Z)
• The Ethics of Advanced AI Assistants [53.89899371095332]
  This paper focuses on the opportunities and the ethical and societal risks posed by advanced AI assistants. We define advanced AI assistants as artificial agents with natural language interfaces, whose function is to plan and execute sequences of actions on behalf of a user. We consider the deployment of advanced assistants at a societal scale, focusing on cooperation, equity and access, misinformation, economic impact, the environment and how best to evaluate advanced AI assistants.
  arXiv  Detail & Related papers  (2024-04-24T23:18:46Z)
• Designing Fiduciary Artificial Intelligence [0.0]
  This article synthesizes recent work in computer science and law to develop a procedure for designing and auditing Fiduciary AI. The designer of a Fiduciary AI should understand the context of the system, identify its principals, and assess the best interests of those principals. We connect the steps in this procedure to dimensions of Trustworthy AI, such as privacy and alignment.
  arXiv  Detail & Related papers  (2023-07-27T15:35:32Z)
• Designing for Responsible Trust in AI Systems: A Communication Perspective [56.80107647520364]
  We draw from communication theories and literature on trust in technologies to develop a conceptual model called MATCH. We highlight transparency and interaction as AI systems' affordances that present a wide range of trustworthiness cues to users. We propose a checklist of requirements to help technology creators identify appropriate cues to use.
  arXiv  Detail & Related papers  (2022-04-29T00:14:33Z)
• Can You be More Social? Injecting Politeness and Positivity into Task-Oriented Conversational Agents [60.27066549589362]
  Social language used by human agents is associated with greater users' responsiveness and task completion. The model uses a sequence-to-sequence deep learning architecture, extended with a social language understanding element. Evaluation in terms of content preservation and social language level using both human judgment and automatic linguistic measures shows that the model can generate responses that enable agents to address users' issues in a more socially appropriate way.
  arXiv  Detail & Related papers  (2020-12-29T08:22:48Z)
• Is your chatbot GDPR compliant? Open issues in agent design [2.66512000865131]
  This paper explores some of the questions with the aim of defining relevant open issues to conversational agent design. We hope that this work can provoke further research into building agents that are effective at user interaction but also respectful of regulations and user privacy.
  arXiv  Detail & Related papers  (2020-05-26T11:54:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.