The Doctor Will (Still) See You Now: On the Structural Limits of Agentic AI in Healthcare

• URL: http://arxiv.org/abs/2602.18460v1
• Date: Fri, 06 Feb 2026 02:53:40 GMT
• Title: The Doctor Will (Still) See You Now: On the Structural Limits of Agentic AI in Healthcare
• Authors: Gabriela Aránguiz Dias, Kiana Jafari, Allie Griffith, Carolina Aránguiz Dias, Grace Ra Kim, Lana Saadeddin, Mykel J. Kochenderfer,
• Abstract summary: We present a qualitative study based on interviews with 20 stakeholders, including developers, implementers, and end users.<n>Our analysis identifies three mutually reinforcing tensions: conceptual fragmentation regarding the definition of agentic'<n>We argue that agentic AI functions as a site of contested meaning-making where technical aspirations, commercial incentives, and clinical constraints intersect.
• Score: 18.202521625229732
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Across healthcare, agentic artificial intelligence (AI) systems are increasingly promoted as capable of autonomous action, yet in practice they currently operate under near-total human oversight due to safety, regulatory, and liability constraints that make autonomous clinical reasoning infeasible in high-stakes environments. While market enthusiasm suggests a revolution in healthcare agents, the conceptual assumptions and accountability structures shaping these systems remain underexamined. We present a qualitative study based on interviews with 20 stakeholders, including developers, implementers, and end users. Our analysis identifies three mutually reinforcing tensions: conceptual fragmentation regarding the definition of `agentic'; an autonomy contradiction where commercial promises exceed operational reality; and an evaluation blind spot that prioritizes technical benchmarks over sociotechnical safety. We argue that agentic {AI} functions as a site of contested meaning-making where technical aspirations, commercial incentives, and clinical constraints intersect, carrying material consequences for patient safety and the distribution of blame.

Related papers

• Assessing Risks of Large Language Models in Mental Health Support: A Framework for Automated Clinical AI Red Teaming [23.573537738272595]
  We introduce an evaluation framework that pairs AI psychotherapists with simulated patient agents equipped with cognitive-affective models.<n>We apply this framework to a high-impact test case, Alcohol Use Disorder, evaluating six AI agents.<n>Our large-scale simulation reveals critical safety gaps in the use of AI for mental health support.
  arXiv  Detail & Related papers  (2026-02-23T15:17:18Z)
• Augmenting Clinical Decision-Making with an Interactive and Interpretable AI Copilot: A Real-World User Study with Clinicians in Nephrology and Obstetrics [36.981753143345664]
  We present AICare, an interactive and interpretable AI copilot for collaborative clinical decision-making.<n>By analyzing longitudinal electronic health records, AICare grounds dynamic risk predictions in scrutable visualizations.
  arXiv  Detail & Related papers  (2026-01-31T13:41:32Z)
• Responsible Evaluation of AI for Mental Health [72.85175110624736]
  Current approaches to evaluating AI tools in mental health care are fragmented and poorly aligned with clinical practice, social context, and first-hand user experience.<n>This paper argues for a rethinking of responsible evaluation by introducing an interdisciplinary framework that integrates clinical soundness, social context, and equity.
  arXiv  Detail & Related papers  (2026-01-20T12:55:10Z)
• From Values to Frameworks: A Qualitative Study of Ethical Reasoning in Agentic AI Practitioners [0.0]
  Agentic artificial intelligence systems are autonomous technologies capable of pursuing complex goals with minimal human oversight.<n>While these systems promise major gains in productivity, they also raise new ethical challenges.<n>This paper investigates the ethical reasoning of AI practitioners through qualitative interviews centered on structured dilemmas in agentic AI deployment.
  arXiv  Detail & Related papers  (2025-12-24T00:58:41Z)
• AI Deception: Risks, Dynamics, and Controls [153.71048309527225]
  This project provides a comprehensive and up-to-date overview of the AI deception field.<n>We identify a formal definition of AI deception, grounded in signaling theory from studies of animal deception.<n>We organize the landscape of AI deception research as a deception cycle, consisting of two key components: deception emergence and deception treatment.
  arXiv  Detail & Related papers  (2025-11-27T16:56:04Z)
• Never Compromise to Vulnerabilities: A Comprehensive Survey on AI Governance [211.5823259429128]
  We propose a comprehensive framework integrating technical and societal dimensions, structured around three interconnected pillars: Intrinsic Security, Derivative Security, and Social Ethics.<n>We identify three core challenges: (1) the generalization gap, where defenses fail against evolving threats; (2) inadequate evaluation protocols that overlook real-world risks; and (3) fragmented regulations leading to inconsistent oversight.<n>Our framework offers actionable guidance for researchers, engineers, and policymakers to develop AI systems that are not only robust and secure but also ethically aligned and publicly trustworthy.
  arXiv  Detail & Related papers  (2025-08-12T09:42:56Z)
• Advancing Responsible Innovation in Agentic AI: A study of Ethical Frameworks for Household Automation [1.6766200616088744]
  This article analyzes agentic AI and its applications, focusing on its move from reactive to proactive autonomy, privacy, fairness and user control.<n>Vulnerable user groups such as elderly individuals, children, and neurodivergent who face higher risks of surveillance, bias, and privacy risks were studied.<n>Design imperatives are highlighted such as tailored explainability, granular consent mechanisms, and robust override controls.
  arXiv  Detail & Related papers  (2025-07-21T06:10:02Z)
• A Desideratum for Conversational Agents: Capabilities, Challenges, and Future Directions [51.96890647837277]
  Large Language Models (LLMs) have propelled conversational AI from traditional dialogue systems into sophisticated agents capable of autonomous actions, contextual awareness, and multi-turn interactions with users.<n>This survey paper presents a desideratum for next-generation Conversational Agents - what has been achieved, what challenges persist, and what must be done for more scalable systems that approach human-level intelligence.
  arXiv  Detail & Related papers  (2025-04-07T21:01:25Z)
• Artificial Intelligence-Driven Clinical Decision Support Systems [5.010570270212569]
  The chapter emphasizes that creating trustworthy AI systems in healthcare requires careful consideration of fairness, explainability, and privacy.<n>The challenge of ensuring equitable healthcare delivery through AI is stressed, discussing methods to identify and mitigate bias in clinical predictive models.<n>The discussion advances in an analysis of privacy vulnerabilities in medical AI systems, from data leakage in deep learning models to sophisticated attacks against model explanations.
  arXiv  Detail & Related papers  (2025-01-16T16:17:39Z)
• Towards Next-Generation Medical Agent: How o1 is Reshaping Decision-Making in Medical Scenarios [46.729092855387165]
  We study the choice of the backbone LLM for medical AI agents, which is the foundation for the agent's overall reasoning and action generation.<n>Our findings demonstrate o1's ability to enhance diagnostic accuracy and consistency, paving the way for smarter, more responsive AI tools.
  arXiv  Detail & Related papers  (2024-11-16T18:19:53Z)
• Visibility into AI Agents [9.067567737098594]
  Increased delegation of commercial, scientific, governmental, and personal activities to AI agents may exacerbate existing societal risks. We assess three categories of measures to increase visibility into AI agents: agent identifiers, real-time monitoring, and activity logging.
  arXiv  Detail & Related papers  (2024-01-23T23:18:33Z)
• An interdisciplinary conceptual study of Artificial Intelligence (AI) for helping benefit-risk assessment practices: Towards a comprehensive qualification matrix of AI programs and devices (pre-print 2020) [55.41644538483948]
  This paper proposes a comprehensive analysis of existing concepts coming from different disciplines tackling the notion of intelligence. The aim is to identify shared notions or discrepancies to consider for qualifying AI systems.
  arXiv  Detail & Related papers  (2021-05-07T12:01:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.