Over-Relying on Reliance: Towards Realistic Evaluations of AI-Based Clinical Decision Support

• URL: http://arxiv.org/abs/2504.07423v1
• Date: Thu, 10 Apr 2025 03:28:56 GMT
• Title: Over-Relying on Reliance: Towards Realistic Evaluations of AI-Based Clinical Decision Support
• Authors: Venkatesh Sivaraman, Katelyn Morrison, Will Epperson, Adam Perer,
• Abstract summary: We advocate for moving beyond evaluation metrics like Trust, Reliance, Acceptance, and Performance on the AI's task.<n>We call on the community to prioritize ecologically valid, domain-appropriate study setups that measure the emergent forms of value that AI can bring to healthcare professionals.
• Score: 12.247046469627554
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As AI-based clinical decision support (AI-CDS) is introduced in more and more aspects of healthcare services, HCI research plays an increasingly important role in designing for complementarity between AI and clinicians. However, current evaluations of AI-CDS often fail to capture when AI is and is not useful to clinicians. This position paper reflects on our work and influential AI-CDS literature to advocate for moving beyond evaluation metrics like Trust, Reliance, Acceptance, and Performance on the AI's task (what we term the "trap" of human-AI collaboration). Although these metrics can be meaningful in some simple scenarios, we argue that optimizing for them ignores important ways that AI falls short of clinical benefit, as well as ways that clinicians successfully use AI. As the fields of HCI and AI in healthcare develop new ways to design and evaluate CDS tools, we call on the community to prioritize ecologically valid, domain-appropriate study setups that measure the emergent forms of value that AI can bring to healthcare professionals.

Related papers

• The case for delegated AI autonomy for Human AI teaming in healthcare [3.441725960809854]
  We propose an advanced approach to integrating artificial intelligence (AI) into healthcare: autonomous decision support.<n>This approach allows the AI algorithm to act autonomously for a subset of patient cases whilst serving a supportive role in other subsets of patient cases based on defined delegation criteria.<n>It ensures safe handling of patient cases and potentially reduces clinician review time, whilst being mindful of AI tool limitations.
  arXiv  Detail & Related papers  (2025-03-24T15:26:54Z)
• Towards Next-Generation Medical Agent: How o1 is Reshaping Decision-Making in Medical Scenarios [46.729092855387165]
  We study the choice of the backbone LLM for medical AI agents, which is the foundation for the agent's overall reasoning and action generation.<n>Our findings demonstrate o1's ability to enhance diagnostic accuracy and consistency, paving the way for smarter, more responsive AI tools.
  arXiv  Detail & Related papers  (2024-11-16T18:19:53Z)
• Clinicians' Voice: Fundamental Considerations for XAI in Healthcare [0.0]
  We conducted semi-structured interviews with clinicians to discuss their thoughts, hopes, and concerns.<n> Clinicians from our sample generally think positively about developing AI-based tools for clinical practice.<n>They have concerns about how these will fit into their workflow and how it will impact clinician-patient relations.
  arXiv  Detail & Related papers  (2024-11-07T16:47:06Z)
• Contrasting Attitudes Towards Current and Future AI Applications for Computerised Interpretation of ECG: A Clinical Stakeholder Interview Study [2.570550251482137]
  We conducted a series of interviews with clinicians in the UK. Our study explores the potential for AI, specifically future 'human-like' computing.
  arXiv  Detail & Related papers  (2024-10-22T10:31:23Z)
• How Performance Pressure Influences AI-Assisted Decision Making [57.53469908423318]
  We show how pressure and explainable AI (XAI) techniques interact with AI advice-taking behavior. Our results show complex interaction effects, with different combinations of pressure and XAI techniques either improving or worsening AI advice taking behavior.
  arXiv  Detail & Related papers  (2024-10-21T22:39:52Z)
• Towards Clinical AI Fairness: Filling Gaps in the Puzzle [15.543248260582217]
  This review systematically pinpoints several deficiencies concerning both healthcare data and the provided AI fairness solutions. We highlight the scarcity of research on AI fairness in many medical domains where AI technology is increasingly utilized. To bridge these gaps, our review advances actionable strategies for both the healthcare and AI research communities.
  arXiv  Detail & Related papers  (2024-05-28T07:42:55Z)
• Beyond Recommender: An Exploratory Study of the Effects of Different AI Roles in AI-Assisted Decision Making [48.179458030691286]
  We examine three AI roles: Recommender, Analyzer, and Devil's Advocate. Our results show each role's distinct strengths and limitations in task performance, reliance appropriateness, and user experience. These insights offer valuable implications for designing AI assistants with adaptive functional roles according to different situations.
  arXiv  Detail & Related papers  (2024-03-04T07:32:28Z)
• Towards A Unified Utilitarian Ethics Framework for Healthcare Artificial Intelligence [0.08192907805418582]
  This study attempts to identify the major ethical principles influencing the utility performance of AI at different technological levels. Justice, privacy, bias, lack of regulations, risks, and interpretability are the most important principles to consider for ethical AI. We propose a new utilitarian ethics-based theoretical framework for designing ethical AI for the healthcare domain.
  arXiv  Detail & Related papers  (2023-09-26T02:10:58Z)
• FUTURE-AI: International consensus guideline for trustworthy and deployable artificial intelligence in healthcare [73.78776682247187]
  Concerns have been raised about the technical, clinical, ethical and legal risks associated with medical AI. This work describes the FUTURE-AI guideline as the first international consensus framework for guiding the development and deployment of trustworthy AI tools in healthcare.
  arXiv  Detail & Related papers  (2023-08-11T10:49:05Z)
• A Conceptual Algorithm for Applying Ethical Principles of AI to Medical Practice [5.005928809654619]
  AI-powered tools are increasingly matching or exceeding specialist-level performance across multiple domains.<n>These systems promise to reduce disparities in care delivery across demographic, racial, and socioeconomic boundaries.<n>The democratization of such AI tools can reduce the cost of care, optimize resource allocation, and improve the quality of care.
  arXiv  Detail & Related papers  (2023-04-23T04:14:18Z)
• The Role of AI in Drug Discovery: Challenges, Opportunities, and Strategies [97.5153823429076]
  The benefits, challenges and drawbacks of AI in this field are reviewed. The use of data augmentation, explainable AI, and the integration of AI with traditional experimental methods are also discussed.
  arXiv  Detail & Related papers  (2022-12-08T23:23:39Z)
• An interdisciplinary conceptual study of Artificial Intelligence (AI) for helping benefit-risk assessment practices: Towards a comprehensive qualification matrix of AI programs and devices (pre-print 2020) [55.41644538483948]
  This paper proposes a comprehensive analysis of existing concepts coming from different disciplines tackling the notion of intelligence. The aim is to identify shared notions or discrepancies to consider for qualifying AI systems.
  arXiv  Detail & Related papers  (2021-05-07T12:01:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.