Safety challenges of AI in medicine in the era of large language models

• URL: http://arxiv.org/abs/2409.18968v2
• Date: Thu, 30 Jan 2025 08:55:23 GMT
• Title: Safety challenges of AI in medicine in the era of large language models
• Authors: Xiaoye Wang, Nicole Xi Zhang, Hongyu He, Trang Nguyen, Kun-Hsing Yu, Hao Deng, Cynthia Brandt, Danielle S. Bitterman, Ling Pan, Ching-Yu Cheng, James Zou, Dianbo Liu,
• Abstract summary: Large language models (LLMs) offer new opportunities for medical practitioners, patients, and researchers. As AI and LLMs become more powerful and especially achieve superhuman performance in some medical tasks, public concerns over their safety have intensified. This review examines emerging risks in AI utilization during the LLM era.
• Score: 23.817939398729955
• License:
• Abstract: Recent advancements in artificial intelligence (AI), particularly in large language models (LLMs), have unlocked significant potential to enhance the quality and efficiency of medical care. By introducing a novel way to interact with AI and data through natural language, LLMs offer new opportunities for medical practitioners, patients, and researchers. However, as AI and LLMs become more powerful and especially achieve superhuman performance in some medical tasks, public concerns over their safety have intensified. These concerns about AI safety have emerged as the most significant obstacles to the adoption of AI in medicine. In response, this review examines emerging risks in AI utilization during the LLM era. First, we explore LLM-specific safety challenges from functional and communication perspectives, addressing issues across data collection, model training, and real-world application. We then consider inherent safety problems shared by all AI systems, along with additional complications introduced by LLMs. Last, we discussed how safety issues of using AI in clinical practice and healthcare system operation would undermine trust among patient, clinicians and the public, and how to build confidence in these systems. By emphasizing the development of safe AI, we believe these technologies can be more rapidly and reliably integrated into everyday medical practice to benefit both patients and clinicians.

Related papers

• Towards Privacy-aware Mental Health AI Models: Advances, Challenges, and Opportunities [61.633126163190724]
  Mental illness is a widespread and debilitating condition with substantial societal and personal costs. Recent advances in Artificial Intelligence (AI) hold great potential for recognizing and addressing conditions such as depression, anxiety disorder, bipolar disorder, schizophrenia, and post-traumatic stress disorder. Privacy concerns, including the risk of sensitive data leakage from datasets and trained models, remain a critical barrier to deploying these AI systems in real-world clinical settings.
  arXiv  Detail & Related papers  (2025-02-01T15:10:02Z)
• Towards Safe AI Clinicians: A Comprehensive Study on Large Language Model Jailbreaking in Healthcare [15.438265972219869]
  Large language models (LLMs) are increasingly utilized in healthcare applications. This study systematically assesses the vulnerabilities of six LLMs to three advanced black-box jailbreaking techniques.
  arXiv  Detail & Related papers  (2025-01-27T22:07:52Z)
• Open Problems in Machine Unlearning for AI Safety [61.43515658834902]
  Machine unlearning -- the ability to selectively forget or suppress specific types of knowledge -- has shown promise for privacy and data removal tasks. In this paper, we identify key limitations that prevent unlearning from serving as a comprehensive solution for AI safety.
  arXiv  Detail & Related papers  (2025-01-09T03:59:10Z)
• A Trilogy of AI Safety Frameworks: Paths from Facts and Knowledge Gaps to Reliable Predictions and New Knowledge [0.0]
  AI Safety has become a vital front-line concern of many scientists within and outside the AI community. There are many immediate and long term anticipated risks that range from existential risk to human existence to deep fakes and bias in machine learning systems.
  arXiv  Detail & Related papers  (2024-10-09T14:43:06Z)
• HAICOSYSTEM: An Ecosystem for Sandboxing Safety Risks in Human-AI Interactions [76.42274173122328]
  We present HAICOSYSTEM, a framework examining AI agent safety within diverse and complex social interactions. We run 1840 simulations based on 92 scenarios across seven domains (e.g., healthcare, finance, education) Our experiments show that state-of-the-art LLMs, both proprietary and open-sourced, exhibit safety risks in over 50% cases.
  arXiv  Detail & Related papers  (2024-09-24T19:47:21Z)
• SoK: Security and Privacy Risks of Medical AI [14.592921477833848]
  The integration of technology and healthcare has ushered in a new era where software systems, powered by artificial intelligence and machine learning, have become essential components of medical products and services. This paper explores the security and privacy threats posed by AI/ML applications in healthcare.
  arXiv  Detail & Related papers  (2024-09-11T16:59:58Z)
• Towards Human-AI Collaboration in Healthcare: Guided Deferral Systems with Large Language Models [1.2281181385434294]
  Large language models (LLMs) present a valuable technology for various applications in healthcare. Their tendency to hallucinate introduces unacceptable uncertainty in critical decision-making situations. Human-AI collaboration can mitigate this uncertainty by combining human and AI strengths for better outcomes. This paper presents a novel guided deferral system that provides intelligent guidance when AI defers cases to human decision-makers.
  arXiv  Detail & Related papers  (2024-06-11T12:41:54Z)
• RAISE -- Radiology AI Safety, an End-to-end lifecycle approach [5.829180249228172]
  The integration of AI into radiology introduces opportunities for improved clinical care provision and efficiency. The focus should be on ensuring models meet the highest standards of safety, effectiveness and efficacy. The roadmap presented herein aims to expedite the achievement of deployable, reliable, and safe AI in radiology.
  arXiv  Detail & Related papers  (2023-11-24T15:59:14Z)
• Managing extreme AI risks amid rapid progress [171.05448842016125]
  We describe risks that include large-scale social harms, malicious uses, and irreversible loss of human control over autonomous AI systems. There is a lack of consensus about how exactly such risks arise, and how to manage them. Present governance initiatives lack the mechanisms and institutions to prevent misuse and recklessness, and barely address autonomous systems.
  arXiv  Detail & Related papers  (2023-10-26T17:59:06Z)
• FUTURE-AI: International consensus guideline for trustworthy and deployable artificial intelligence in healthcare [73.78776682247187]
  Concerns have been raised about the technical, clinical, ethical and legal risks associated with medical AI. This work describes the FUTURE-AI guideline as the first international consensus framework for guiding the development and deployment of trustworthy AI tools in healthcare.
  arXiv  Detail & Related papers  (2023-08-11T10:49:05Z)
• The Role of AI in Drug Discovery: Challenges, Opportunities, and Strategies [97.5153823429076]
  The benefits, challenges and drawbacks of AI in this field are reviewed. The use of data augmentation, explainable AI, and the integration of AI with traditional experimental methods are also discussed.
  arXiv  Detail & Related papers  (2022-12-08T23:23:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.