SoK: Privacy-aware LLM in Healthcare: Threat Model, Privacy Techniques, Challenges and Recommendations

• URL: http://arxiv.org/abs/2601.10004v1
• Date: Thu, 15 Jan 2026 02:28:57 GMT
• Title: SoK: Privacy-aware LLM in Healthcare: Threat Model, Privacy Techniques, Challenges and Recommendations
• Authors: Mohoshin Ara Tahera, Karamveer Singh Sidhu, Shuvalaxmi Dass, Sajal Saha,
• Abstract summary: Large Language Models (LLMs) are increasingly adopted in healthcare to support clinical decision-making and enhance patient care.<n>This work examines the evolving threat landscape across the three core LLM phases: Data preprocessing, Fine-tuning, and Inference within realistic healthcare settings.<n>We present a detailed threat model that characterizes adversaries, capabilities, and attack surfaces at each phase, and we systematize how existing privacy-preserving techniques (PPTs) attempt to mitigate these vulnerabilities.
• Score: 0.6533091401094101
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Models (LLMs) are increasingly adopted in healthcare to support clinical decision-making, summarize electronic health records (EHRs), and enhance patient care. However, this integration introduces significant privacy and security challenges, driven by the sensitivity of clinical data and the high-stakes nature of medical workflows. These risks become even more pronounced across heterogeneous deployment environments, ranging from small on-premise hospital systems to regional health networks, each with unique resource limitations and regulatory demands. This Systematization of Knowledge (SoK) examines the evolving threat landscape across the three core LLM phases: Data preprocessing, Fine-tuning, and Inference within realistic healthcare settings. We present a detailed threat model that characterizes adversaries, capabilities, and attack surfaces at each phase, and we systematize how existing privacy-preserving techniques (PPTs) attempt to mitigate these vulnerabilities. While existing defenses show promise, our analysis identifies persistent limitations in securing sensitive clinical data across diverse operational tiers. We conclude with phase-aware recommendations and future research directions aimed at strengthening privacy guarantees for LLMs in regulated environments. This work provides a foundation for understanding the intersection of LLMs, threats, and privacy in healthcare, offering a roadmap toward more robust and clinically trustworthy AI systems.

Related papers

• SoK: Privacy Risks and Mitigations in Retrieval-Augmented Generation Systems [53.51921540246166]
  Retrieval-Augmented Generation (RAG) techniques have become widely popular.<n>RAG involves the coupling of Large Language Models (LLMs) with domain-specific knowledge bases.<n>The proliferation of RAG has sparked concerns about data privacy.
  arXiv  Detail & Related papers  (2026-01-07T14:50:41Z)
• A Practical Framework for Evaluating Medical AI Security: Reproducible Assessment of Jailbreaking and Privacy Vulnerabilities Across Clinical Specialties [11.500745861209774]
  Medical Large Language Models (LLMs) are increasingly deployed for clinical decision support across diverse specialties.<n>Existing security benchmarks require GPU clusters, commercial API access, or protected health data.<n>We propose a practical, fully reproducible framework for evaluating medical AI security under realistic resource constraints.
  arXiv  Detail & Related papers  (2025-12-09T02:28:15Z)
• Privacy Challenges and Solutions in Retrieval-Augmented Generation-Enhanced LLMs for Healthcare Chatbots: A Review of Applications, Risks, and Future Directions [3.36168223686933]
  Retrieval-augmented generation (RAG) has rapidly emerged as a transformative approach for integrating large language models into clinical and biomedical systems.<n>This review provides a thorough analysis of the current landscape of RAG applications in healthcare.
  arXiv  Detail & Related papers  (2025-11-14T14:33:58Z)
• Exploring Membership Inference Vulnerabilities in Clinical Large Language Models [42.52690697965999]
  We present an exploratory empirical study on membership inference vulnerabilities in clinical large language models (LLMs)<n>Using a state-of-the-art clinical question-answering model, Llemr, we evaluate both canonical loss-based attacks and a domain-motivated paraphrasing-based perturbation strategy.<n>Results motivate continued development of context-aware, domain-specific privacy evaluations and defenses.
  arXiv  Detail & Related papers  (2025-10-21T14:27:48Z)
• Medical Red Teaming Protocol of Language Models: On the Importance of User Perspectives in Healthcare Settings [48.096652370210016]
  We introduce a safety evaluation protocol tailored to the medical domain in both patient user and clinician user perspectives.<n>This is the first work to define safety evaluation criteria for medical LLMs through targeted red-teaming taking three different points of view.
  arXiv  Detail & Related papers  (2025-07-09T19:38:58Z)
• A Comprehensive Survey on the Trustworthiness of Large Language Models in Healthcare [8.378348088931578]
  The application of large language models (LLMs) in healthcare holds significant promise for enhancing clinical decision-making, medical research, and patient care.<n>Their integration into real-world clinical settings raises critical concerns around trustworthiness, particularly around dimensions of truthfulness, privacy, safety, robustness, fairness, and explainability.
  arXiv  Detail & Related papers  (2025-02-21T18:43:06Z)
• Large Language Models in Healthcare [4.119811542729794]
  Large language models (LLMs) hold promise for transforming healthcare.<n>Their successful integration requires rigorous development, adaptation, and evaluation strategies tailored to clinical needs.
  arXiv  Detail & Related papers  (2025-02-06T20:53:33Z)
• Towards Privacy-aware Mental Health AI Models: Advances, Challenges, and Opportunities [58.61680631581921]
  Mental health disorders create profound personal and societal burdens, yet conventional diagnostics are resource-intensive and limit accessibility.<n>This paper examines these challenges and proposes solutions, including anonymization, synthetic data, and privacy-preserving training.<n>It aims to advance reliable, privacy-aware AI tools that support clinical decision-making and improve mental health outcomes.
  arXiv  Detail & Related papers  (2025-02-01T15:10:02Z)
• Trustworthy and Practical AI for Healthcare: A Guided Deferral System with Large Language Models [1.2281181385434294]
  Large language models (LLMs) offer a valuable technology for various applications in healthcare.<n>Their tendency to hallucinate and the existing reliance on proprietary systems pose challenges in environments concerning critical decision-making.<n>This paper presents a novel HAIC guided deferral system that can simultaneously parse medical reports for disorder classification, and defer uncertain predictions with intelligent guidance to humans.
  arXiv  Detail & Related papers  (2024-06-11T12:41:54Z)
• Large Language Models for Healthcare Data Augmentation: An Example on Patient-Trial Matching [49.78442796596806]
  We propose an innovative privacy-aware data augmentation approach for patient-trial matching (LLM-PTM) Our experiments demonstrate a 7.32% average improvement in performance using the proposed LLM-PTM method, and the generalizability to new data is improved by 12.12%.
  arXiv  Detail & Related papers  (2023-03-24T03:14:00Z)
• The Medkit-Learn(ing) Environment: Medical Decision Modelling through Simulation [81.72197368690031]
  We present a new benchmarking suite designed specifically for medical sequential decision making. The Medkit-Learn(ing) Environment is a publicly available Python package providing simple and easy access to high-fidelity synthetic medical data.
  arXiv  Detail & Related papers  (2021-06-08T10:38:09Z)
• Privacy-preserving medical image analysis [53.4844489668116]
  We present PriMIA, a software framework designed for privacy-preserving machine learning (PPML) in medical imaging. We show significantly better classification performance of a securely aggregated federated learning model compared to human experts on unseen datasets. We empirically evaluate the framework's security against a gradient-based model inversion attack.
  arXiv  Detail & Related papers  (2020-12-10T13:56:00Z)
• COVI White Paper [67.04578448931741]
  Contact tracing is an essential tool to change the course of the Covid-19 pandemic. We present an overview of the rationale, design, ethical considerations and privacy strategy of COVI,' a Covid-19 public peer-to-peer contact tracing and risk awareness mobile application developed in Canada.
  arXiv  Detail & Related papers  (2020-05-18T07:40:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.