When RAG Chatbots Expose Their Backend: An Anonymized Case Study of Privacy and Security Risks in Patient-Facing Medical AI

• URL: http://arxiv.org/abs/2605.00796v1
• Date: Fri, 01 May 2026 17:29:09 GMT
• Title: When RAG Chatbots Expose Their Backend: An Anonymized Case Study of Privacy and Security Risks in Patient-Facing Medical AI
• Abstract summary: AI-assisted development lowers the barrier to building them, but they still demand rigorous security, privacy, and governance controls.<n>We report an anonymized, non-destructive security assessment of a publicly accessible patient-facing medical RAG chatbots.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Background: Patient-facing medical chatbots based on retrieval-augmented generation (RAG) are increasingly promoted to deliver accessible, grounded health information. AI-assisted development lowers the barrier to building them, but they still demand rigorous security, privacy, and governance controls. Objective: To report an anonymized, non-destructive security assessment of a publicly accessible patient-facing medical RAG chatbot and identify governance lessons for safe deployment of generative AI in health. Methods: We used a two-stage strategy. First, Claude Opus 4.6 supported exploratory prompt-based testing and structured vulnerability hypotheses. Second, candidate findings were manually verified using Chrome Developer Tools, inspecting browser-visible network traffic, payloads, API schemas, configuration objects, and stored interaction data. Results: The LLM-assisted phase identified a critical vulnerability: sensitive system and RAG configuration appeared exposed through client-server communication rather than restricted server-side. Manual verification confirmed that ordinary browser inspection allowed collection of the system prompt, model and embedding configuration, retrieval parameters, backend endpoints, API schema, document and chunk metadata, knowledge-base content, and the 1,000 most recent patient-chatbot conversations. The deployment also contradicted its privacy assurances: full conversation records, including health-related queries, were retrievable without authentication. Conclusions: Serious privacy and security failures in patient-facing RAG chatbots can be identified with standard browser tools, without specialist skills or authentication; independent review should be a prerequisite for deployment. Commercial LLMs accelerated this assessment, including under a false developer persona; assistance available to auditors is equally available to adversaries.

Related papers

• Adversarial Attacks for Good: A Survey of Proactive Protection across the Visual Content Lifecycle [65.6642776933861]
  This survey examines the protective paradigm that has grown around this intervention point.<n>We find that most protections are still validated mainly against static or weakly adaptive adversaries.<n>We close by consolidating cross-stage countermeasures and open problems for robust, composable, and deployable owner-side protection.
  arXiv  Detail & Related papers  (2026-08-05T00:46:50Z)
• An Evaluation of Data Leakage Risks in Tool-Using LLM Agents in Realistic Scenarios [3.9838595042419076]
  We report a joint evaluation by the Singapore AI Safety Institute and the Korea AI Safety Institute.<n>The evaluation covers five risk types: lack of data awareness, audience awareness, policy compliance, data minimization, and access-boundary awareness.<n>Overall, the results indicate that operational data leakage is a first-order agent-safety concern distinct from adversarial exfiltration.
  arXiv  Detail & Related papers  (2026-06-15T09:16:38Z)
• WebAgentGuard: A Reasoning-Driven Guard Model for Detecting Prompt Injection Attacks in Web Agents [117.65855863464863]
  Web agents are highly vulnerable to prompt injection attacks.<n>Existing defenses, including system prompt defenses and direct fine-tuning of agents, have shown limited effectiveness.<n>We introduce WebAgentGuard, a reasoning-driven, multimodal guard model for prompt injection detection.
  arXiv  Detail & Related papers  (2026-04-14T04:50:35Z)
• Caging the Agents: A Zero Trust Security Architecture for Autonomous AI in Healthcare [0.0]
  This paper presents a security architecture deployed for nine autonomous AI agents in production at a healthcare technology company.<n>We develop a six-domain threat model for agentic AI in healthcare covering credential exposure, execution capability abuse, network egress exfiltration, prompt integrity failures, database access risks, and fleet configuration drift.<n>All configurations, audit tooling, and the prompt integrity framework are released as open source.
  arXiv  Detail & Related papers  (2026-03-18T06:54:47Z)
• Semantic Risk Scoring of Aggregated Metrics: An AI-Driven Approach for Healthcare Data Governance [0.0]
  Large healthcare institutions typically operate multiple business intelligence (BI) teams.<n>Due to HIPAA, FERPA, and IRB restrictions, these teams face challenges in sharing patient-level data needed for analytics.<n>A metric aggregation table is proposed, which is a precomputed, privacy-compliant summary.
  arXiv  Detail & Related papers  (2026-03-09T03:36:11Z)
• The Trojan Knowledge: Bypassing Commercial LLM Guardrails via Harmless Prompt Weaving and Adaptive Tree Search [58.8834056209347]
  Large language models (LLMs) remain vulnerable to jailbreak attacks that bypass safety guardrails to elicit harmful outputs.<n>We introduce the Correlated Knowledge Attack Agent (CKA-Agent), a dynamic framework that reframes jailbreaking as an adaptive, tree-structured exploration of the target model's knowledge base.
  arXiv  Detail & Related papers  (2025-12-01T07:05:23Z)
• VERA-MH Concept Paper [0.0]
  We introduce VERA-MH, an automated evaluation of the safety of AI chatbots used in mental health contexts.<n>To fully automate the process, we used two ancillary AI agents.<n>Simulated conversations are then passed to a judge-agent who scores them based on the rubric.
  arXiv  Detail & Related papers  (2025-10-17T04:07:29Z)
• Malice in Agentland: Down the Rabbit Hole of Backdoors in the AI Supply Chain [82.98626829232899]
  Fine-tuning AI agents on data from their own interactions introduces a critical security vulnerability within the AI supply chain.<n>We show that adversaries can easily poison the data collection pipeline to embed hard-to-detect backdoors.
  arXiv  Detail & Related papers  (2025-10-03T12:47:21Z)
• DRIFT: Dynamic Rule-Based Defense with Injection Isolation for Securing LLM Agents [52.92354372596197]
  Large Language Models (LLMs) are increasingly central to agentic systems due to their strong reasoning and planning capabilities.<n>This interaction also introduces the risk of prompt injection attacks, where malicious inputs from external sources can mislead the agent's behavior.<n>We propose a Dynamic Rule-based Isolation Framework for Trustworthy agentic systems, which enforces both control and data-level constraints.
  arXiv  Detail & Related papers  (2025-06-13T05:01:09Z)
• VPI-Bench: Visual Prompt Injection Attacks for Computer-Use Agents [74.6761188527948]
  Computer-Use Agents (CUAs) with full system access pose significant security and privacy risks.<n>We investigate Visual Prompt Injection (VPI) attacks, where malicious instructions are visually embedded within rendered user interfaces.<n>Our empirical study shows that current CUAs and BUAs can be deceived at rates of up to 51% and 100%, respectively, on certain platforms.
  arXiv  Detail & Related papers  (2025-06-03T05:21:50Z)
• Decoding User Concerns in AI Health Chatbots: An Exploration of Security and Privacy in App Reviews [1.2437039433843042]
  This study evaluates the effectiveness of automated methods, specifically BART and Gemini GenAI, in identifying security privacy related (SPR) concerns.<n>Our results indicate that while Gemini's performance in SPR classification is comparable to manual labeling, both automated methods have limitations.
  arXiv  Detail & Related papers  (2025-01-31T00:38:37Z)
• Confidential and Protected Disease Classifier using Fully Homomorphic Encryption [0.09424565541639365]
  Many users seek potential causes on platforms like ChatGPT or Bard before consulting a medical professional for their ailment. Despite the convenience of such platforms, sharing personal medical data online poses risks, including the presence of malicious platforms. We propose a novel framework combining FHE and Deep Learning for a secure and private diagnosis system.
  arXiv  Detail & Related papers  (2024-05-05T02:10:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.