Toward Trustworthy Large Language Model Agents in Healthcare

• URL: http://arxiv.org/abs/2607.05055v1
• Date: Mon, 06 Jul 2026 13:29:07 GMT
• Title: Toward Trustworthy Large Language Model Agents in Healthcare
• Abstract summary: CareConnect is a safety-first conversational agent for healthcare logistics automation.<n>The system orchestrates eight domain-specific tools to support appointment booking, modification, cancellation, and facility information retrieval.
• Score: 0.6633201258809686
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Healthcare appointment scheduling remains a persistent operational bottleneck, driven by manual coordination, fragmented legacy systems, and high administrative overhead. These inefficiencies constrain provider availability and degrade patient access to care. This paper presents CareConnect, a safety-first conversational agent for healthcare logistics automation that leverages large language model (LLM) function calling, retrieval-augmented generation (RAG), and layered deterministic safety guardrails. The system orchestrates eight domain-specific tools to support appointment booking, modification, cancellation, and facility information retrieval, while enforcing strict scope constraints that prohibit medical advice or diagnosis. Safety-critical situations are handled through deterministic short-circuit mechanisms for emergency detection and medical intent refusal. We evaluate CareConnect on a comprehensive benchmark of 680 task-oriented scenarios spanning end-to-end workflows, multi-turn interactions, and edge cases. Experimental results demonstrate a 91.8% task completion rate with a median per-request latency of 2.2 seconds, 96.0% safety compliance on the dedicated safety-critical evaluation subset, and an average operational cost of $0.0324 per appointment, yielding a significant cost reduction compared to manual human scheduling. These findings show that carefully scoped and rigorously safeguarded LLM-based agents can reliably automate complex healthcare operational workflows while maintaining safety guarantees and achieving substantial cost efficiency. The source code and system implementation are publicly available at https://github.com/Hadi-Hsn/CareConnect.

Related papers

• SeClaw: Spec-Driven Security Task Synthesis for Evaluating Autonomous Agents [87.26967184869198]
  SeClaw is a framework that combines specification-driven security task synthesis with execution-based security evaluation for Autonomous agents.<n> benchmark covers risks arising from resources, user tasks, environments, and intrinsic agent behaviors.
  arXiv  Detail & Related papers  (2026-06-01T14:23:42Z)
• CHI-Bench: Can AI Agents Automate End-to-End, Long-Horizon, Policy-Rich Healthcare Workflows? [150.84850629123287]
  End-to-end automation of realistic healthcare operations stresses three capabilities underrepresented in current benchmarks.<n>$$-Bench is a benchmark of long-horizon healthcare across three domains: provider prior authorization, payer utilization management, and care management.<n>Across 30 agent harness/models configurations, the best agent resolves only 28.0% of tasks, no agent clears 20% on strict pass3, and executing all tasks in a single session slumps the performance to 3.8%.
  arXiv  Detail & Related papers  (2026-05-15T22:34:31Z)
• CareGuardAI: Context-Aware Multi-Agent Guardrails for Clinical Safety & Hallucination Mitigation in Patient-Facing LLMs [5.437368195729715]
  CareGuardAI is a risk-aware safety framework for patient-facing medical question answering.<n>At inference time, CareGuardAI employs a multi-stage pipeline consisting of a controller agent, safety-constrained generation, and dual risk evaluation.<n>We evaluate CareGuardAI on PatientSafeBench, MedSafetyBench, and MedHallu, covering both safety and hallucination detection.
  arXiv  Detail & Related papers  (2026-04-07T10:54:57Z)
• LiaisonAgent: An Multi-Agent Framework for Autonomous Risk Investigation and Governance [11.009578207207419]
  LiaisonAgent is an autonomous multi-agent system designed to bridge the gap between technical risk detection and business-level risk governance.<n>The system achieves an end-to-end tool-calling success rate of 97.8% and a risk judgment accuracy of 95%.
  arXiv  Detail & Related papers  (2026-02-27T08:42:52Z)
• Towards Verifiably Safe Tool Use for LLM Agents [53.55621104327779]
  Large language model (LLM)-based AI agents extend capabilities by enabling access to tools such as data sources, APIs, search engines, code sandboxes, and even other agents.<n>LLMs may invoke unintended tool interactions and introduce risks, such as leaking sensitive data or overwriting critical records.<n>Current approaches to mitigate these risks, such as model-based safeguards, enhance agents' reliability but cannot guarantee system safety.
  arXiv  Detail & Related papers  (2026-01-12T21:31:38Z)
• Reliable LLM-Based Edge-Cloud-Expert Cascades for Telecom Knowledge Systems [54.916243942641444]
  Large language models (LLMs) are emerging as key enablers of automation in domains such as telecommunications.<n>We study an edge-cloud-expert cascaded LLM-based knowledge system that supports decision-making through a question-and-answer pipeline.
  arXiv  Detail & Related papers  (2025-12-23T03:10:09Z)
• Differential Privacy for Secure Machine Learning in Healthcare IoT-Cloud Systems [11.452686322120428]
  We propose a multi-layer IoT, Edge and Cloud architecture to enhance the speed of response for emergency healthcare.<n>Privacy of patient data is assured by proposing a Differential Privacy framework across several machine learning models.
  arXiv  Detail & Related papers  (2025-12-11T08:37:37Z)
• From Coordination to Personalization: A Trust-Aware Simulation Framework for Emergency Department Decision Support [0.0]
  This study proposes a simulation-based framework for modeling doctors and nurses as intelligent agents guided by computational trust mechanisms.<n>The proposed framework demonstrates the potential of computational trust for evidence-based decision support in emergency medicine.
  arXiv  Detail & Related papers  (2025-09-09T18:00:44Z)
• Beyond Benchmarks: Dynamic, Automatic And Systematic Red-Teaming Agents For Trustworthy Medical Language Models [87.66870367661342]
  Large language models (LLMs) are used in AI applications in healthcare.<n>Red-teaming framework that continuously stress-test LLMs can reveal significant weaknesses in four safety-critical domains.<n>A suite of adversarial agents is applied to autonomously mutate test cases, identify/evolve unsafe-triggering strategies, and evaluate responses.<n>Our framework delivers an evolvable, scalable, and reliable safeguard for the next generation of medical AI.
  arXiv  Detail & Related papers  (2025-07-30T08:44:22Z)
• U-PASS: an Uncertainty-guided deep learning Pipeline for Automated Sleep Staging [61.6346401960268]
  We propose a machine learning pipeline called U-PASS tailored for clinical applications that incorporates uncertainty estimation at every stage of the process. We apply our uncertainty-guided deep learning pipeline to the challenging problem of sleep staging and demonstrate that it systematically improves performance at every stage.
  arXiv  Detail & Related papers  (2023-06-07T08:27:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.