Privacy-Preserving Local Language Models for Longitudinal Data Retrieval in Chronic Dermatologic Disease: Implementation in Pemphigus Patients

• URL: http://arxiv.org/abs/2605.25020v1
• Date: Sun, 24 May 2026 12:00:13 GMT
• Title: Privacy-Preserving Local Language Models for Longitudinal Data Retrieval in Chronic Dermatologic Disease: Implementation in Pemphigus Patients
• Abstract summary: Chronic dermatologic diseases such as pemphigus require long-term follow-up.<n>Privacy-preserving small language model (SLM) could retrieve structured clinical features and generate longitudinal summaries.
• Score: 0.28909295536379814
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Chronic dermatologic diseases such as pemphigus require long-term follow-up, generating extensive longitudinal clinical documentation that is difficult to review comprehensively during routine visits and increasing clinician workload as well as the risk of missing critical historical information. We evaluated whether a locally deployed, privacy-preserving small language model (SLM) could retrieve structured clinical features and generate longitudinal summaries from long-term dermatology follow-up records. In this retrospective case series, thirty pemphigus patients contributed 541 visit notes that were aggregated into full longitudinal records (89,336 words); 56 clinically relevant features were annotated by two expert dermatologists. The locally deployed SLM (Qwen3 4B Thinking 2507) was queried with each complete record to retrieve 56 features and generate one final report summaries. Across 1,680 feature retrieval tasks, mean accuracy was 82.25%. Dermatologists' ratings of AI-generated summaries were high for overall quality (8.23-8.47), clinical accuracy (7.93-8.20), and usefulness (8.47-8.50), with no significant inter-evaluator differences and an overall preference for AI summaries in 53.3% of evaluations. These findings suggest that privacy-preserving, locally deployed SLMs can outperform medical experts and reliably generate clinically meaningful longitudinal summaries. SLMs may support clinical decision-making when integrated with appropriate oversight.

Related papers

• AIPatient Arena: EHR-grounded evaluation of large language models in end-to-end clinical consultation workflows [7.354281305130301]
  Large language models (LLMs) are increasingly considered for use in clinical consultation tasks.<n>We propose AIPatient Arena, an EHRs-grounded evaluation framework for assessing the clinical utility of LLMs.
  arXiv  Detail & Related papers  (2026-06-16T03:35:17Z)
• Evaluating Large Language Models in Dynamic Clinical Decision-Making with Standardized Patient Cases [71.12461204050985]
  MedSP1000 is an SP-derived interactive benchmark for clinical-agent evaluation.<n>It converts peer-reviewed SP teaching cases into executable scenarios with defined SP case scripts, clinical environment contexts, and human-validated structured rubrics.<n>Applying MedSP1000 to a range of general-purpose and medically specialized LLMs, we find that performance on static benchmarks does not reliably translate to such educational scenarios.
  arXiv  Detail & Related papers  (2026-06-03T17:17:16Z)
• Detecting Clinical Discrepancies in Health Coaching Agents: A Dual-Stream Memory and Reconciliation Architecture [71.46525715889656]
  General-purpose agent memory systems optimize for coherence by overwriting older facts with the user's latest statement.<n>We introduce a Dual-Stream Memory Architecture that strictly separates the patient narrative from the structured clinical record.<n>We evaluate this architecture on 26 patients across 675 longitudinal wellness coaching sessions.
  arXiv  Detail & Related papers  (2026-04-29T17:59:28Z)
• A Decade-Scale Benchmark Evaluating LLMs' Clinical Practice Guidelines Detection and Adherence in Multi-turn Conversations [60.2076951536797]
  Large Language Models (LLMs) are increasingly deployed in healthcare scenarios.<n>It is unclear to which extend LLMs could identify and adhere to clinical guidelines during conversations.<n>CPGBench is an automated framework benchmarking the clinical guideline detection and adherence capabilities of LLMs.
  arXiv  Detail & Related papers  (2026-03-26T09:00:55Z)
• Cerebra: A Multidisciplinary AI Board for Multimodal Dementia Characterization and Risk Assessment [56.62016795093786]
  Cerebra is an interactive multi-agent AI team that coordinates specialized agents for EHR, clinical notes, and medical imaging analysis.<n>It supports privacy-preserving deployment by operating on structured representations and remains robust when modalities are incomplete.<n>Cerebra significantly improved expert performance, increasing accuracy by 17.5 percentage points in prospective dementia risk estimation.
  arXiv  Detail & Related papers  (2026-03-23T05:46:45Z)
• QuarkMedBench: A Real-World Scenario Driven Benchmark for Evaluating Large Language Models [8.050059911826338]
  Large Language Models (LLMs) excel on standardized medical exams, but high scores often fail to translate to high-quality responses for real-world medical queries.<n>We introduce QuarkMedBench, an ecologically valid benchmark tailored for real-world medical LLM assessment.
  arXiv  Detail & Related papers  (2026-03-14T01:51:43Z)
• PanCanBench: A Comprehensive Benchmark for Evaluating Large Language Models in Pancreatic Oncology [48.732366302949515]
  Large language models (LLMs) have achieved expert-level performance on standardized examinations, yet multiple-choice accuracy poorly reflects real-world clinical utility and safety.<n>We developed a human-in-the-loop pipeline to create expert rubrics for de-identified patient questions.<n>We evaluated 22 proprietary and open-source LLMs using an LLM-as-a-judge framework, measuring clinical completeness, factual accuracy, and web-search integration.
  arXiv  Detail & Related papers  (2026-03-02T00:50:39Z)
• LiveClin: A Live Clinical Benchmark without Leakage [50.45415584327275]
  LiveClin is a live benchmark designed for approximating real-world clinical practice.<n>We transform authentic patient cases into complex, multimodal evaluation scenarios that span the entire clinical pathway.<n>Our evaluation of 26 models on LiveClin reveals the profound difficulty of these real-world scenarios, with the top-performing model achieving a Case Accuracy of just 35.7%.
  arXiv  Detail & Related papers  (2026-02-18T03:59:46Z)
• End-To-End Clinical Trial Matching with Large Language Models [0.6151041580858937]
  We present an end-to-end pipeline for clinical trial matching using Large Language Models (LLMs) Our approach identifies relevant candidate trials in 93.3% of cases and achieves a preliminary accuracy of 88.0%. Our fully end-to-end pipeline can operate autonomously or with human supervision and is not restricted to oncology.
  arXiv  Detail & Related papers  (2024-07-18T12:36:26Z)
• LongHealth: A Question Answering Benchmark with Long Clinical Documents [36.05587855811346]
  We present the LongHealth benchmark, comprising 20 detailed fictional patient cases across various diseases. The benchmark challenges LLMs with 400 multiple-choice questions in three categories: information extraction, negation, and sorting. We evaluated nine open-source LLMs with a minimum of 16,000 tokens and also included OpenAI's proprietary and cost-efficient GPT-3.5 Turbo for comparison.
  arXiv  Detail & Related papers  (2024-01-25T19:57:00Z)
• CORAL: Expert-Curated medical Oncology Reports to Advance Language Model Inference [2.1067045507411195]
  Large language models (LLMs) have recently exhibited impressive performance on various medical natural language processing tasks. We developed a detailed schema for annotating textual oncology information, encompassing patient characteristics, tumor characteristics, tests, treatments, and temporality. The GPT-4 model exhibited overall best performance, with an average BLEU score of 0.73, an average ROUGE score of 0.72, an exact-match F1-score of 0.51, and an average accuracy of 68% on complex tasks.
  arXiv  Detail & Related papers  (2023-08-07T18:03:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.