EPPCMinerBen: A Novel Benchmark for Evaluating Large Language Models on Electronic Patient-Provider Communication via the Patient Portal

• URL: http://arxiv.org/abs/2603.00028v1
• Date: Wed, 04 Feb 2026 04:41:11 GMT
• Title: EPPCMinerBen: A Novel Benchmark for Evaluating Large Language Models on Electronic Patient-Provider Communication via the Patient Portal
• Abstract summary: We introduce EPPCMinerBen, a benchmark for analyzing electronic patient-communication (EPPC) data.<n>EPPCMinerBen includes three sub-tasks: Code Classification, Subcode Classification, and Evidence Extraction.<n>Using 1,933 expert annotated sentences from 752 secure messages of the patient portal at Yale New Haven Hospital, it evaluates LLMs on identifying communicative intent and supportive text.
• Score: 3.045013061022589
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Effective communication in health care is critical for treatment outcomes and adherence. With patient-provider exchanges shifting to secure messaging, analyzing electronic patient-communication (EPPC) data is both essential and challenging. We introduce EPPCMinerBen, a benchmark for evaluating LLMs in detecting communication patterns and extracting insights from electronic patient-provider messages. EPPCMinerBen includes three sub-tasks: Code Classification, Subcode Classification, and Evidence Extraction. Using 1,933 expert annotated sentences from 752 secure messages of the patient portal at Yale New Haven Hospital, it evaluates LLMs on identifying communicative intent and supportive text. Benchmarks span various LLMs under zero-shot and few-shot settings, with data to be released via the NCI Cancer Data Service. Model performance varied across tasks and settings. Llama-3.1-70B led in evidence extraction (F1: 82.84%) and performed well in classification. Llama-3.3-70b-Instruct outperformed all models in code classification (F1: 67.03%). DeepSeek-R1-Distill-Qwen-32B excelled in subcode classification (F1: 48.25%), while sdoh-llama-3-70B showed consistent performance. Smaller models underperformed, especially in subcode classification (>30% F1). Few-shot prompting improved most tasks. Our results show that large, instruction-tuned models generally perform better in EPPCMinerBen tasks, particularly evidence extraction while smaller models struggle with fine-grained reasoning. EPPCMinerBen provides a benchmark for discourse-level understanding, supporting future work on model generalization and patient-provider communication analysis. Keywords: Electronic Patient-Provider Communication, Large language models, Data collection, Prompt engineering

Related papers

• PVminerLLM: Structured Extraction of Patient Voice from Patient-Generated Text using Large Language Models [0.0]
  Patient-generated text contains critical information about patients' lived experiences, social circumstances, and engagement in care.<n>These patient voice signals are rarely available in structured form, limiting their use in patient-centered outcomes research and clinical quality improvement.<n>We introduce PVminer, a benchmark for structured extraction of patient voice, and propose PVminerLLM, a supervised fine-tuned large language model.
  arXiv  Detail & Related papers  (2026-03-06T00:16:05Z)
• PVminer: A Domain-Specific Tool to Detect the Patient Voice in Patient Generated Data [2.6791290096531455]
  We introduce PVminer, a domain-adapted NLP framework for structuring patient voice in secure patient-provider communication.<n>PVminer formulates PV detection as a multi-label, multi-class prediction task integrating patient-specific BERT encoders.<n>PVminer achieves strong performance across hierarchical tasks and outperforms biomedical and clinical pre-trained baselines.
  arXiv  Detail & Related papers  (2026-02-24T18:10:00Z)
• Enhancing the Medical Context-Awareness Ability of LLMs via Multifaceted Self-Refinement Learning [49.559151128219725]
  Large language models (LLMs) have shown great promise in the medical domain, achieving strong performance on several benchmarks.<n>However, they continue to underperform in real-world medical scenarios, which often demand stronger context-awareness.<n>We propose Multifaceted Self-Refinement (MuSeR), a data-driven approach that enhances LLMs' context-awareness along three key facets.
  arXiv  Detail & Related papers  (2025-11-13T08:13:23Z)
• DispatchMAS: Fusing taxonomy and artificial intelligence agents for emergency medical services [49.70819009392778]
  Large Language Models (LLMs) and Multi-Agent Systems (MAS) offer opportunities to augment dispatchers.<n>This study aimed to develop and evaluate a taxonomy-grounded, multi-agent system for simulating realistic scenarios.
  arXiv  Detail & Related papers  (2025-10-24T08:01:21Z)
• MaLei at MultiClinSUM: Summarisation of Clinical Documents using Perspective-Aware Iterative Self-Prompting with LLMs [5.40185721303932]
  This paper presents the methodology we applied in the MultiClinSUM shared task for summarising clinical case documents.<n>We used an Iterative Self-Prompting technique on large language models (LLMs) by asking LLMs to generate task-specific prompts.<n>We used lexical and embedding space metrics, ROUGE and BERT-score, to guide the model fine-tuning with epochs.
  arXiv  Detail & Related papers  (2025-09-09T11:52:16Z)
• From Staff Messages to Actionable Insights: A Multi-Stage LLM Classification Framework for Healthcare Analytics [0.0]
  This paper presents a framework that identifies staff message topics and classifies messages by their reasons in a multi-class fashion.<n>The best-performing model was o3, achieving 78.4% weighted F1-score and 79.2% accuracy.<n>The proposed methodology incorporates data security measures and HIPAA compliance requirements essential for healthcare environments.
  arXiv  Detail & Related papers  (2025-09-05T20:15:52Z)
• Benchmarking Open-Source Large Language Models on Healthcare Text Classification Tasks [2.7729041396205014]
  This study evaluates the classification performance of five open-source large language models (LLMs)<n>We report precision, recall, and F1 scores with 95% confidence intervals for all model-task combinations.
  arXiv  Detail & Related papers  (2025-03-19T12:51:52Z)
• Enhanced Electronic Health Records Text Summarization Using Large Language Models [0.0]
  This project builds on prior work by creating a system that generates clinician-preferred, focused summaries. The proposed system leverages the Flan-T5 model to generate tailored EHR summaries based on clinician-specified topics.
  arXiv  Detail & Related papers  (2024-10-12T19:36:41Z)
• Prompt Perturbation Consistency Learning for Robust Language Models [47.021022978847036]
  Large language models (LLMs) have demonstrated impressive performance on a number of natural language processing tasks. We show that fine-tuning sufficiently large LLMs can produce IC-SF performance comparable to discriminative models. We propose an efficient mitigation approach, Prompt Perturbation Consistency Learning (PPCL), which works by regularizing the divergence between losses from clean and perturbed samples.
  arXiv  Detail & Related papers  (2024-02-24T15:00:58Z)
• MedAlign: A Clinician-Generated Dataset for Instruction Following with Electronic Medical Records [60.35217378132709]
  Large language models (LLMs) can follow natural language instructions with human-level fluency. evaluating LLMs on realistic text generation tasks for healthcare remains challenging. We introduce MedAlign, a benchmark dataset of 983 natural language instructions for EHR data.
  arXiv  Detail & Related papers  (2023-08-27T12:24:39Z)
• Few-Shot Cross-lingual Transfer for Coarse-grained De-identification of Code-Mixed Clinical Texts [56.72488923420374]
  Pre-trained language models (LMs) have shown great potential for cross-lingual transfer in low-resource settings. We show the few-shot cross-lingual transfer property of LMs for named recognition (NER) and apply it to solve a low-resource and real-world challenge of code-mixed (Spanish-Catalan) clinical notes de-identification in the stroke.
  arXiv  Detail & Related papers  (2022-04-10T21:46:52Z)
• DeepEnroll: Patient-Trial Matching with Deep Embedding and Entailment Prediction [67.91606509226132]
  Clinical trials are essential for drug development but often suffer from expensive, inaccurate and insufficient patient recruitment. DeepEnroll is a cross-modal inference learning model to jointly encode enrollment criteria (tabular data) into a shared latent space for matching inference.
  arXiv  Detail & Related papers  (2020-01-22T17:51:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.