LLM-Based Data Generation and Clinical Skills Evaluation for Low-Resource French OSCEs

• URL: http://arxiv.org/abs/2604.08126v1
• Date: Thu, 09 Apr 2026 11:46:39 GMT
• Title: LLM-Based Data Generation and Clinical Skills Evaluation for Low-Resource French OSCEs
• Abstract summary: In France, the organization of training sessions is limited by human and logistical constraints.<n>Recent advances in Natural Language Processing (NLP) and Large Language Models (LLMs) now offer the opportunity to automatically evaluate medical interviews.<n>We introduce a controlled pipeline that produces synthetic doctor-patient interview transcripts guided by scenario-specific evaluation criteria.
• Score: 7.636286614517115
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Objective Structured Clinical Examinations (OSCEs) are the standard method for assessing medical students' clinical and communication skills through structured patient interviews. In France, however, the organization of training sessions is limited by human and logistical constraints, restricting students' access to repeated practice and structured feedback. Recent advances in Natural Language Processing (NLP) and Large Language Models (LLMs) now offer the opportunity to automatically evaluate such medical interviews, thereby alleviating the need for human examiners during training. Yet, real French OSCE annotated transcripts remain extremely scarce, limiting reproducible research and reliable benchmarking. To address these challenges, we investigate the use of LLMs for both generating and evaluating French OSCE dialogues in a low-resource context. We introduce a controlled pipeline that produces synthetic doctor-patient interview transcripts guided by scenario-specific evaluation criteria, combining ideal and perturbed performances to simulate varying student skill levels. The resulting dialogues are automatically silver-labeled through an LLM-assisted framework supporting adjustable evaluation strictness. Benchmarking multiple open-source and proprietary LLMs shows that mid-size models ($\le$32B parameters) achieve accuracies comparable to GPT-4o ($\sim$90\%) on synthetic data, highlighting the feasibility of locally deployable, privacy-preserving evaluation systems for medical education.

Related papers

• A French OSCE Dialogue Dataset and Controllable Virtual Patient System for Clinical Training [4.13196819783484]
  We introduce a French OSCE dialogue dataset comprising 240 student-patient training interactions.<n>We build upon it a controllable LLM-based pipeline to generate synthetic OSCE dialogues.<n>We also propose a multi-level evaluation framework assessing patient simulation quality, student performance, and linguistic quality.
  arXiv  Detail & Related papers  (2026-06-26T18:21:58Z)
• EHRBench: An Automated and Reliable EHR-based Benchmark for Clinical Decision Making with LLMs [51.129595320595094]
  Clinical decision-making (CDM) is central to real-world clinical, where clinicians infer diagnoses, select treatments, or anticipate future health outcomes under incomplete evidence.<n>LLM models are increasingly used to support these decisions due to strong language capabilities, broad biomedical knowledge, and efficiency.<n>The reliability of LLMs on real-world clinical decision tasks remains insufficiently understood.
  arXiv  Detail & Related papers  (2026-05-28T22:38:26Z)
• A Federated and Parameter-Efficient Framework for Large Language Model Training in Medicine [59.78991974851707]
  Large language models (LLMs) have demonstrated strong performance on medical benchmarks, including question answering and diagnosis.<n>Most medical LLMs are trained on data from a single institution, which faces limitations in generalizability and safety in heterogeneous systems.<n>We introduce the model-agnostic and parameter-efficient federated learning framework for adapting LLMs to medical applications.
  arXiv  Detail & Related papers  (2026-01-29T18:48:21Z)
• ClinDEF: A Dynamic Evaluation Framework for Large Language Models in Clinical Reasoning [58.01333341218153]
  We propose ClinDEF, a dynamic framework for assessing clinical reasoning in LLMs through simulated diagnostic dialogues.<n>Our method generates patient cases and facilitates multi-turn interactions between an LLM-based doctor and an automated patient agent.<n>Experiments show that ClinDEF effectively exposes critical clinical reasoning gaps in state-of-the-art LLMs.
  arXiv  Detail & Related papers  (2025-12-29T12:58:58Z)
• TRUST: An LLM-Based Dialogue System for Trauma Understanding and Structured Assessments [8.618945530676614]
  This study aims to bridge the gap in mental healthcare accessibility by developing an LLM-powered dialogue system that replicates clinician behavior.<n>We introduce TRUST, a framework of cooperative LLM modules capable of conducting formal diagnostic interviews and assessments for PTSD.<n>We develop a patient simulation approach based on real-life interview transcripts to replace time-consuming and costly manual testing by clinicians.
  arXiv  Detail & Related papers  (2025-04-30T17:58:06Z)
• Med-CoDE: Medical Critique based Disagreement Evaluation Framework [72.42301910238861]
  The reliability and accuracy of large language models (LLMs) in medical contexts remain critical concerns.<n>Current evaluation methods often lack robustness and fail to provide a comprehensive assessment of LLM performance.<n>We propose Med-CoDE, a specifically designed evaluation framework for medical LLMs to address these challenges.
  arXiv  Detail & Related papers  (2025-04-21T16:51:11Z)
• LlaMADRS: Prompting Large Language Models for Interview-Based Depression Assessment [75.44934940580112]
  This study introduces LlaMADRS, a novel framework leveraging open-source Large Language Models (LLMs) to automate depression severity assessment.<n>We employ a zero-shot prompting strategy with carefully designed cues to guide the model in interpreting and scoring transcribed clinical interviews.<n>Our approach, tested on 236 real-world interviews, demonstrates strong correlations with clinician assessments.
  arXiv  Detail & Related papers  (2025-01-07T08:49:04Z)
• PALLM: Evaluating and Enhancing PALLiative Care Conversations with Large Language Models [10.258261180305439]
  Large language models (LLMs) offer a new approach to assessing complex communication metrics. LLMs offer the potential to advance the field through integration into passive sensing and just-in-time intervention systems. This study explores LLMs as evaluators of palliative care communication quality, leveraging their linguistic, in-context learning, and reasoning capabilities.
  arXiv  Detail & Related papers  (2024-09-23T16:39:12Z)
• Towards Automatic Evaluation for LLMs' Clinical Capabilities: Metric, Data, and Algorithm [15.627870862369784]
  Large language models (LLMs) are gaining increasing interests to improve clinical efficiency for medical diagnosis. We propose an automatic evaluation paradigm tailored to assess the LLMs' capabilities in delivering clinical services.
  arXiv  Detail & Related papers  (2024-03-25T06:17:54Z)
• Automatic Interactive Evaluation for Large Language Models with State Aware Patient Simulator [21.60103376506254]
  Large Language Models (LLMs) have demonstrated remarkable proficiency in human interactions. This paper introduces the Automated Interactive Evaluation (AIE) framework and the State-Aware Patient Simulator (SAPS) AIE and SAPS provide a dynamic, realistic platform for assessing LLMs through multi-turn doctor-patient simulations.
  arXiv  Detail & Related papers  (2024-03-13T13:04:58Z)
• Attribute Structuring Improves LLM-Based Evaluation of Clinical Text Summaries [56.31117605097345]
  Large language models (LLMs) have shown the potential to generate accurate clinical text summaries, but still struggle with issues regarding grounding and evaluation.<n>Here, we explore a general mitigation framework using Attribute Structuring (AS), which structures the summary evaluation process.<n>AS consistently improves the correspondence between human annotations and automated metrics in clinical text summarization.
  arXiv  Detail & Related papers  (2024-03-01T21:59:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.