Fully Open Meditron: An Auditable Pipeline for Clinical LLMs

• URL: http://arxiv.org/abs/2605.16215v1
• Date: Fri, 15 May 2026 17:29:08 GMT
• Title: Fully Open Meditron: An Auditable Pipeline for Clinical LLMs
• Abstract summary: We introduce Fully Open Meditron, the first fully open pipeline for building clinical decision support systems.<n>It consists of a clinician-audited training corpus, a reproducible data construction and training framework, and a use-aligned evaluation protocol.<n>Apertus-70B-MeditronFO improves +6.6 points over its base (47.2% to 53.8%) on aggregate medical benchmarks, establishing a new FO SoTA.
• Score: 0.8322575744308107
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Clinical decision support systems (CDSS) require scrutable, auditable pipelines that enable rigorous, reproducible validation. Yet current LLM-based CDSS remain largely opaque. Most "open" models are open-weight only, releasing parameters while withholding the data provenance, curation procedures, and generation pipelines that determine model behavior. Fully Open (FO) models, which expose the complete training stack end-to-end, do not currently exist in medicine. We introduce Fully Open Meditron, the first fully open pipeline for building LLM-CDSS, comprising a clinician-audited training corpus, a reproducible data construction and training framework, and a use-aligned evaluation protocol. The corpus unifies eight public medical QA datasets into a normalized conversational format and expands coverage with three clinician-vetted synthetic extensions: exam-style QA, guideline-grounded QA derived from 46,469 clinical practice guidelines, and clinical vignettes. The pipeline enforces system-wide decontamination, gold-label resampling of teacher generations, and end-to-end validation by a four-physician panel. We evaluate using an LLM-as-a-judge protocol over expert-written clinical vignettes, calibrated against 204 human raters. We apply the recipe to five FO base models (Apertus-70B/8B-Instruct, OLMo-2-32B-SFT, EuroLLM-22B/9B-Instruct). All MeditronFO variants are preferred over their bases. Apertus-70B-MeditronFO improves +6.6 points over its base (47.2% to 53.8%) on aggregate medical benchmarks, establishing a new FO SoTA. Gemma-3-27B-MeditronFO is preferred over MedGemma in 58.6% of LLM-as-a-judge comparisons and outperforms it on HealthBench (58% vs 55.9%). These results show that fully open pipelines can achieve state-of-the-art domain-specific performance without sacrificing auditability or reproducibility.

Related papers

• NebulaExp-8B: An Empirical Post-Training Pipeline via Full-Scale Ablation Research [5.287668610154937]
  Post-training alignment determines the reasoning and human preference following capabilities of large language models.<n>This work presents NebulaExp, a fully transparent, ablation-driven post-training pipeline built on Qwen3-8B-base.
  arXiv  Detail & Related papers  (2026-06-25T07:03:25Z)
• Medmarks: A Comprehensive Open-Source LLM Benchmark Suite for Medical Tasks [12.131182745922843]
  We introduce Medmarks, a fully open-source evaluation suite with 30 benchmarks spanning question answering, information extraction, medical calculations, and open-ended clinical reasoning.<n>We perform a systematic evaluation of 61 models across 71 configurations using verifiable metrics and LLM-as-a-Judge.
  arXiv  Detail & Related papers  (2026-05-02T12:29:03Z)
• Evaluating Small Open LLMs for Medical Question Answering: A Practical Framework [0.0]
  Large language models (LLMs) in medical question answering demands more than high average accuracy.<n>Online health communities such as Reddit have become a primary source of medical information for millions of users.<n>We present a practical, open-source evaluation framework for assessing small, locally-deployable open-weight LLMs on medical question answering.
  arXiv  Detail & Related papers  (2026-04-12T08:56:15Z)
• A Federated and Parameter-Efficient Framework for Large Language Model Training in Medicine [59.78991974851707]
  Large language models (LLMs) have demonstrated strong performance on medical benchmarks, including question answering and diagnosis.<n>Most medical LLMs are trained on data from a single institution, which faces limitations in generalizability and safety in heterogeneous systems.<n>We introduce the model-agnostic and parameter-efficient federated learning framework for adapting LLMs to medical applications.
  arXiv  Detail & Related papers  (2026-01-29T18:48:21Z)
• Optimizing Medical Question-Answering Systems: A Comparative Study of Fine-Tuned and Zero-Shot Large Language Models with RAG Framework [0.0]
  We present a retrieval-augmented generation (RAG) based medical QA system that combines domain-specific knowledge retrieval with open-source LLMs to answer medical questions.<n>We fine-tune two state-of-the-art open LLMs (LLaMA2 and Falcon) using Low-Rank Adaptation (LoRA) for efficient domain specialization.<n>Our fine-tuned LLaMA2 model achieves 71.8% accuracy on PubMedQA, substantially improving over the 55.4% zero-shot baseline.
  arXiv  Detail & Related papers  (2025-12-05T16:38:47Z)
• EchoBench: Benchmarking Sycophancy in Medical Large Vision-Language Models [82.43729208063468]
  Recent benchmarks for medical Large Vision-Language Models (LVLMs) emphasize leaderboard accuracy, overlooking reliability and safety.<n>We study sycophancy -- models' tendency to uncritically echo user-provided information.<n>We introduce EchoBench, a benchmark to systematically evaluate sycophancy in medical LVLMs.
  arXiv  Detail & Related papers  (2025-09-24T14:09:55Z)
• From Scores to Steps: Diagnosing and Improving LLM Performance in Evidence-Based Medical Calculations [45.414878840652115]
  Large language models (LLMs) have demonstrated promising performance on medical benchmarks.<n>However, their ability to perform medical calculations remains underexplored and poorly evaluated.<n>In this work, we revisit medical calculation evaluation with a stronger focus on clinical trustworthiness.
  arXiv  Detail & Related papers  (2025-09-20T09:10:26Z)
• A Modular Approach for Clinical SLMs Driven by Synthetic Data with Pre-Instruction Tuning, Model Merging, and Clinical-Tasks Alignment [46.776978552161395]
  Small language models (SLMs) offer a cost-effective alternative to large language models such as GPT-4.<n>SLMs offer a cost-effective alternative, but their limited capacity requires biomedical domain adaptation.<n>We propose a novel framework for adapting SLMs into high-performing clinical models.
  arXiv  Detail & Related papers  (2025-05-15T21:40:21Z)
• LEME: Open Large Language Models for Ophthalmology with Advanced Reasoning and Clinical Validation [29.913581347375256]
  Large language models (LLMs) offer a promising path to reduce documentation workload and support clinical decision-making.<n>Here, we present LEME, a suite of open-weight LLMs developed through a two-stage process.<n>LEME was evaluated on five curated zero-shot benchmarks spanning tasks such as patient QA, consultation, and treatment planning.
  arXiv  Detail & Related papers  (2024-10-01T02:43:54Z)
• DataComp-LM: In search of the next generation of training sets for language models [200.5293181577585]
  DataComp for Language Models (DCLM) is a testbed for controlled dataset experiments with the goal of improving language models.<n>We provide a standardized corpus of 240T tokens extracted from Common Crawl, effective pretraining recipes based on the OpenLM framework, and a broad suite of 53 downstream evaluations.<n>Participants in the DCLM benchmark can experiment with data curation strategies such as deduplication, filtering, and data mixing at model scales ranging from 412M to 7B parameters.
  arXiv  Detail & Related papers  (2024-06-17T17:42:57Z)
• Towards a clinically accessible radiology foundation model: open-access and lightweight, with automated evaluation [113.5002649181103]
  Training open-source small multimodal models (SMMs) to bridge competency gaps for unmet clinical needs in radiology. For training, we assemble a large dataset of over 697 thousand radiology image-text pairs. For evaluation, we propose CheXprompt, a GPT-4-based metric for factuality evaluation, and demonstrate its parity with expert evaluation. The inference of LlaVA-Rad is fast and can be performed on a single V100 GPU in private settings, offering a promising state-of-the-art tool for real-world clinical applications.
  arXiv  Detail & Related papers  (2024-03-12T18:12:02Z)
• MEDITRON-70B: Scaling Medical Pretraining for Large Language Models [91.25119823784705]
  Large language models (LLMs) can potentially democratize access to medical knowledge. We release MEDITRON: a suite of open-source LLMs with 7B and 70B parameters adapted to the medical domain.
  arXiv  Detail & Related papers  (2023-11-27T18:49:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.