MedCritical: Enhancing Medical Reasoning in Small Language Models via Self-Collaborative Correction

• URL: http://arxiv.org/abs/2509.23368v1
• Date: Sat, 27 Sep 2025 15:30:20 GMT
• Title: MedCritical: Enhancing Medical Reasoning in Small Language Models via Self-Collaborative Correction
• Authors: Xinchun Su, Chunxu Luo, Yixuan Li, Weidong Yang, Lipeng Ma,
• Abstract summary: Small language models often underperform compared to large language models like GPT-4 and Deepseek.<n>Recent knowledge distillation-based methods aim to address these issues through teacher-guided error correction.<n>We propose a novel two-stage framework, MedCritical, which uses a small language model fine-tuned by a large teacher model to play against itself.
• Score: 22.35140929464229
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In the field of medicine, complex reasoning tasks such as clinical diagnosis, treatment planning, and medical knowledge integration pose significant challenges, where small language models often underperform compared to large language models like GPT-4 and Deepseek. Recent knowledge distillation-based methods aim to address these issues through teacher-guided error correction, but this LLM as judge approach remains challenging in terms of cost, time, and efficiency. To circumvent this issue, we propose a novel two-stage framework, MedCritical, which uses a small language model fine-tuned by a large teacher model to play against itself. In the first stage, we extract high-level and detailed long-chain thought templates from the teacher model to guide the student model to generate more complex reasoning thoughts. In the second stage, we introduce direct preference optimization (DPO) through model self-iteration collaboration to enhance the reasoning ability of the student model by playing against the correction trajectory of the fine-tuned model during training. This model self-learning DPO approach teaches the student model to use its own error-driven insights to consolidate its skills and knowledge to solve complex problems, and achieves comparable results to traditional knowledge distillation methods using teacher models at a lower cost. Notably, our MedCritical 7B model outperforms the Taiyi and Huatuo-o1-7B models by 3.04\% and 10.12\% respectively on the CMExam benchmark, achieving new SOTA performance among 7B-class small models.

Related papers

• Matryoshka Model Learning for Improved Elastic Student Models [62.154536258259384]
  MatTA is a framework for training multiple accurate Student models using a novel Teacher-TA-Student recipe.<n>We demonstrate our method on GPT-2 Medium, a public model, and achieve relative improvements of over 24% on SAT Math and over 10% on the LAMBADA benchmark.
  arXiv  Detail & Related papers  (2025-05-29T10:54:58Z)
• CustomKD: Customizing Large Vision Foundation for Edge Model Improvement via Knowledge Distillation [57.91828170220308]
  We propose a knowledge distillation approach, CustomKD, that effectively leverages large vision foundation models (LVFMs) to enhance the performance of edge models.<n>Our simple yet effective CustomKD customizes the well-generalized features inherent in LVFMs to a given student model in order to reduce model discrepancies.
  arXiv  Detail & Related papers  (2025-03-23T23:53:08Z)
• SuperCorrect: Advancing Small LLM Reasoning with Thought Template Distillation and Self-Correction [89.56181323849512]
  SuperCorrect is a novel two-stage framework that uses a large teacher model to supervise and correct both the reasoning and reflection processes of a smaller student model.<n>In the first stage, we extract hierarchical high-level and detailed thought templates from the teacher model to guide the student model in eliciting more fine-grained reasoning thoughts.<n>In the second stage, we introduce cross-model collaborative direct preference optimization (DPO) to enhance the self-correction abilities of the student model.
  arXiv  Detail & Related papers  (2024-10-11T17:25:52Z)
• Exploring and Enhancing the Transfer of Distribution in Knowledge Distillation for Autoregressive Language Models [62.5501109475725]
  Knowledge distillation (KD) is a technique that compresses large teacher models by training smaller student models to mimic them. This paper introduces Online Knowledge Distillation (OKD), where the teacher network integrates small online modules to concurrently train with the student model. OKD achieves or exceeds the performance of leading methods in various model architectures and sizes, reducing training time by up to fourfold.
  arXiv  Detail & Related papers  (2024-09-19T07:05:26Z)
• Teaching Language Models to Self-Improve through Interactive Demonstrations [83.9421355808174]
  Self-improving ability of large language models has been shown to be absent and difficult to learn for smaller models. We introduce TriPosT, a training algorithm that endows smaller models with such self-improvement ability. We show that our approach can improve a LLaMA-7b's performance on math and reasoning tasks by up to 7.13%.
  arXiv  Detail & Related papers  (2023-10-20T14:11:04Z)
• Knowledge Distillation for Adaptive MRI Prostate Segmentation Based on Limit-Trained Multi-Teacher Models [4.711401719735324]
  Knowledge Distillation (KD) has been proposed as a compression method and an acceleration technology. KD is an efficient learning strategy that can transfer knowledge from a burdensome model to a lightweight model. We develop a KD-based deep model for prostate MRI segmentation in this work by combining features-based distillation with Kullback-Leibler divergence, Lovasz, and Dice losses.
  arXiv  Detail & Related papers  (2023-03-16T17:15:08Z)
• KDSM: An uplift modeling framework based on knowledge distillation and sample matching [2.036924568983982]
  Uplift modeling aims to estimate the treatment effect on individuals. Tree-based methods are adept at fitting increment and generalization, while neural-network-based models excel at predicting absolute value and precision. In this paper, we proposed an uplift modeling framework based on Knowledge Distillation and Sample Matching (KDSM)
  arXiv  Detail & Related papers  (2023-03-06T09:15:28Z)
• A multi-stage machine learning model on diagnosis of esophageal manometry [50.591267188664666]
  The framework includes deep-learning models at the swallow-level stage and feature-based machine learning models at the study-level stage. This is the first artificial-intelligence-style model to automatically predict CC diagnosis of HRM study from raw multi-swallow data.
  arXiv  Detail & Related papers  (2021-06-25T20:09:23Z)
• Collective Wisdom: Improving Low-resource Neural Machine Translation using Adaptive Knowledge Distillation [42.38435539241788]
  Scarcity of parallel sentence-pairs poses a significant hurdle for training high-quality Neural Machine Translation (NMT) models in bilingually low-resource scenarios. We propose an adaptive knowledge distillation approach to dynamically adjust the contribution of the teacher models during the distillation process. Experiments on transferring from a collection of six language pairs from IWSLT to five low-resource language-pairs from TED Talks demonstrate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2020-10-12T04:26:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.