OctoMed: Data Recipes for State-of-the-Art Multimodal Medical Reasoning

• URL: http://arxiv.org/abs/2511.23269v1
• Date: Fri, 28 Nov 2025 15:21:51 GMT
• Title: OctoMed: Data Recipes for State-of-the-Art Multimodal Medical Reasoning
• Authors: Timothy Ossowski, Sheng Zhang, Qianchu Liu, Guanghui Qin, Reuben Tan, Tristan Naumann, Junjie Hu, Hoifung Poon,
• Abstract summary: We investigate strategies for training and data curation to develop a robust multimodal reasoning model in the medical domain.<n>We scale experiments to a dataset of over 8 million examples and 6.8 billion response tokens.<n>Our results indicate that curating a high-quality, diverse training dataset with varying structured reasoning trace lengths enables the fine-tuned model to self-calibrate its reasoning trajectory lengths.
• Score: 26.144214758141874
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: High-quality and carefully curated data is a cornerstone of training medical large language models, as it directly impacts both generalization and robustness to unseen clinical tasks. We investigate strategies for training and data curation to develop a robust multimodal reasoning model in the medical domain. Our work focuses on supervised fine-tuning (SFT) and explores data recipes that leverage structured reasoning traces. Using our proposed data recipe, we scale experiments to a dataset of over 8 million examples and 6.8 billion response tokens, achieving state-of-the-art performance among open-source models across diverse out-of-distribution medical benchmark tasks. Our results further indicate that curating a high-quality, diverse training dataset with varying structured reasoning trace lengths enables the fine-tuned model to self-calibrate its reasoning trajectory lengths based on the downstream task, without explicit supervision. We present key insights, describe the data curation strategy, and outline next steps toward developing robust medical vision-language reasoning system.

Related papers

• Integrating Genomics into Multimodal EHR Foundation Models [56.31910745104141]
  This paper introduces an innovative EHR foundation model that integrates Polygenic Risk Scores (PRS) as a foundational data modality.<n>The framework aims to learn complex relationships between clinical data and genetic predispositions.<n>This approach is pivotal for unlocking new insights into disease prediction, proactive health management, risk stratification, and personalized treatment strategies.
  arXiv  Detail & Related papers  (2025-10-24T15:56:40Z)
• MedAlign: A Synergistic Framework of Multimodal Preference Optimization and Federated Meta-Cognitive Reasoning [52.064286116035134]
  We develop MedAlign, a framework to ensure visually accurate LVLM responses for Medical Visual Question Answering (Med-VQA)<n>We first propose a multimodal Direct Preference Optimization (mDPO) objective to align preference learning with visual context.<n>We then design a Retrieval-Aware Mixture-of-Experts (RA-MoE) architecture that utilizes image and text similarity to route queries to a specialized and context-augmented LVLM.
  arXiv  Detail & Related papers  (2025-10-24T02:11:05Z)
• Multimodal Foundation Models for Early Disease Detection [0.0]
  We present a foundation model that consolidates diverse patient data through an attention-based transformer framework.<n>The architecture is made for pretraining on many tasks, which makes it easy to adapt to new diseases and datasets with little extra work.<n>We provide an experimental strategy that uses benchmark datasets in oncology, cardiology, and neurology, with the goal of testing early detection tasks.
  arXiv  Detail & Related papers  (2025-10-02T11:12:57Z)
• GEMeX-RMCoT: An Enhanced Med-VQA Dataset for Region-Aware Multimodal Chain-of-Thought Reasoning [60.03671205298294]
  Medical visual question answering aims to support clinical decision-making by enabling models to answer natural language questions based on medical images.<n>Current methods still suffer from limited answer reliability and poor interpretability.<n>This work first proposes a Region-Aware Multimodal Chain-of-Thought dataset, in which the process of producing an answer is preceded by a sequence of intermediate reasoning steps.
  arXiv  Detail & Related papers  (2025-06-22T08:09:58Z)
• Efficient Medical VIE via Reinforcement Learning [10.713109515157475]
  Visual Information Extraction (VIE) converts unstructured document images into structured formats like, structured formats like, critical for medical applications like report analysis and online consultations.<n>Traditional methods rely on OCR and language models, while end-to-end multimodal models offer direct generation.<n>We base our approach on the Reinforcement Learning with Verifiable Rewards (RLVR) framework to address these challenges using only 100 annotated samples.
  arXiv  Detail & Related papers  (2025-06-16T11:10:25Z)
• Confounder-Aware Medical Data Selection for Fine-Tuning Pretrained Vision Models [11.10271979935354]
  We present a confounder-aware medical data selection approach for medical dataset curation.<n>Our approach first identifies confounding variables within data and then develops a distance-based data selection strategy.<n>We validate the superiority of our approach through extensive experiments across diverse medical imaging modalities.
  arXiv  Detail & Related papers  (2025-03-02T05:50:25Z)
• STLLaVA-Med: Self-Training Large Language and Vision Assistant for Medical Question-Answering [58.79671189792399]
  STLLaVA-Med is designed to train a policy model capable of auto-generating medical visual instruction data. We validate the efficacy and data efficiency of STLLaVA-Med across three major medical Visual Question Answering (VQA) benchmarks.
  arXiv  Detail & Related papers  (2024-06-28T15:01:23Z)
• Medical Vision-Language Pre-Training for Brain Abnormalities [96.1408455065347]
  We show how to automatically collect medical image-text aligned data for pretraining from public resources such as PubMed. In particular, we present a pipeline that streamlines the pre-training process by initially collecting a large brain image-text dataset. We also investigate the unique challenge of mapping subfigures to subcaptions in the medical domain.
  arXiv  Detail & Related papers  (2024-04-27T05:03:42Z)
• Learnable Weight Initialization for Volumetric Medical Image Segmentation [66.3030435676252]
  We propose a learnable weight-based hybrid medical image segmentation approach. Our approach is easy to integrate into any hybrid model and requires no external training data. Experiments on multi-organ and lung cancer segmentation tasks demonstrate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2023-06-15T17:55:05Z)
• P-Transformer: A Prompt-based Multimodal Transformer Architecture For Medical Tabular Data [2.4688646371447898]
  We propose PTransformer, a underlinePrompt-based multimodal underlineTransformer architecture designed specifically for medical tabular data.<n>The framework efficiently encodes diverse modalities from both structured and unstructured data into a harmonized language semantic space.<n>PTransformer demonstrated the improvements with 10.9%/11.0% on RMSE/MAE, 0.5%/2.2% on RMSE/MAE, and 1.6%/0.8% on BACC/AUROC compared to state-of-the-art (SOTA) baselines in predictability.
  arXiv  Detail & Related papers  (2023-03-30T14:25:44Z)
• MMLN: Leveraging Domain Knowledge for Multimodal Diagnosis [10.133715767542386]
  We propose a knowledge-driven and data-driven framework for lung disease diagnosis. We formulate diagnosis rules according to authoritative clinical medicine guidelines and learn the weights of rules from text data. A multimodal fusion consisting of text and image data is designed to infer the marginal probability of lung disease.
  arXiv  Detail & Related papers  (2022-02-09T04:12:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.