OntoBook: Ontology-Grounded Synthetic Textbooks for Medical Encoder Pretraining

• URL: http://arxiv.org/abs/2607.18927v1
• Date: Tue, 21 Jul 2026 10:11:10 GMT
• Title: OntoBook: Ontology-Grounded Synthetic Textbooks for Medical Encoder Pretraining
• Abstract summary: We present OntoBook, a method that converts medical ontology structure into signal encoder language models.<n>Our approach has three stages: random walks through graphs capture hierarchical and causal relations between medical codes.<n>A large model language reformulates these walks into fluent textbook-style prose, and the resulting text is used to train ModernmBERT.
• Score: 2.745009293909444
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present OntoBook, a method that converts medical ontology structure into pretraining signal for encoder language models. Our approach has three stages: random walks through ontology graphs capture hierarchical and causal relations between medical codes, a large language model reformulates these walks into fluent textbook-style prose, and the resulting text is used to train ModernCamemBERT, a 149M-parameter French encoder, with two objectives on the same data: masked language modeling and relation prediction between code pairs. On three French medical coding benchmarks (FRACCO, Cantemist-FR, Distemist-FR), OntoBook achieves significant improvements over MLM-only pretraining, with +2.5 micro-F1 on FRACCO and +8.0 micro-F1 on Distemist. We find that alignment between objectives is necessary: misaligned training, where each task uses different data, causes a 30-point degradation. We release 1.3 million LLM-reformulated medical textbooks across three French ontologies (CIM-10, CCAM, ATC) and pretrained model checkpoints.

Related papers

• The Capacity of Thought: Benchmarking Llama 3.2 in Semantic fMRI Neural Language Decoding and Improving the Huth Encoding-Model Baseline [0.0]
  We improve the Huth et al. ridge regression encoding pipeline through expanded voxel selection.<n>We introduce fMRIFlamingo, which maps BOLD activity to a frozen Llama-3.2-1B.<n>Despite achieving 42.86% Top-1 accuracy on a 1-in-100 ranking task, well above chance, a blind control ablation with zeroed fMRI inputs yields near-identical scores.
  arXiv  Detail & Related papers  (2026-07-13T18:56:17Z)
• U-VLM: Hierarchical Vision Language Modeling for Report Generation [20.09433657986766]
  We propose U-VLM, which enables hierarchical vision-language modeling in both training and architecture.<n>U-VLM achieves state-of-the-art performance on CT-RATE and AbdomenAtlas 3.0 using only a 0.1B decoder trained from scratch.
  arXiv  Detail & Related papers  (2026-02-28T05:43:11Z)
• ExGra-Med: Extended Context Graph Alignment for Medical Vision-Language Models [95.47808515575382]
  ExGra-Med is a novel framework for vision-language integration in medical AI.<n>It aligns images, instruction responses, and extended captions in the latent space, advancing semantic grounding and cross-modal coherence.<n>It matches LLaVA-Med's performance using just 10% of the pre-training data, achieving a 20.13% gain on VQA-RAD and approaching full-data performance.
  arXiv  Detail & Related papers  (2024-10-03T15:52:03Z)
• Customizing General-Purpose Foundation Models for Medical Report Generation [64.31265734687182]
  The scarcity of labelled medical image-report pairs presents great challenges in the development of deep and large-scale neural networks. We propose customizing off-the-shelf general-purpose large-scale pre-trained models, i.e., foundation models (FMs) in computer vision and natural language processing.
  arXiv  Detail & Related papers  (2023-06-09T03:02:36Z)
• CodeGen2: Lessons for Training LLMs on Programming and Natural Languages [116.74407069443895]
  We unify encoder and decoder-based models into a single prefix-LM. For learning methods, we explore the claim of a "free lunch" hypothesis. For data distributions, the effect of a mixture distribution and multi-epoch training of programming and natural languages on model performance is explored.
  arXiv  Detail & Related papers  (2023-05-03T17:55:25Z)
• Automated Medical Coding on MIMIC-III and MIMIC-IV: A Critical Review and Replicability Study [60.56194508762205]
  We reproduce, compare, and analyze state-of-the-art automated medical coding machine learning models. We show that several models underperform due to weak configurations, poorly sampled train-test splits, and insufficient evaluation. We present the first comprehensive results on the newly released MIMIC-IV dataset using the reproduced models.
  arXiv  Detail & Related papers  (2023-04-21T11:54:44Z)
• Vision-Language Modelling For Radiological Imaging and Reports In The Low Data Regime [70.04389979779195]
  This paper explores training medical vision-language models (VLMs) where the visual and language inputs are embedded into a common space. We explore several candidate methods to improve low-data performance, including adapting generic pre-trained models to novel image and text domains. Using text-to-image retrieval as a benchmark, we evaluate the performance of these methods with variable sized training datasets of paired chest X-rays and radiological reports.
  arXiv  Detail & Related papers  (2023-03-30T18:20:00Z)
• Multi-label Few-shot ICD Coding as Autoregressive Generation with Prompt [7.554528566861559]
  This study transforms this multi-label classification task into an autoregressive generation task. Instead of directly predicting the high dimensional space of ICD codes, our model generates the lower dimension of text descriptions. Experiments on MIMIC-III-few show that our model performs with a marco F1 30.2, which substantially outperforms the previous MIMIC-III-full SOTA model.
  arXiv  Detail & Related papers  (2022-11-24T22:10:50Z)
• Z-Code++: A Pre-trained Language Model Optimized for Abstractive Summarization [108.09419317477986]
  Z-Code++ is a new pre-trained language model optimized for abstractive text summarization. The model is first pre-trained using text corpora for language understanding, and then is continually pre-trained on summarization corpora for grounded text generation. Our model is parameter-efficient in that it outperforms the 600x larger PaLM-540B on XSum, and the finetuned 200x larger GPT3-175B on SAMSum.
  arXiv  Detail & Related papers  (2022-08-21T01:00:54Z)
• Does the Magic of BERT Apply to Medical Code Assignment? A Quantitative Study [2.871614744079523]
  It is not clear if pretrained models are useful for medical code prediction without further architecture engineering. We propose a hierarchical fine-tuning architecture to capture interactions between distant words and adopt label-wise attention to exploit label information. Contrary to current trends, we demonstrate that a carefully trained classical CNN outperforms attention-based models on a MIMIC-III subset with frequent codes.
  arXiv  Detail & Related papers  (2021-03-11T07:23:45Z)
• An Explainable CNN Approach for Medical Codes Prediction from Clinical Text [1.7746314978241657]
  We develop CNN-based methods for automatic ICD coding based on clinical text from intensive care unit (ICU) stays. We come up with the Shallow and Wide Attention convolutional Mechanism (SWAM), which allows our model to learn local and low-level features for each label.
  arXiv  Detail & Related papers  (2021-01-14T02:05:34Z)
• Explicit Alignment Objectives for Multilingual Bidirectional Encoders [111.65322283420805]
  We present a new method for learning multilingual encoders, AMBER (Aligned Multilingual Bi-directional EncodeR) AMBER is trained on additional parallel data using two explicit alignment objectives that align the multilingual representations at different granularities. Experimental results show that AMBER obtains gains of up to 1.1 average F1 score on sequence tagging and up to 27.3 average accuracy on retrieval over the XLMR-large model.
  arXiv  Detail & Related papers  (2020-10-15T18:34:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.