Automated Histopathology Report Generation via Pyramidal Feature Extraction and the UNI Foundation Model

• URL: http://arxiv.org/abs/2602.16422v1
• Date: Wed, 18 Feb 2026 12:55:20 GMT
• Title: Automated Histopathology Report Generation via Pyramidal Feature Extraction and the UNI Foundation Model
• Abstract summary: We propose a hierarchical vision language framework that combines a frozen pathology foundation model with a Transformer decoder for report generation.<n>To make WSI processing tractable, we perform multi resolution pyramidal patch selection and remove background and artifacts.<n>We tokenize the output using BioGPT to better represent biomedical terminology.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Generating diagnostic text from histopathology whole slide images (WSIs) is challenging due to the gigapixel scale of the input and the requirement for precise, domain specific language. We propose a hierarchical vision language framework that combines a frozen pathology foundation model with a Transformer decoder for report generation. To make WSI processing tractable, we perform multi resolution pyramidal patch selection (downsampling factors 2^3 to 2^6) and remove background and artifacts using Laplacian variance and HSV based criteria. Patch features are extracted with the UNI Vision Transformer and projected to a 6 layer Transformer decoder that generates diagnostic text via cross attention. To better represent biomedical terminology, we tokenize the output using BioGPT. Finally, we add a retrieval based verification step that compares generated reports with a reference corpus using Sentence BERT embeddings; if a high similarity match is found, the generated report is replaced with the retrieved ground truth reference to improve reliability.

Related papers

• Layout-Guided Controllable Pathology Image Generation with In-Context Diffusion Transformers [57.54843029965778]
  Controllable pathology image synthesis requires reliable regulation of spatial layout, tissue morphology, and semantic detail.<n>We propose In-Context Diffusion Transformer (IC-DiT), a layout-aware generative model that incorporates spatial layouts, textual descriptions, and visual embeddings into a unified diffusion transformer.<n>IC-DiT achieves higher fidelity, stronger spatial controllability, and better diagnostic consistency than existing methods.
  arXiv  Detail & Related papers  (2026-03-11T06:14:11Z)
• BUSTR: Breast Ultrasound Text Reporting with a Descriptor-Aware Vision-Language Model [0.0]
  Automated radiology report generation (RRG) for breast ultrasound (BUS) is limited by the lack of paired image-report datasets.<n>We propose BUSTR, a multitask vision-language framework that generates BUS reports without requiring paired image-report supervision.
  arXiv  Detail & Related papers  (2025-11-26T01:22:29Z)
• M2OST: Many-to-one Regression for Predicting Spatial Transcriptomics from Digital Pathology Images [16.19308597273405]
  We propose M2OST, a many-to-one regression Transformer that can accommodate the hierarchical structure of pathology images.<n>Unlike traditional models that are trained with one-to-one image-label pairs, M2OST uses multiple images from different levels of the digital pathology image to jointly predict the gene expressions in their common corresponding spot.<n>M2OST can achieve state-of-the-art performance with fewer parameters and floating-point operations (FLOPs)
  arXiv  Detail & Related papers  (2024-09-23T15:06:37Z)
• M2ORT: Many-To-One Regression Transformer for Spatial Transcriptomics Prediction from Histopathology Images [17.158450092707042]
  M2ORT is a many-to-one regression Transformer that can accommodate the hierarchical structure of pathology images. We have tested M2ORT on three public ST datasets and the experimental results show that M2ORT can achieve state-of-the-art performance.
  arXiv  Detail & Related papers  (2024-01-19T10:37:27Z)
• Automatic Report Generation for Histopathology images using pre-trained Vision Transformers and BERT [1.0819408603463427]
  We show that using an existing pre-trained Vision Transformer (ViT) to encode 4096x4096 sized patches of the Whole Slide Image (WSI) and a pre-trained Bidirectional Representations from Transformers (BERT) model for report generation, we can build a performant and portable report generation mechanism. Our method allows us to not only generate and evaluate captions that describe the image, but also helps us classify the image into tissue types and the gender of the patient as well.
  arXiv  Detail & Related papers  (2023-12-03T15:56:09Z)
• WsiCaption: Multiple Instance Generation of Pathology Reports for Gigapixel Whole-Slide Images [5.960501267687475]
  We investigate how to generate pathology reports given whole slide images. We curated the largest WSI-text dataset (PathText) On the model end, we propose the multiple instance generative model (MI-Gen)
  arXiv  Detail & Related papers  (2023-11-27T05:05:41Z)
• Domain Adaptive Synapse Detection with Weak Point Annotations [63.97144211520869]
  We present AdaSyn, a framework for domain adaptive synapse detection with weak point annotations. In the WASPSYN challenge at I SBI 2023, our method ranks the 1st place.
  arXiv  Detail & Related papers  (2023-08-31T05:05:53Z)
• Hierarchical Transformer for Survival Prediction Using Multimodality Whole Slide Images and Genomics [63.76637479503006]
  Learning good representation of giga-pixel level whole slide pathology images (WSI) for downstream tasks is critical. This paper proposes a hierarchical-based multimodal transformer framework that learns a hierarchical mapping between pathology images and corresponding genes. Our architecture requires fewer GPU resources compared with benchmark methods while maintaining better WSI representation ability.
  arXiv  Detail & Related papers  (2022-11-29T23:47:56Z)
• BatchFormerV2: Exploring Sample Relationships for Dense Representation Learning [88.82371069668147]
  BatchFormerV2 is a more general batch Transformer module, which enables exploring sample relationships for dense representation learning. BatchFormerV2 consistently improves current DETR-based detection methods by over 1.3%.
  arXiv  Detail & Related papers  (2022-04-04T05:53:42Z)
• AlignTransformer: Hierarchical Alignment of Visual Regions and Disease Tags for Medical Report Generation [50.21065317817769]
  We propose an AlignTransformer framework, which includes the Align Hierarchical Attention (AHA) and the Multi-Grained Transformer (MGT) modules. Experiments on the public IU-Xray and MIMIC-CXR datasets show that the AlignTransformer can achieve results competitive with state-of-the-art methods on the two datasets.
  arXiv  Detail & Related papers  (2022-03-18T13:43:53Z)
• REPLICA: Enhanced Feature Pyramid Network by Local Image Translation and Conjunct Attention for High-Resolution Breast Tumor Detection [6.112883009328882]
  We call our method enhanced featuREsynthesis network by Local Image translation and Conjunct Attention, or REPLICA. We use a convolutional autoencoder as a generator to create new images by injecting objects into images via local Pyramid and reconstruction of their features extracted in hidden layers. Then due to the larger number of simulated images, we use a visual transformer to enhance outputs of each ResNet layer that serve as inputs to a feature pyramid network.
  arXiv  Detail & Related papers  (2021-11-22T21:33:02Z)
• TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation [78.01570371790669]
  Medical image segmentation is an essential prerequisite for developing healthcare systems. On various medical image segmentation tasks, the u-shaped architecture, also known as U-Net, has become the de-facto standard. We propose TransUNet, which merits both Transformers and U-Net, as a strong alternative for medical image segmentation.
  arXiv  Detail & Related papers  (2021-02-08T16:10:50Z)
• Text Mining to Identify and Extract Novel Disease Treatments From Unstructured Datasets [56.38623317907416]
  We use Google Cloud to transcribe podcast episodes of an NPR radio show. We then build a pipeline for systematically pre-processing the text. Our model successfully identified that Omeprazole can help treat heartburn.
  arXiv  Detail & Related papers  (2020-10-22T19:52:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.