RIHA: Report-Image Hierarchical Alignment for Radiology Report Generation

• URL: http://arxiv.org/abs/2604.27559v1
• Date: Thu, 30 Apr 2026 08:08:46 GMT
• Title: RIHA: Report-Image Hierarchical Alignment for Radiology Report Generation
• Abstract summary: Radiology report generation (RRG) has emerged as a promising approach to alleviate radiologists' workload and reduce human errors.<n>A key challenge in RRG is achieving fine-grained alignment between complex visual features and the structure of long-form radiology reports.<n>We propose RIHA, a novel end-to-end framework that performs multi-level alignment between radiological images and their corresponding reports.
• Score: 14.781219175876883
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Radiology report generation (RRG) has emerged as a promising approach to alleviate radiologists' workload and reduce human errors by automatically generating diagnostic reports from medical images. A key challenge in RRG is achieving fine-grained alignment between complex visual features and the hierarchical structure of long-form radiology reports. Although recent methods have improved image-text representation learning, they often treat reports as flat sequences, overlooking their structured sections and semantic hierarchies. This simplification hinders precise cross-modal alignment and weakens RRG accuracy. To address this challenge, we propose RIHA (Report-Image Hierarchical Alignment Transformer), a novel end-to-end framework that performs multi-level alignment between radiological images and their corresponding reports across paragraph, sentence, and word levels. This hierarchical alignment enables more precise cross-modal mapping, essential for capturing the nuanced semantics embedded in clinical narratives. Specifically, RIHA introduces a Visual Feature Pyramid (VFP) to extract multi-scale visual features and a Text Feature Pyramid (TFP) to represent multi-granularity textual structures. These components are integrated through a Cross-modal Hierarchical Alignment (CHA) module, leveraging optimal transport to effectively align visual and textual features across various levels. Furthermore, we incorporate Relative Positional Encoding (RPE) into the decoder to model spatial and semantic relationships among tokens, enhancing the token-level alignment between visual features and generated text. Extensive experiments on two benchmark chest X-ray datasets, IU-Xray and MIMIC-CXR, demonstrate that RIHA outperforms existing state-of-the-art models in both natural language generation and clinical efficacy metrics.

Related papers

• Layout-Guided Controllable Pathology Image Generation with In-Context Diffusion Transformers [57.54843029965778]
  Controllable pathology image synthesis requires reliable regulation of spatial layout, tissue morphology, and semantic detail.<n>We propose In-Context Diffusion Transformer (IC-DiT), a layout-aware generative model that incorporates spatial layouts, textual descriptions, and visual embeddings into a unified diffusion transformer.<n>IC-DiT achieves higher fidelity, stronger spatial controllability, and better diagnostic consistency than existing methods.
  arXiv  Detail & Related papers  (2026-03-11T06:14:11Z)
• Structure Observation Driven Image-Text Contrastive Learning for Computed Tomography Report Generation [51.509572354327986]
  This work introduces a novel two-stage (structure- and report-learning) framework tailored for Computed Tomography Report Generation (CTRG)<n>In the first stage, a set of learnable structure-specific visual queries observe corresponding structures in a CT image. The resulting observation tokens are contrasted with structure-specific textual features extracted from the accompanying radiology report with a structure-wise image-text contrastive loss.<n>In the second stage, the visual structure queries are frozen and used to select the critical image patch embeddings depicting each anatomical structure, minimizing distractions from irrelevant areas while reducing memory consumption.
  arXiv  Detail & Related papers  (2026-03-05T07:07:07Z)
• SimCroP: Radiograph Representation Learning with Similarity-driven Cross-granularity Pre-training [25.763109982379703]
  We propose a Similarity-Driven Cross-Granularity Pre-training framework on chest CTs.<n>It combines similarity-driven alignment and cross-granularity fusion to improve radiograph interpretation.<n>SimCroP is pre-trained on a large-scale paired CT-reports dataset and validated on image classification and segmentation tasks.
  arXiv  Detail & Related papers  (2025-09-10T06:20:53Z)
• Bridging Vision and Language: Optimal Transport-Driven Radiology Report Generation via LLMs [4.273291010923853]
  Large language models (LLMs) have demonstrated remarkable performance across various domains.<n>We propose Optimal Transport-Driven Radiology Report Generation (OTDRG) to align image features with disease labels extracted from reports.<n>OTDRG achieves state-of-the-art performance in both natural language generation (NLG) and clinical efficacy (CE) metrics.
  arXiv  Detail & Related papers  (2025-07-05T05:48:48Z)
• Anatomical Attention Alignment representation for Radiology Report Generation [10.887845017050477]
  Anatomical Attention Alignment Network (A3Net) is a framework that enhance visual-textual understanding by constructing hyper-visual representations.<n>Our approach integrates a knowledge dictionary of anatomical structures with patch-level visual features, enabling the model to effectively associate image regions with their corresponding anatomical entities.<n> Experimental results on IU X-Ray and MIMIC-CXR datasets demonstrate that A3Net significantly improves both visual perception and text generation quality.
  arXiv  Detail & Related papers  (2025-05-12T15:54:50Z)
• RadIR: A Scalable Framework for Multi-Grained Medical Image Retrieval via Radiology Report Mining [64.66825253356869]
  We propose a novel methodology that leverages dense radiology reports to define image-wise similarity ordering at multiple granularities.<n>We construct two comprehensive medical imaging retrieval datasets: MIMIC-IR for Chest X-rays and CTRATE-IR for CT scans.<n>We develop two retrieval systems, RadIR-CXR and model-ChestCT, which demonstrate superior performance in traditional image-image and image-report retrieval tasks.
  arXiv  Detail & Related papers  (2025-03-06T17:43:03Z)
• SERPENT-VLM : Self-Refining Radiology Report Generation Using Vision Language Models [9.390882250428305]
  Radiology Report Generation (R2Gen) demonstrates how Multi-modal Large Language Models (MLLMs) can automate the creation of accurate and coherent radiological reports. Existing methods often hallucinate details in text-based reports that don't accurately reflect the image content. We introduce a novel strategy, which improves the R2Gen task by integrating a self-refining mechanism into the MLLM framework.
  arXiv  Detail & Related papers  (2024-04-27T13:46:23Z)
• Radiology Report Generation Using Transformers Conditioned with Non-imaging Data [55.17268696112258]
  This paper proposes a novel multi-modal transformer network that integrates chest x-ray (CXR) images and associated patient demographic information. The proposed network uses a convolutional neural network to extract visual features from CXRs and a transformer-based encoder-decoder network that combines the visual features with semantic text embeddings of patient demographic information.
  arXiv  Detail & Related papers  (2023-11-18T14:52:26Z)
• Unify, Align and Refine: Multi-Level Semantic Alignment for Radiology Report Generation [48.723504098917324]
  We propose an Unify, Align and then Refine (UAR) approach to learn multi-level cross-modal alignments. We introduce three novel modules: Latent Space Unifier, Cross-modal Representation Aligner and Text-to-Image Refiner. Experiments and analyses on IU-Xray and MIMIC-CXR benchmark datasets demonstrate the superiority of our UAR against varied state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-03-28T12:42:12Z)
• Cross-Modal Causal Intervention for Medical Report Generation [107.76649943399168]
  Radiology Report Generation (RRG) is essential for computer-aided diagnosis and medication guidance.<n> generating accurate lesion descriptions remains challenging due to spurious correlations from visual-linguistic biases.<n>We propose a two-stage framework named CrossModal Causal Representation Learning (CMCRL)<n> Experiments on IU-Xray and MIMIC-CXR show that our CMCRL pipeline significantly outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-03-16T07:23:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.