Resolution Meets Reduction: Efficient Visual Context for 3D Radiology Report Generation

• URL: http://arxiv.org/abs/2608.08713v1
• Date: Sun, 09 Aug 2026 13:57:50 GMT
• Title: Resolution Meets Reduction: Efficient Visual Context for 3D Radiology Report Generation
• Abstract summary: Vision-language models offer a promising path toward automating radiology report generation, but applying them to full 3D CT poses substantial computational challenges.<n>Modern foundation vision encoders (VEs) can produce tens of thousands of vision tokens per scan.<n>How this vision-token budget should be allocated across input field of view, spatial resolution, and vision-to-language projection remains an open design question.
• Score: 7.073662229057507
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-language models offer a promising path toward automating radiology report generation, but applying them to full 3D CT volumes poses substantial computational challenges. Modern foundation vision encoders (VEs) can produce tens of thousands of vision tokens per scan, making the visual sequence passed to the large language model (LLM) a primary computational bottleneck. Vision-to-language projectors can compress this sequence to reduce computation, but may discard clinically relevant detail; conversely, effective compression can accommodate higher-resolution inputs while keeping the downstream token count fixed. How this vision-token budget should be allocated across input field of view, spatial resolution, and vision-to-language projection therefore remains an open design question. We systematically evaluate four heterogeneous VEs (CNN- and ViT-based), five token-reducing projectors at up to 64x compression alongside a non-reducing MLP projector baseline, and five instruction-tuned LLMs (1.7B--4B) on two large-scale CT report datasets (CT-RATE and Merlin). At matched LLM token budgets, anatomy-guided region of interest cropping is the most consistent strategy, improving clinical macro F1 in 19 of 20 settings by +3.7 points on average for the 3D ViT Primus encoder and +1.1 for the slice-based 2D ViT Curia encoder. Increasing input resolution further is strongly projector-dependent: the PerceiverResampler, paired with higher-resolution Curia features, yields the strongest configuration in the resolution study on both datasets. Our best configurations achieve state-of-the-art clinical macro F1 on the test sets, reaching 49.5 on CT-RATE and 49.0 on Merlin. Code and models will be published upon publication.

Related papers

• MedFG-VQA: Low-Frequency Memory and Graph Attention for Lightweight Medical VQA [55.12170695219291]
  MedFG-VQA is a lightweight framework that augments DCT-based low-frequency features and employs graph-enhanced cross-attention for effective visual-textual alignment.<n>To address data scarcity, we construct SynMed-VQA, a large-scale synthetic dataset comprising over 2 million question-answer pairs across 9 imaging modalities and 10 major organs.
  arXiv  Detail & Related papers  (2026-08-27T09:16:14Z)
• Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Model [60.85752511708308]
  We present Mage-VL, an efficient streaming foundation model for real-time multimodal understanding and interaction.<n>At its core, our custom tokenizer, Mage-ViT, replaces uniform frame sampling by selectively encoding dynamic, entropy-rich regions.<n>We establish AI4AI data pipelines encompassing prompt-code joint optimization for captioning and AI-driven performance diagnosis.
  arXiv  Detail & Related papers  (2026-07-27T17:59:53Z)
• Disease-Centric Vision-Language Pretraining with Hybrid Visual Encoding for 3D Computed Tomography [75.7789001619107]
  Vision-language pre-training holds great promise for general-purpose medical AI.<n>We propose a tailored framework featuring three key components.<n>A diagnosis-aware prompt strategy employs real clinical phrases and aggregated disease prototypes to bridge the pre-training-inference gap.
  arXiv  Detail & Related papers  (2026-06-24T08:24:45Z)
• Simple Token-Efficient Vision-Language Model for Case-level Pathology Synoptic Report Generation [9.268049949671958]
  We present a simple token-efficient vision-language model for case-level synoptic report generation.<n>Our architecture follows a minimal three-component design: a frozen pathology patch, a lightweight two-layer vision-language aligner, and a large language model decoder.<n>Our approach achieves high ROUGE-L/METEOR/BLEU-4 scores while being substantially more efficient in memory and runtime.
  arXiv  Detail & Related papers  (2026-05-29T01:15:13Z)
• Curriculum-Driven 3D CT Report Generation via Language-Free Visual Grafting and Zone-Constrained Compression [0.0]
  We present Ker-VLJEPA-3B, a four-phase curriculum learning framework for free-text report generation from thoracic CT volumes.<n>A phased training curriculum adapts a Llama 3.2 3B decoder to ground its output in visual features from a frozen, self-supervised encoder.<n>Our visual backbone (LeJEPA ViT-Large) is trained via self-supervised joint-embedding prediction on unlabeled CTs, without text supervision.
  arXiv  Detail & Related papers  (2026-03-24T15:13:30Z)
• MedPruner: Training-Free Hierarchical Token Pruning for Efficient 3D Medical Image Understanding in Vision-Language Models [59.180043227905294]
  We propose MedPruner, a training-free and model-agnostic hierarchical token pruning framework for efficient 3D medical image understanding.<n>We show that MedPruner enables models such as MedGemma to maintain or even exceed their original performance while retaining fewer than 5% of visual tokens.
  arXiv  Detail & Related papers  (2026-03-12T07:37:00Z)
• Hulu-Med: A Transparent Generalist Model towards Holistic Medical Vision-Language Understanding [112.46150793476603]
  We introduce Hulu-Med, a transparent, generalist medical Vision-Language Model (VLM)<n>Hulu-Med is trained on a curated corpus of 16.7 million samples, spanning 12 major anatomical systems and 14 medical imaging modalities.<n>Hulu-Med surpasses existing open-source models on 27 of 30 benchmarks and outperforms proprietary systems such as GPT-4o on 16 benchmarks.
  arXiv  Detail & Related papers  (2025-10-09T17:06:42Z)
• DCFormer: Efficient 3D Vision-Language Modeling with Decomposed Convolutions [6.464464511743737]
  We introduce DCFormer, an efficient 3D image encoder that factorizes 3D convolutions into three parallel 1D convolutions along the depth, height, and width dimensions.<n>In zero-shot and fine-tuned detection of 18 pathologies, DCFormer consistently outperforms state-of-the-art 3D vision encoders.
  arXiv  Detail & Related papers  (2025-02-07T17:10:22Z)
• Argus: Benchmarking and Enhancing Vision-Language Models for 3D Radiology Report Generation [15.897686345011731]
  There has been no comprehensive benchmark for 3D radiograph report generation (3DRRG)<n>We curate **CT-3DRRG**, the largest **publicly** available 3D CT-report dataset, establishing a robust and diverse benchmark for evaluating VLM performance on 3DRRG.<n>We propose a comprehensive training recipe for building high-performing VLMs for 3DRRG, exploring key factors such as vision encoder pretraining strategies, visual token compression, and the impact of data & model scale.
  arXiv  Detail & Related papers  (2024-06-11T10:45:59Z)
• CT-GLIP: 3D Grounded Language-Image Pretraining with CT Scans and Radiology Reports for Full-Body Scenarios [53.94122089629544]
  We introduce CT-GLIP (Grounded Language-Image Pretraining with CT scans), a novel method that constructs organ-level image-text pairs to enhance multimodal contrastive learning. Our method, trained on a multimodal CT dataset comprising 44,011 organ-level vision-text pairs from 17,702 patients across 104 organs, demonstrates it can identify organs and abnormalities in a zero-shot manner using natural languages.
  arXiv  Detail & Related papers  (2024-04-23T17:59:01Z)
• Focused Decoding Enables 3D Anatomical Detection by Transformers [64.36530874341666]
  We propose a novel Detection Transformer for 3D anatomical structure detection, dubbed Focused Decoder. Focused Decoder leverages information from an anatomical region atlas to simultaneously deploy query anchors and restrict the cross-attention's field of view. We evaluate our proposed approach on two publicly available CT datasets and demonstrate that Focused Decoder not only provides strong detection results and thus alleviates the need for a vast amount of annotated data but also exhibits exceptional and highly intuitive explainability of results via attention weights.
  arXiv  Detail & Related papers  (2022-07-21T22:17:21Z)
• PAENet: A Progressive Attention-Enhanced Network for 3D to 2D Retinal Vessel Segmentation [0.0]
  3D to 2D retinal vessel segmentation is a challenging problem in Optical Coherence Tomography Angiography ( OCTA) images. We propose a Progressive Attention-Enhanced Network (PAENet) based on attention mechanisms to extract rich feature representation. Our proposed algorithm achieves state-of-the-art performance compared with previous methods.
  arXiv  Detail & Related papers  (2021-08-26T10:27:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.