Curriculum-Driven 3D CT Report Generation via Language-Free Visual Grafting and Zone-Constrained Compression

• URL: http://arxiv.org/abs/2603.23308v1
• Date: Tue, 24 Mar 2026 15:13:30 GMT
• Title: Curriculum-Driven 3D CT Report Generation via Language-Free Visual Grafting and Zone-Constrained Compression
• Abstract summary: We present Ker-VLJEPA-3B, a four-phase curriculum learning framework for free-text report generation from thoracic CT volumes.<n>A phased training curriculum adapts a Llama 3.2 3B decoder to ground its output in visual features from a frozen, self-supervised encoder.<n>Our visual backbone (LeJEPA ViT-Large) is trained via self-supervised joint-embedding prediction on unlabeled CTs, without text supervision.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Automated radiology report generation from 3D computed tomography (CT) volumes is challenging due to extreme sequence lengths, severe class imbalance, and the tendency of large language models (LLMs) to ignore visual tokens in favor of linguistic priors. We present Ker-VLJEPA-3B, a four-phase curriculum learning framework for free-text report generation from thoracic CT volumes. A phased training curriculum progressively adapts a Llama 3.2 3B decoder to ground its output in visual features from a frozen, self-supervised encoder. Our visual backbone (LeJEPA ViT-Large) is trained via self-supervised joint-embedding prediction on unlabeled CTs, without text supervision. Unlike contrastive models (CLIP, BiomedCLIP), this language-free backbone yields modality-pure representations. Vision-language alignment is deferred to the curriculum's bridge and generation phases. This modality-agnostic design can integrate any self-supervised encoder into an LLM without paired text during foundation training. Methodological innovations include: (1) zone-constrained cross-attention compressing slice embeddings into 32 spatially-grounded visual tokens; (2) PCA whitening of anisotropic LLM embeddings; (3) a positive-findings-only strategy eliminating posterior collapse; (4) warm bridge initialization transferring projection weights; and (5) selective cross-attention freezing with elastic weight consolidation to prevent catastrophic forgetting. Evaluated on the CT-RATE benchmark (2,984 validation volumes, 18 classes), Ker-VLJEPA-3B achieves a macro F1 of 0.429, surpassing the state-of-the-art (U-VLM, macro F1 = 0.414) by 3.6%, and reaching 0.448 (+8.2%) with threshold optimization. Ablation studies confirm 56.6% of generation quality derives from patient-specific visual content. Code and weights are available.

Related papers

• Architecture-Dependent Causal Transfer of Activation States Across Large Language Models [51.56484100374058]
  Internal language activation states can be transferred causally between different large language models.<n>Injecting projected activations into a target model during generation produces a statistically significant, preregistered causal effect.<n>We conclude that end-to-end activation-state transfer between LLMs, as currently implemented, is architecture-dependent rather than universal.
  arXiv  Detail & Related papers  (2026-08-17T09:53:27Z)
• MOSS-VL Technical Report [66.58420246189313]
  We present MOSS-VL, an open vision-language model family that treats real-time interaction as a first-class capability.<n>It is co-designed across the stack: the language decoder attends to vision only through gated cross-attention.<n> Offline, MOSS-VL-Instruct is competitive at comparable scale and leads temporal-reasoning video sets.
  arXiv  Detail & Related papers  (2026-08-15T05:12:53Z)
• Resolution Meets Reduction: Efficient Visual Context for 3D Radiology Report Generation [7.073662229057507]
  Vision-language models offer a promising path toward automating radiology report generation, but applying them to full 3D CT poses substantial computational challenges.<n>Modern foundation vision encoders (VEs) can produce tens of thousands of vision tokens per scan.<n>How this vision-token budget should be allocated across input field of view, spatial resolution, and vision-to-language projection remains an open design question.
  arXiv  Detail & Related papers  (2026-08-09T13:57:50Z)
• SCALPEL: Semantic Cross-modal Alignment via LLM-Powered Encoder Learning for Medical Vision-Language Representation [5.487826115955642]
  Vision-language pre-training serves as a cornerstone for medical multimodal representation learning.<n>We propose textbfSCALPEL, a textbfSemantic textbfCross-modal textbfAlignment framework via textbfLLM-textbfPowered textbfEncoder textbfLearning.
  arXiv  Detail & Related papers  (2026-07-29T13:12:32Z)
• Disease-Centric Vision-Language Pretraining with Hybrid Visual Encoding for 3D Computed Tomography [75.7789001619107]
  Vision-language pre-training holds great promise for general-purpose medical AI.<n>We propose a tailored framework featuring three key components.<n>A diagnosis-aware prompt strategy employs real clinical phrases and aggregated disease prototypes to bridge the pre-training-inference gap.
  arXiv  Detail & Related papers  (2026-06-24T08:24:45Z)
• DALE-CT: Depth-Aware Foundation Models for Computed Tomography [0.0]
  In this work, we explore 2D slice-based architectures as a flexible alternative to native 3D models for processing volumetric CT data.<n>To enhance representation quality, we developed a novel 3D depth-aware pre-training strategy anchored by dense auxiliary supervision from both automated anatomical masks and human-annotated abnormalities.<n>This performance demonstrates near-parity with state-of-the-art 3D vision-language models, achieved entirely from scratch with significantly less data and no textual supervision.
  arXiv  Detail & Related papers  (2026-06-05T18:39:05Z)
• PRIME: Prototype-Driven Multimodal Pretraining for Cancer Prognosis with Missing Modalities [86.63247982275396]
  PRIME is a missing-aware multimodal self-supervised pretraining framework.<n>It learns robust and transferable representations from partially observed cohorts.<n>We evaluate PRIME on The Cancer Genome Atlas with label-free pretraining on 32 cancer types.
  arXiv  Detail & Related papers  (2026-04-05T21:14:27Z)
• MedPruner: Training-Free Hierarchical Token Pruning for Efficient 3D Medical Image Understanding in Vision-Language Models [59.180043227905294]
  We propose MedPruner, a training-free and model-agnostic hierarchical token pruning framework for efficient 3D medical image understanding.<n>We show that MedPruner enables models such as MedGemma to maintain or even exceed their original performance while retaining fewer than 5% of visual tokens.
  arXiv  Detail & Related papers  (2026-03-12T07:37:00Z)
• U-VLM: Hierarchical Vision Language Modeling for Report Generation [20.09433657986766]
  We propose U-VLM, which enables hierarchical vision-language modeling in both training and architecture.<n>U-VLM achieves state-of-the-art performance on CT-RATE and AbdomenAtlas 3.0 using only a 0.1B decoder trained from scratch.
  arXiv  Detail & Related papers  (2026-02-28T05:43:11Z)
• SCALE-VLP: Soft-Weighted Contrastive Volumetric Vision-Language Pre-training with Spatial-Knowledge Semantics [31.49282177777596]
  Existing approaches often treat scans as independent 2D slices, compromising spatial coherence and underutilizing clinical semantics.<n>We propose a soft-weighted contrastive vision-language-training framework that integrates spatial semantics and domain-aware representations.<n>This yields structurally consistent and semantically grounded representations underg supervision, demonstrating cross-task transferability (retrieval, report generation, and classification) and cross-domain generalizability with consistent gains without further fine-tuning.
  arXiv  Detail & Related papers  (2025-11-04T21:03:17Z)
• Exploring the Capabilities of LLM Encoders for Image-Text Retrieval in Chest X-rays [8.019362739504087]
  Vision-language pretraining has advanced image-text alignment, yet progress in radiology remains constrained by the heterogeneity of clinical reports.<n>We ask whether large language model (LLM) encoders can provide robust clinical representations that transfer across diverse styles.<n>We introduce LLM2VEC4CXR, a domain-adapted encoder for chest X-ray reports, and LLM2CLIP4CXR, a dual-tower framework that couples this encoder with a vision backbone.
  arXiv  Detail & Related papers  (2025-09-17T09:44:59Z)
• More performant and scalable: Rethinking contrastive vision-language pre-training of radiology in the LLM era [7.5669441185108015]
  Large Language Models (LLMs) can facilitate large-scale supervised pre-training.<n>LLMs can extract diagnostic labels from radiology reports with remarkable precision.<n>We show that supervised pre-training fundamentally improves contrastive vision-language alignment.
  arXiv  Detail & Related papers  (2025-09-16T15:27:14Z)
• VELVET-Med: Vision and Efficient Language Pre-training for Volumetric Imaging Tasks in Medicine [11.993301266706139]
  We propose a vision-language pre-training framework, termed as textbfVELVET-Med, specifically designed for limited volumetric data such as 3D CT and associated radiology reports.<n>Our approach seeks to uncover rich spatial and semantic relationships embedded in volumetric medical images and corresponding clinical narratives.<n>The resulting encoders exhibit strong transferability, achieving state-of-the-art performance across a wide range of downstream tasks.
  arXiv  Detail & Related papers  (2025-08-16T17:08:43Z)
• Compile Scene Graphs with Reinforcement Learning [69.36723767339001]
  Next-token prediction is the fundamental principle for training large language models (LLMs)<n>We introduce R1-SGG, a multimodal LLM (M-LLM) initially trained via supervised fine-tuning (SFT) on the scene graph dataset.<n>We design a set of graph-centric rewards, including three recall-based variants -- Hard Recall, Hard Recall+Relax, and Soft Recall.
  arXiv  Detail & Related papers  (2025-04-18T10:46:22Z)
• BrainECHO: Semantic Brain Signal Decoding through Vector-Quantized Spectrogram Reconstruction for Whisper-Enhanced Text Generation [48.20672677492805]
  Current EEG/MEG-to-text decoding systems suffer from three key limitations.<n>BrainECHO is a multi-stage framework that employs decoupled representation learning.<n>BrainECHO demonstrates robustness across sentence, session, and subject-independent conditions.
  arXiv  Detail & Related papers  (2024-10-19T04:29:03Z)
• Affinity-Graph-Guided Contractive Learning for Pretext-Free Medical Image Segmentation with Minimal Annotation [55.325956390997]
  This paper proposes an affinity-graph-guided semi-supervised contrastive learning framework (Semi-AGCL) for medical image segmentation. The framework first designs an average-patch-entropy-driven inter-patch sampling method, which can provide a robust initial feature space. With merely 10% of the complete annotation set, our model approaches the accuracy of the fully annotated baseline, manifesting a marginal deviation of only 2.52%.
  arXiv  Detail & Related papers  (2024-10-14T10:44:47Z)
• Learning to Decompose Visual Features with Latent Textual Prompts [140.2117637223449]
  We propose Decomposed Feature Prompting (DeFo) to improve vision-language models. Our empirical study shows DeFo's significance in improving the vision-language models.
  arXiv  Detail & Related papers  (2022-10-09T15:40:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.