DiffVP: Differential Visual Semantic Prompting for LLM-Based CT Report Generation

• URL: http://arxiv.org/abs/2603.17718v1
• Date: Wed, 18 Mar 2026 13:38:26 GMT
• Title: DiffVP: Differential Visual Semantic Prompting for LLM-Based CT Report Generation
• Abstract summary: We propose Differential Visual Prompting (DiffVP), which conditions report generation on explicit, high-level semantic scan-to-reference differences.<n>On two large-scale benchmarks, DiffVP consistently outperforms prior methods, improving the average BLEU-1-4 by +10.98 and +4.36, respectively.
• Score: 18.257492970454898
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While large language models (LLMs) have advanced CT report generation, existing methods typically encode 3D volumes holistically, failing to distinguish informative cues from redundant anatomical background. Inspired by radiological cognitive subtraction, we propose Differential Visual Prompting (DiffVP), which conditions report generation on explicit, high-level semantic scan-to-reference differences rather than solely on absolute visual features. DiffVP employs a hierarchical difference extractor to capture complementary global and local semantic discrepancies into a shared latent space, along with a difference-to-prompt generator that transforms these signals into learnable visual prefix tokens for LLM conditioning. These difference prompts serve as structured conditioning signals that implicitly suppress invariant anatomy while amplifying diagnostically relevant visual evidence, thereby facilitating accurate report generation without explicit lesion localization. On two large-scale benchmarks, DiffVP consistently outperforms prior methods, improving the average BLEU-1-4 by +10.98 and +4.36, respectively, and further boosts clinical efficacy on RadGenome-ChestCT (F1 score 0.421). All codes will be released at https://github.com/ArielTYH/DiffVP/.

Related papers

• Bridging Vision Foundation Model Priors with CLIP for Spatial-aware Few-shot Anomaly Detection in Medical Images [48.646448077284276]
  We propose Spatial-FAD, a framework for effective few-shot medical anomaly detection.<n>It improves lesion localization by combining VFM spatial priors with CLIP semantics.<n>Our method achieves an average improvement of over 11.4% in Dice score and 1.8% in AUC.
  arXiv  Detail & Related papers  (2026-09-11T05:21:01Z)
• Beyond Static Anchors: Bounded Prototype Conditioning for Language-Free Medical Anomaly Detection [52.986874008247554]
  We propose ReCAP, a language-free framework that replaces static anchors with input-conditioned visual prototypes.<n>ReCAP re-centers separated normal and abnormal prototypes for each image through a bounded gated modulation.<n>It achieves the best image-level AUROC on all zero-shot and 23 of 24 few-shot settings, and the best zero-shot pixel-level AUROC on all three segmentation datasets.
  arXiv  Detail & Related papers  (2026-08-01T04:46:55Z)
• Deep Residual Injection for Full-Spectrum Forensic Signal Perception in Multimodal Large Language Models [51.099179436350376]
  Multimodal large language models (MLLMs) have been increasingly adopted in forensics for their robust semantic understanding.<n>We propose Deep Visual Residual MLLM (Deep-VRM) to preserve early semantic processing while injecting artifact-specific visual signals as a residual path into an intermediate layer.<n>This enables later layers to jointly model semantic reasoning and signal-level forensic cues, and surprisingly, the model learns to adaptively leverage different levels of forensic signals depending on the input.
  arXiv  Detail & Related papers  (2026-06-14T16:05:45Z)
• Geometry-Aware Uncertainty Coresets for Robust Visual In-Context Learning in Histopathology [8.2867621159451]
  We propose GAUC, a training-free coreset selection method operating directly in the pre-trained multimodal embedding space.<n>On CRC-100K and MHIST across multiple open-source VLM architectures, GAUC consistently improves accuracy, calibration, and prompt gradient robustness.
  arXiv  Detail & Related papers  (2026-05-18T13:54:04Z)
• SPEGC: Continual Test-Time Adaptation via Semantic-Prompt-Enhanced Graph Clustering for Medical Image Segmentation [12.0982298854338]
  Continual Test-Time Adaptation (CTTA) aims to enable pre-trained models to adapt to continuously changing unlabeled domains.<n>We propose a CTTA via Semantic-Prompt-Enhanced Graph Clustering (SPEGC) for medical image segmentation.
  arXiv  Detail & Related papers  (2026-03-12T03:22:43Z)
• EVLF: Early Vision-Language Fusion for Generative Dataset Distillation [6.913041518668593]
  We introduce an Early Vision-Language Fusion (EVLF) method that aligns textual and visual embeddings at the transition between the encoder and the generative backbone.<n>EVLF is plug-and-play and can be easily integrated into any diffusion-based dataset distillation pipeline with an encoder.<n>Experiments demonstrate that EVLF generates semantically faithful and visually coherent synthetic data, yielding consistent improvements in downstream classification accuracy across varied settings.
  arXiv  Detail & Related papers  (2026-03-08T05:34:07Z)
• Deep Expert Injection for Anchoring Retinal VLMs with Domain-Specific Knowledge [66.67024684187915]
  Large Vision Language Models (LVLMs) show immense potential for automated ophthalmic diagnosis.<n>Their clinical deployment is severely hindered by lacking domain-specific knowledge.<n>EyExIn is a framework designed to anchor retinal VLMs with expert knowledge via a Deep Expert Injection mechanism.
  arXiv  Detail & Related papers  (2026-03-07T09:43:49Z)
• ZeroDiff++: Substantial Unseen Visual-semantic Correlation in Zero-shot Learning [42.855022683292276]
  We introduce metrics to quantify spuriousness for seen and unseen classes.<n>We propose ZeroDiff++, a diffusion-based generative framework.<n>Experiments on three ZSL benchmarks demonstrate that ZeroDiff++ not only achieves significant improvements over existing ZSL methods but also maintains robust performance even with scarce training data.
  arXiv  Detail & Related papers  (2026-02-12T20:52:07Z)
• LEAD: Layer-wise Expert-aligned Decoding for Faithful Radiology Report Generation [25.67810663907638]
  Radiology Report Generation (RRG) aims to produce accurate and coherent diagnostics from medical images.<n>Large vision language models (LVLM) exhibit hallucinations, generating plausible yet image-ungrounded pathological details.<n>We propose Layer-wise Expert-aligned Decoding (LEAD), a novel method to inherently modify the LVLM decoding trajectory.
  arXiv  Detail & Related papers  (2026-02-04T14:45:49Z)
• HAAF: Hierarchical Adaptation and Alignment of Foundation Models for Few-Shot Pathology Anomaly Detection [10.649984141835189]
  We propose the Hierarchical Adaptation and Alignment Framework (HAAF)<n>At its core is a novel Cross-Level Scaled Alignment mechanism that enforces a sequential calibration order.<n>A dual-branch inference strategy integrates semantic scores with geometric prototypes to ensure stability in few-shot settings.
  arXiv  Detail & Related papers  (2026-01-24T10:31:21Z)
• Crane: Context-Guided Prompt Learning and Attention Refinement for Zero-Shot Anomaly Detection [50.343419243749054]
  Anomaly detection is critical in fields such as medical diagnostics and industrial defect detection.<n> CLIP's coarse-grained image-text alignment limits localization and detection performance for fine-grained anomalies.<n>Crane improves the state-of-the-art ZSAD from 2% to 28%, at both image and pixel levels, while remaining competitive in inference speed.
  arXiv  Detail & Related papers  (2025-04-15T10:42:25Z)
• Contrastive Learning with Counterfactual Explanations for Radiology Report Generation [83.30609465252441]
  We propose a textbfCountertextbfFactual textbfExplanations-based framework (CoFE) for radiology report generation. Counterfactual explanations serve as a potent tool for understanding how decisions made by algorithms can be changed by asking what if'' scenarios. Experiments on two benchmarks demonstrate that leveraging the counterfactual explanations enables CoFE to generate semantically coherent and factually complete reports.
  arXiv  Detail & Related papers  (2024-07-19T17:24:25Z)
• Cross-Modal Causal Intervention for Medical Report Generation [107.76649943399168]
  Radiology Report Generation (RRG) is essential for computer-aided diagnosis and medication guidance.<n> generating accurate lesion descriptions remains challenging due to spurious correlations from visual-linguistic biases.<n>We propose a two-stage framework named CrossModal Causal Representation Learning (CMCRL)<n> Experiments on IU-Xray and MIMIC-CXR show that our CMCRL pipeline significantly outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-03-16T07:23:55Z)
• GSMFlow: Generation Shifts Mitigating Flow for Generalized Zero-Shot Learning [55.79997930181418]
  Generalized Zero-Shot Learning aims to recognize images from both the seen and unseen classes by transferring semantic knowledge from seen to unseen classes. It is a promising solution to take the advantage of generative models to hallucinate realistic unseen samples based on the knowledge learned from the seen classes. We propose a novel flow-based generative framework that consists of multiple conditional affine coupling layers for learning unseen data generation.
  arXiv  Detail & Related papers  (2022-07-05T04:04:37Z)
• Generalized Zero-Shot Learning via VAE-Conditioned Generative Flow [83.27681781274406]
  Generalized zero-shot learning aims to recognize both seen and unseen classes by transferring knowledge from semantic descriptions to visual representations. Recent generative methods formulate GZSL as a missing data problem, which mainly adopts GANs or VAEs to generate visual features for unseen classes. We propose a conditional version of generative flows for GZSL, i.e., VAE-Conditioned Generative Flow (VAE-cFlow)
  arXiv  Detail & Related papers  (2020-09-01T09:12:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.