MedSAD-CLIP: Supervised CLIP with Token-Patch Cross-Attention for Medical Anomaly Detection and Segmentation

• URL: http://arxiv.org/abs/2603.17325v1
• Date: Wed, 18 Mar 2026 03:37:44 GMT
• Title: MedSAD-CLIP: Supervised CLIP with Token-Patch Cross-Attention for Medical Anomaly Detection and Segmentation
• Abstract summary: Medical anomaly detection (MAD) and segmentation play a critical role in assisting clinical diagnosis.<n>Recent CLIP-based studies are promising for anomaly detection in zero-/few-shot settings.<n>We study supervised adaptation of CLIP for MAD under a realistic clinical setting.
• Score: 1.055294253870608
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Medical anomaly detection (MAD) and segmentation play a critical role in assisting clinical diagnosis by identifying abnormal regions in medical images and localizing pathological regions. Recent CLIP-based studies are promising for anomaly detection in zero-/few-shot settings, and typically rely on global representations and weak supervision, often producing coarse localization and limited segmentation quality. In this work, we study supervised adaptation of CLIP for MAD under a realistic clinical setting where a limited yet meaningful amount of labeled abnormal data is available. Our model MedSAD-CLIP leverages fine-grained text-visual cues via the Token-Patch Cross-Attention(TPCA) to improve lesion localization while preserving the generalization capability of CLIP representations. Lightweight image adapters and learnable prompt tokens efficiently adapt the pretrained CLIP encoder to the medical domain while preserving its rich semantic alignment. Furthermore, a Margin-based image-text Contrastive Loss is designed to enhance global feature discrimination between normal and abnormal representations. Extensive experiments on four diverse benchmarks-Brain, Retina, Lung, and Breast datasets-demonstrate the effectiveness of our approach, achieving superior performance in both pixel-level segmentation and image-level classification over state-of-the-art methods. Our results highlight the potential of supervised CLIP adaptation as a unified and scalable paradigm for medical anomaly understanding. Code will be made available at https://github.com/thuy4tbn99/MedSAD-CLIP

Related papers

• Towards Continual Visual Anomaly Detection in the Medical Domain [11.262875405161417]
  Visual Anomaly Detection (VAD) seeks to identify abnormal images and precisely localize the corresponding anomalous regions.<n>Continual Learning (CL) provides a framework to incrementally adapt models while preserving previously acquired knowledge.<n>This study explores for the first time the application of VAD models in a CL scenario for the medical field.
  arXiv  Detail & Related papers  (2025-08-25T13:28:15Z)
• Multimodal Causal-Driven Representation Learning for Generalizable Medical Image Segmentation [56.52520416420957]
  We propose Multimodal Causal-Driven Representation Learning (MCDRL) to tackle domain generalization in medical image segmentation.<n>MCDRL consistently outperforms competing methods, yielding superior segmentation accuracy and exhibiting robust generalizability.
  arXiv  Detail & Related papers  (2025-08-07T03:41:41Z)
• AF-CLIP: Zero-Shot Anomaly Detection via Anomaly-Focused CLIP Adaptation [8.252046294696585]
  We propose AF-CLIP (Anomaly-Focused CLIP) by dramatically enhancing its visual representations to focus on local defects.<n>Our approach introduces a lightweight adapter that emphasizes anomaly-relevant patterns in visual features.<n>Our method is also extended to few-shot scenarios by extra memory banks.
  arXiv  Detail & Related papers  (2025-07-26T13:34:38Z)
• MADPOT: Medical Anomaly Detection with CLIP Adaptation and Partial Optimal Transport [14.023527193608142]
  We propose a novel approach combining visual adapters and prompt learning with Partial Optimal Transport (POT) and contrastive learning (CL) to improve CLIP's adaptability to medical images.<n>Our method achieves state-of-the-art results in few-shot, zero-shot, and cross-dataset scenarios without synthetic data or memory banks.
  arXiv  Detail & Related papers  (2025-07-09T10:45:52Z)
• MadCLIP: Few-shot Medical Anomaly Detection with CLIP [14.023527193608142]
  An innovative few-shot anomaly detection approach is presented, leveraging the pre-trained CLIP model for medical data.<n>A dual-branch design is proposed to separately capture normal and abnormal features through learnable adapters.<n>To improve semantic alignment, learnable text prompts are employed to link visual features.
  arXiv  Detail & Related papers  (2025-06-30T12:56:17Z)
• Crane: Context-Guided Prompt Learning and Attention Refinement for Zero-Shot Anomaly Detection [50.343419243749054]
  Anomaly detection is critical in fields such as medical diagnostics and industrial defect detection.<n> CLIP's coarse-grained image-text alignment limits localization and detection performance for fine-grained anomalies.<n>Crane improves the state-of-the-art ZSAD from 2% to 28%, at both image and pixel levels, while remaining competitive in inference speed.
  arXiv  Detail & Related papers  (2025-04-15T10:42:25Z)
• Visual Prompt Engineering for Vision Language Models in Radiology [0.17183214167143138]
  Contrastive Language-Image Pretraining (CLIP) offers a promising solution by enabling zero-shot classification through multimodal large-scale pretraining.<n>While CLIP effectively captures global image content, radiology requires a more localized focus on specific pathology regions to enhance both interpretability and diagnostic accuracy.<n>We explore the potential of incorporating visual cues into zero-shot classification, embedding visual markers, such as arrows, bounding boxes, and circles, directly into radiological images to guide model attention.
  arXiv  Detail & Related papers  (2024-08-28T13:53:27Z)
• Adapting Visual-Language Models for Generalizable Anomaly Detection in Medical Images [68.42215385041114]
  This paper introduces a novel lightweight multi-level adaptation and comparison framework to repurpose the CLIP model for medical anomaly detection. Our approach integrates multiple residual adapters into the pre-trained visual encoder, enabling a stepwise enhancement of visual features across different levels. Our experiments on medical anomaly detection benchmarks demonstrate that our method significantly surpasses current state-of-the-art models.
  arXiv  Detail & Related papers  (2024-03-19T09:28:19Z)
• Rethinking Semi-Supervised Medical Image Segmentation: A Variance-Reduction Perspective [51.70661197256033]
  We propose ARCO, a semi-supervised contrastive learning framework with stratified group theory for medical image segmentation. We first propose building ARCO through the concept of variance-reduced estimation and show that certain variance-reduction techniques are particularly beneficial in pixel/voxel-level segmentation tasks. We experimentally validate our approaches on eight benchmarks, i.e., five 2D/3D medical and three semantic segmentation datasets, with different label settings.
  arXiv  Detail & Related papers  (2023-02-03T13:50:25Z)
• PCA: Semi-supervised Segmentation with Patch Confidence Adversarial Training [52.895952593202054]
  We propose a new semi-supervised adversarial method called Patch Confidence Adrial Training (PCA) for medical image segmentation. PCA learns the pixel structure and context information in each patch to get enough gradient feedback, which aids the discriminator in convergent to an optimal state. Our method outperforms the state-of-the-art semi-supervised methods, which demonstrates its effectiveness for medical image segmentation.
  arXiv  Detail & Related papers  (2022-07-24T07:45:47Z)
• Cross-modal Clinical Graph Transformer for Ophthalmic Report Generation [116.87918100031153]
  We propose a Cross-modal clinical Graph Transformer (CGT) for ophthalmic report generation (ORG) CGT injects clinical relation triples into the visual features as prior knowledge to drive the decoding procedure. Experiments on the large-scale FFA-IR benchmark demonstrate that the proposed CGT is able to outperform previous benchmark methods.
  arXiv  Detail & Related papers  (2022-06-04T13:16:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.