AdaCLIP: Adapting CLIP with Hybrid Learnable Prompts for Zero-Shot Anomaly Detection

• URL: http://arxiv.org/abs/2407.15795v1
• Date: Mon, 22 Jul 2024 16:52:37 GMT
• Title: AdaCLIP: Adapting CLIP with Hybrid Learnable Prompts for Zero-Shot Anomaly Detection
• Authors: Yunkang Cao, Jiangning Zhang, Luca Frittoli, Yuqi Cheng, Weiming Shen, Giacomo Boracchi,
• Abstract summary: This study introduces AdaCLIP for the ZSAD task, leveraging a pre-trained vision-language model (VLM), CLIP. AdaCLIP incorporates learnable prompts into CLIP and optimize them through training on auxiliary annotated anomaly detection data. Experiments conducted across 14 real-world anomaly detection datasets from industrial and medical domains indicate that AdaCLIP outperforms other ZSAD methods.
• Score: 14.916862007773341
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Zero-shot anomaly detection (ZSAD) targets the identification of anomalies within images from arbitrary novel categories. This study introduces AdaCLIP for the ZSAD task, leveraging a pre-trained vision-language model (VLM), CLIP. AdaCLIP incorporates learnable prompts into CLIP and optimizes them through training on auxiliary annotated anomaly detection data. Two types of learnable prompts are proposed: static and dynamic. Static prompts are shared across all images, serving to preliminarily adapt CLIP for ZSAD. In contrast, dynamic prompts are generated for each test image, providing CLIP with dynamic adaptation capabilities. The combination of static and dynamic prompts is referred to as hybrid prompts, and yields enhanced ZSAD performance. Extensive experiments conducted across 14 real-world anomaly detection datasets from industrial and medical domains indicate that AdaCLIP outperforms other ZSAD methods and can generalize better to different categories and even domains. Finally, our analysis highlights the importance of diverse auxiliary data and optimized prompts for enhanced generalization capacity. Code is available at https://github.com/caoyunkang/AdaCLIP.

Related papers

• SDVPT: Semantic-Driven Visual Prompt Tuning for Open-World Object Counting [70.49268117587562]
  We propose a plug-and-play Semantic-Driven Visual Prompt Tuning framework (SDVPT) that transfers knowledge from the training set to unseen categories. During inference, we dynamically synthesize the visual prompts for unseen categories based on the semantic correlation between unseen and training categories.
  arXiv  Detail & Related papers  (2025-04-24T09:31:08Z)
• GenCLIP: Generalizing CLIP Prompts for Zero-shot Anomaly Detection [13.67800822455087]
  A key challenge in ZSAD is learning general prompts stably and utilizing them effectively. We propose GenCLIP, a novel framework that learns and leverages general prompts more effectively. We introduce a dual-branch inference strategy, where a vision-enhanced branch captures fine-grained category-specific features, while a query-only branch prioritizes generalization.
  arXiv  Detail & Related papers  (2025-04-21T07:38:25Z)
• Crane: Context-Guided Prompt Learning and Attention Refinement for Zero-Shot Anomaly Detections [50.343419243749054]
  Anomaly Detection (AD) involves identifying deviations from normal data distributions. We propose a novel approach that conditions the prompts of the text encoder based on image context extracted from the vision encoder. Our method achieves state-of-the-art performance, improving performance by 2% to 29% across different metrics on 14 datasets.
  arXiv  Detail & Related papers  (2025-04-15T10:42:25Z)
• PE-CLIP: A Parameter-Efficient Fine-Tuning of Vision Language Models for Dynamic Facial Expression Recognition [7.966499123076283]
  Vision-Language Models (VLMs) like CLIP offer promising solutions for Dynamic Facial Expression Recognition (DFER) We propose PE-CLIP, a parameter-efficient fine-tuning framework that adapts CLIP for DFER while significantly reducing trainable parameters. By balancing efficiency and accuracy, PE-CLIP sets a new benchmark in resource-efficient DFER.
  arXiv  Detail & Related papers  (2025-03-21T08:45:50Z)
• AA-CLIP: Enhancing Zero-shot Anomaly Detection via Anomaly-Aware CLIP [33.213400694016]
  Anomaly detection (AD) identifies outliers for applications like defect and lesion detection. We propose Anomaly-Aware CLIP (AA-CLIP), which enhances CLIP's anomaly discrimination ability in both text and visual spaces. AA-CLIP is achieved through a straightforward yet effective two-stage approach.
  arXiv  Detail & Related papers  (2025-03-09T15:22:52Z)
• KAnoCLIP: Zero-Shot Anomaly Detection through Knowledge-Driven Prompt Learning and Enhanced Cross-Modal Integration [9.688664292809785]
  Zero-shot anomaly detection (ZSAD) identifies anomalies without needing training samples from the target dataset. vision-language models like CLIP show potential in ZSAD but have limitations. We introduce KAnoCLIP, a novel ZSAD framework that leverages vision-language models. KAnoCLIP achieves state-of-the-art performance in ZSAD across 12 industrial and medical datasets.
  arXiv  Detail & Related papers  (2025-01-07T13:51:41Z)
• GlocalCLIP: Object-agnostic Global-Local Prompt Learning for Zero-shot Anomaly Detection [5.530212768657544]
  We introduce glocal contrastive learning to improve the learning of global and local prompts, effectively detecting abnormal patterns across various domains. The generalization performance of GlocalCLIP in ZSAD was demonstrated on 15 real-world datasets from both the industrial and medical domains.
  arXiv  Detail & Related papers  (2024-11-09T05:22:13Z)
• C2P-CLIP: Injecting Category Common Prompt in CLIP to Enhance Generalization in Deepfake Detection [98.34703790782254]
  We introduce Category Common Prompt CLIP, which integrates the category common prompt into the text encoder to inject category-related concepts into the image encoder. Our method achieves a 12.41% improvement in detection accuracy compared to the original CLIP, without introducing additional parameters during testing.
  arXiv  Detail & Related papers  (2024-08-19T02:14:25Z)
• WATT: Weight Average Test-Time Adaptation of CLIP [17.74824534094739]
  We present Weight Average Test-Time Adaptation (WATT) of CLIP, a pioneering approach facilitating full test-time adaptation. Our method employs a diverse set of templates for text prompts, augmenting the existing framework of CLIP. Our findings underscore the efficacy of WATT in enhancing performance across diverse datasets.
  arXiv  Detail & Related papers  (2024-06-19T22:37:42Z)
• CLIPArTT: Light-weight Adaptation of CLIP to New Domains at Test Time [19.0284321951354]
  We introduce CLIP Adaptation duRing Test-Time (CLIPArTT), a fully test-time adaptation (TTA) approach for pre-trained vision-language models (VLMs) Our method employs a unique, minimally invasive text prompt tuning process, wherein multiple predicted classes are aggregated into a single new text prompt, used as pseudo label to re-classify inputs. Our findings demonstrate that, without requiring additional transformations nor new trainable modules, CLIPArTT enhances performance dynamically across non-corrupted datasets.
  arXiv  Detail & Related papers  (2024-05-01T07:24:30Z)
• Spectral Prompt Tuning:Unveiling Unseen Classes for Zero-Shot Semantic Segmentation [20.880942041889444]
  We propose SPT-SEG, a one-stage approach that improves CLIP's adaptability from image to pixel. Specifically, we introduce Spectral Prompt Tuning (SPT), incorporating spectral prompts into the CLIP visual encoder's shallow layers. We demonstrate the superiority of our method over state-of-the-art approaches, performing well across all classes and particularly excelling in handling unseen classes.
  arXiv  Detail & Related papers  (2023-12-20T04:27:13Z)
• Meta-Adapter: An Online Few-shot Learner for Vision-Language Model [64.21017759533474]
  Contrastive vision-language pre-training, known as CLIP, demonstrates remarkable potential in perceiving open-world visual concepts. Few-shot learning methods based on CLIP typically require offline fine-tuning of the parameters on few-shot samples. We propose the Meta-Adapter, a lightweight residual-style adapter, to refine the CLIP features guided by the few-shot samples in an online manner.
  arXiv  Detail & Related papers  (2023-11-07T07:27:16Z)
• GridCLIP: One-Stage Object Detection by Grid-Level CLIP Representation Learning [55.77244064907146]
  One-stage detector GridCLIP learns grid-level representations to adapt to the intrinsic principle of one-stage detection learning. Experiments show that the learned CLIP-based grid-level representations boost the performance of undersampled (infrequent and novel) categories.
  arXiv  Detail & Related papers  (2023-03-16T12:06:02Z)
• Generalized Few-Shot Continual Learning with Contrastive Mixture of Adapters [59.82088750033897]
  We set up a Generalized FSCL (GFSCL) protocol involving both class- and domain-incremental situations. We find that common continual learning methods have poor generalization ability on unseen domains. In this way, we propose a rehearsal-free framework based on Vision Transformer (ViT) named Contrastive Mixture of Adapters (CMoA)
  arXiv  Detail & Related papers  (2023-02-12T15:18:14Z)
• Non-Contrastive Learning Meets Language-Image Pre-Training [145.6671909437841]
  We study the validity of non-contrastive language-image pre-training (nCLIP) We introduce xCLIP, a multi-tasking framework combining CLIP and nCLIP, and show that nCLIP aids CLIP in enhancing feature semantics.
  arXiv  Detail & Related papers  (2022-10-17T17:57:46Z)
• Adversarial Feature Augmentation and Normalization for Visual Recognition [109.6834687220478]
  Recent advances in computer vision take advantage of adversarial data augmentation to ameliorate the generalization ability of classification models. Here, we present an effective and efficient alternative that advocates adversarial augmentation on intermediate feature embeddings. We validate the proposed approach across diverse visual recognition tasks with representative backbone networks.
  arXiv  Detail & Related papers  (2021-03-22T20:36:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.