AnomalyVFM -- Transforming Vision Foundation Models into Zero-Shot Anomaly Detectors

• URL: http://arxiv.org/abs/2601.20524v1
• Date: Wed, 28 Jan 2026 12:02:58 GMT
• Title: AnomalyVFM -- Transforming Vision Foundation Models into Zero-Shot Anomaly Detectors
• Authors: Matic Fučka, Vitjan Zavrtanik, Danijel Skočaj,
• Abstract summary: AnomalyVFM is a framework that turns any pretrained VFM into a strong zero-shot anomaly detector.<n>Our approach combines a robust three-stage synthetic dataset generation scheme with a parameter-efficient adaptation mechanism.<n>It achieves an average image-level AUROC of 94.1% across 9 diverse datasets, surpassing previous methods by significant 3.3 percentage points.
• Score: 6.6016630449883955
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Zero-shot anomaly detection aims to detect and localise abnormal regions in the image without access to any in-domain training images. While recent approaches leverage vision-language models (VLMs), such as CLIP, to transfer high-level concept knowledge, methods based on purely vision foundation models (VFMs), like DINOv2, have lagged behind in performance. We argue that this gap stems from two practical issues: (i) limited diversity in existing auxiliary anomaly detection datasets and (ii) overly shallow VFM adaptation strategies. To address both challenges, we propose AnomalyVFM, a general and effective framework that turns any pretrained VFM into a strong zero-shot anomaly detector. Our approach combines a robust three-stage synthetic dataset generation scheme with a parameter-efficient adaptation mechanism, utilising low-rank feature adapters and a confidence-weighted pixel loss. Together, these components enable modern VFMs to substantially outperform current state-of-the-art methods. More specifically, with RADIO as a backbone, AnomalyVFM achieves an average image-level AUROC of 94.1% across 9 diverse datasets, surpassing previous methods by significant 3.3 percentage points. Project Page: https://maticfuc.github.io/anomaly_vfm/

Related papers

• Are Vision Foundation Models Foundational for Electron Microscopy Image Segmentation? [0.6302369456012739]
  We study the problem of mitochondria electron microscopy (EM) images using two popular public datasets (Lucchi++ and OpenCLIP)<n>We observe that training on a single EM dataset yields good segmentation performance (quantified as foreground Intersection-over-Union)<n>Training on multiple EM datasets leads to severe performance degradation for all models considered.
  arXiv  Detail & Related papers  (2026-02-09T10:55:18Z)
• Beyond Boundaries: Leveraging Vision Foundation Models for Source-Free Object Detection [34.292554427633505]
  Source-Free Object Detection (SFOD) aims to adapt a source-pretrained object detector to a target domain without access to source data.<n> Vision Foundation Models (VFMs), pretrained on massive and diverse data, exhibit strong perception capabilities and broad generalization.<n>We propose a novel SFOD framework that leverages VFMs as external knowledge sources to jointly enhance feature alignment and label quality.
  arXiv  Detail & Related papers  (2025-11-10T17:06:01Z)
• NexViTAD: Few-shot Unsupervised Cross-Domain Defect Detection via Vision Foundation Models and Multi-Task Learning [1.7603474309877931]
  NexViTAD is a cross-domain anomaly detection framework based on vision foundation models.<n>It addresses domain-shift challenges in industrial anomaly detection through innovative shared subspace projection mechanisms.<n>It delivers state-of-the-art performance with an AUC of 97.5%, AP of 70.4%, and PRO of 95.2% in the target domains.
  arXiv  Detail & Related papers  (2025-07-10T09:29:26Z)
• CLIP Meets Diffusion: A Synergistic Approach to Anomaly Detection [49.11819337853632]
  Anomaly detection is a complex problem due to the ambiguity in defining anomalies, the diversity of anomaly types, and the scarcity of training data.<n>We propose CLIPfusion, a method that leverages both discriminative and generative foundation models.<n>We believe that our method underscores the effectiveness of multi-modal and multi-model fusion in tackling the multifaceted challenges of anomaly detection.
  arXiv  Detail & Related papers  (2025-06-13T13:30:15Z)
• Adapting Visual-Language Models for Generalizable Anomaly Detection in Medical Images [68.42215385041114]
  This paper introduces a novel lightweight multi-level adaptation and comparison framework to repurpose the CLIP model for medical anomaly detection. Our approach integrates multiple residual adapters into the pre-trained visual encoder, enabling a stepwise enhancement of visual features across different levels. Our experiments on medical anomaly detection benchmarks demonstrate that our method significantly surpasses current state-of-the-art models.
  arXiv  Detail & Related papers  (2024-03-19T09:28:19Z)
• Self-supervised Feature Adaptation for 3D Industrial Anomaly Detection [59.41026558455904]
  We focus on multi-modal anomaly detection. Specifically, we investigate early multi-modal approaches that attempted to utilize models pre-trained on large-scale visual datasets. We propose a Local-to-global Self-supervised Feature Adaptation (LSFA) method to finetune the adaptors and learn task-oriented representation toward anomaly detection.
  arXiv  Detail & Related papers  (2024-01-06T07:30:41Z)
• Diffusion-Based Particle-DETR for BEV Perception [94.88305708174796]
  Bird-Eye-View (BEV) is one of the most widely-used scene representations for visual perception in Autonomous Vehicles (AVs) Recent diffusion-based methods offer a promising approach to uncertainty modeling for visual perception but fail to effectively detect small objects in the large coverage of the BEV. Here, we address this problem by combining the diffusion paradigm with current state-of-the-art 3D object detectors in BEV.
  arXiv  Detail & Related papers  (2023-12-18T09:52:14Z)
• DiAD: A Diffusion-based Framework for Multi-class Anomaly Detection [55.48770333927732]
  We propose a Difusion-based Anomaly Detection (DiAD) framework for multi-class anomaly detection. It consists of a pixel-space autoencoder, a latent-space Semantic-Guided (SG) network with a connection to the stable diffusion's denoising network, and a feature-space pre-trained feature extractor. Experiments on MVTec-AD and VisA datasets demonstrate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2023-12-11T18:38:28Z)
• Cluster-level pseudo-labelling for source-free cross-domain facial expression recognition [94.56304526014875]
  We propose the first Source-Free Unsupervised Domain Adaptation (SFUDA) method for Facial Expression Recognition (FER) Our method exploits self-supervised pretraining to learn good feature representations from the target data. We validate the effectiveness of our method in four adaptation setups, proving that it consistently outperforms existing SFUDA methods when applied to FER.
  arXiv  Detail & Related papers  (2022-10-11T08:24:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.