Who Needs Labels? Adapting Vision Foundation Models With the Metadata You Already Have

• URL: http://arxiv.org/abs/2606.05107v1
• Date: Wed, 03 Jun 2026 17:10:11 GMT
• Title: Who Needs Labels? Adapting Vision Foundation Models With the Metadata You Already Have
• Abstract summary: We propose a label-free approach to adapt powerful but generic vision foundation models to specialized scientific domains.<n>Our method, FINO, combines a standard self-supervised objective with flexible metadata guidance.<n>It consistently outperforms standard unsupervised domain adaptation and fully supervised adaptation.
• Score: 28.360344714913193
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We propose a label-free approach to adapt powerful but generic vision foundation models to specialized scientific domains. Standard supervised fine-tuning is often ill-suited to these settings: labels are scarce, and task-specific training can collapse the model's generality and hurt robustness. We instead leverage metadata to adapt representations to new domains in a self-supervised manner. Our method, FINO, combines a standard self-supervised objective with flexible metadata guidance that handles both highly granular discrete metadata and continuous metadata. It encourages the representation to preserve informative factors while suppressing spurious ones. Across subcellular fluorescence microscopy, Earth observation, wildlife monitoring, and medical imaging, FINO consistently outperforms standard unsupervised domain adaptation and fully supervised adaptation. It also exceeds highly-specialized domain-specific state of the art, while using no task labels for backbone adaptation and only lightweight probes for supervision.

Related papers

• Dual-Teacher Distillation with Subnetwork Rectification for Black-Box Domain Adaptation [69.7036779439312]
  Black box domain adaptation represents a highly practical yet extremely challenging setting.<n>We propose a dual teacher distillation with subnetwork rectification (DDSR) model that exploits specific knowledge embedded in black box source models.<n>Our approach demonstrates consistent improvements over state of the art methods, including those using source data or models.
  arXiv  Detail & Related papers  (2026-03-24T07:54:19Z)
• Automated Standardization of Legacy Biomedical Metadata Using an Ontology-Constrained LLM Agent [0.1105171275481129]
  We present an automated standardization system for biomedical metadata.<n>The system queries authoritative terminology services in real time to retrieve canonically correct vocabulary terms on demand.<n>Our evaluation shows that augmenting the LLM with real-time tool access consistently improves prediction accuracy.
  arXiv  Detail & Related papers  (2026-03-10T18:47:30Z)
• Correcting False Alarms from Unseen: Adapting Graph Anomaly Detectors at Test Time [60.341117019125214]
  We propose a lightweight and plug-and-play Test-time adaptation framework for correcting Unseen Normal pattErns in graph anomaly detection (GAD)<n>To address semantic confusion, a graph aligner is employed to align the shifted data to the original one at the graph attribute level.<n>Extensive experiments on 10 real-world datasets demonstrate that TUNE significantly enhances the generalizability of pre-trained GAD models to both synthetic and real unseen normal patterns.
  arXiv  Detail & Related papers  (2025-11-10T12:10:05Z)
• Adapting Vision-Language Models Without Labels: A Comprehensive Survey [74.17944178027015]
  Vision-Language Models (VLMs) have demonstrated remarkable generalization capabilities across a wide range of tasks.<n>Recent research has increasingly focused on unsupervised adaptation methods that do not rely on labeled data.<n>We propose a taxonomy based on the availability and nature of unlabeled visual data, categorizing existing approaches into four key paradigms.
  arXiv  Detail & Related papers  (2025-08-07T16:27:37Z)
• ConformalSAM: Unlocking the Potential of Foundational Segmentation Models in Semi-Supervised Semantic Segmentation with Conformal Prediction [57.930531826380836]
  This work explores whether a foundational segmentation model can address label scarcity in the pixel-level vision task as an annotator for unlabeled images.<n>We propose ConformalSAM, a novel SSSS framework which first calibrates the foundation model using the target domain's labeled data and then filters out unreliable pixel labels of unlabeled data.
  arXiv  Detail & Related papers  (2025-07-21T17:02:57Z)
• Crane: Context-Guided Prompt Learning and Attention Refinement for Zero-Shot Anomaly Detection [50.343419243749054]
  Anomaly detection is critical in fields such as medical diagnostics and industrial defect detection.<n> CLIP's coarse-grained image-text alignment limits localization and detection performance for fine-grained anomalies.<n>Crane improves the state-of-the-art ZSAD from 2% to 28%, at both image and pixel levels, while remaining competitive in inference speed.
  arXiv  Detail & Related papers  (2025-04-15T10:42:25Z)
• Not All Data are Good Labels: On the Self-supervised Labeling for Time Series Forecasting [37.189362258417624]
  This paper explores a novel self-supervised approach to re-label time series datasets by inherently constructing candidate datasets.<n>During the optimization of a simple reconstruction network, intermediates are used as pseudo labels in a self-supervised paradigm.<n>Experiments on eleven real-world datasets demonstrate that SCAM consistently improves the performance of various backbone models.
  arXiv  Detail & Related papers  (2025-02-20T16:29:37Z)
• GeneralAD: Anomaly Detection Across Domains by Attending to Distorted Features [68.14842693208465]
  GeneralAD is an anomaly detection framework designed to operate in semantic, near-distribution, and industrial settings. We propose a novel self-supervised anomaly generation module that employs straightforward operations like noise addition and shuffling to patch features. We extensively evaluated our approach on ten datasets, achieving state-of-the-art results in six and on-par performance in the remaining.
  arXiv  Detail & Related papers  (2024-07-17T09:27:41Z)
• UDAMA: Unsupervised Domain Adaptation through Multi-discriminator Adversarial Training with Noisy Labels Improves Cardio-fitness Prediction [16.26599832125242]
  We introduce UDAMA, a method with two key components: Unsupervised Domain Adaptation and Multidiscriminator Adversarial Training. In particular, we showcase the practical potential of UDAMA by applying it to Cardio-respiratory fitness (CRF) prediction. Our results show promising performance by alleviating distribution shifts in various label shift settings.
  arXiv  Detail & Related papers  (2023-07-31T13:31:53Z)
• The Norm Must Go On: Dynamic Unsupervised Domain Adaptation by Normalization [10.274423413222763]
  Domain adaptation is crucial to adapt a learned model to new scenarios, such as domain shifts or changing data distributions. Current approaches usually require a large amount of labeled or unlabeled data from the shifted domain. We propose Dynamic Unsupervised Adaptation (DUA) to overcome this problem.
  arXiv  Detail & Related papers  (2021-12-01T12:43:41Z)
• Semi-supervised Anatomical Landmark Detection via Shape-regulated Self-training [37.691539309804426]
  Well-annotated medical images are costly and sometimes even impossible to acquire, hindering landmark detection accuracy to some extent. We propose a model-agnostic shape-regulated self-training framework for semi-supervised landmark detection. Our framework is flexible and can be used as a plug-and-play module integrated into most supervised methods to improve performance further.
  arXiv  Detail & Related papers  (2021-05-28T05:23:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.