Adapting Vision-Language Models from Iconic to Inclusive for Multi-Label Recognition Without Labels

• URL: http://arxiv.org/abs/2606.11626v1
• Date: Wed, 10 Jun 2026 03:41:50 GMT
• Title: Adapting Vision-Language Models from Iconic to Inclusive for Multi-Label Recognition Without Labels
• Abstract summary: Vision-language models (VLMs) enable zero-shot recognition without labeled data.<n>These models often prioritize the most iconic object and omit other contextual positives.<n>We propose an unsupervised framework that adapts VLMs from iconic recognition toward inclusive understanding.
• Score: 14.373553090930612
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Understanding multi-label images remains a challenging task in computer vision. With the rapid progress of vision-language multimodal learning, vision-language models (VLMs) enable zero-shot recognition without labeled data. However, due to their intrinsic design, these models often prioritize the most iconic object and omit other contextual positives. This intrinsic bias conflicts with the nature of multi-label learning, thereby limiting their applicability. In this work, we propose an unsupervised framework that adapts VLMs from iconic recognition toward inclusive understanding, enabling label-free multi-label image recognition. Our approach consists of two key stages, ``cutting'' and ``sewing'': In the cutting stage, we present the multi-sampling response estimator to prevent the model from concentrating only on one single object. In the second sewing stage, the multi-object blend adaptation is introduced to adjust the labels to better conform to the multi-label distribution while preserving the intrinsic characteristics of the original model within only one epoch. Extensive experiments show that our framework significantly outperforms existing unsupervised approaches on four public datasets, even surpassing several representative weakly supervised baselines. These results demonstrate the potential of adapting pre-trained VLMs for more comprehensive visual understanding without manual annotations. Our code is publicly available at https://github.com/iCVTEAM/TailorCLIP.

Related papers

• Unified Multimodal Autoregressive Modeling with Shared Context-Visual Tokenizer is Key to Unification [80.62512020268626]
  UniAR is a unified autoregressive framework where a single visual tokenizer serves as the key bridge between understanding and generation.<n>UniAR adapts a pretrained vision encoder with multi-level feature fusion and a lookup-free bitwise quantization scheme.<n>A diffusion-based visual decoder operates on discrete visual tokens to decode high-fidelity images.
  arXiv  Detail & Related papers  (2026-06-16T17:59:22Z)
• [CLS] is Not Enough: Multi-Label Recognition via Patch-Level Inference and Adaptive Aggregation [20.637119409165418]
  PIAA formulates prediction as Patch-level Inference followed by Adaptive Aggregation.<n>We introduce an adaptive aggregation module that consolidates patch-level scores into the final multi-label prediction.<n>Experiments show that our method achieves strong improvements with minimal extra computation.
  arXiv  Detail & Related papers  (2026-05-25T13:19:12Z)
• Conformal Cross-Modal Active Learning [16.42479423972555]
  Active Learning aims to minimize annotation costs by strategically selecting the most informative samples for labeling.<n>We introduce Conformal Cross-Modal Acquisition ( CCMA), a novel AL framework that bridges vision and language modalities through a teacher-student architecture.
  arXiv  Detail & Related papers  (2026-03-24T12:59:47Z)
• Unleashing the Power of Vision-Language Models for Long-Tailed Multi-Label Visual Recognition [55.189113121465816]
  We propose a novel correlation adaptation prompt network (CAPNET) for long-tailed multi-label visual recognition.<n>CAPNET explicitly models correlations from CLIP's textual encoder.<n>It improves generalization through test-time ensembling and realigns visual-textual modalities.
  arXiv  Detail & Related papers  (2025-11-25T18:57:28Z)
• Vision Large Language Models Are Good Noise Handlers in Engagement Analysis [54.397912827957164]
  We propose a framework leveraging Vision Large Language Models (VLMs) to refine annotations and guide the training process.<n>Our framework uses a questionnaire to extract behavioral cues and split data into high- and low-reliability subsets.<n>We demonstrate that classical computer vision models trained on refined high-reliability subsets and enhanced with our curriculum strategy show improvements.
  arXiv  Detail & Related papers  (2025-11-18T18:50:26Z)
• No Labels Needed: Zero-Shot Image Classification with Collaborative Self-Learning [0.0]
  Vision-language models (VLMs) and transfer learning with pre-trained visual models appear as promising techniques to deal with this problem.<n>This paper proposes a novel zero-shot image classification framework that combines a VLM and a pre-trained visual model within a self-learning cycle.
  arXiv  Detail & Related papers  (2025-09-23T12:54:52Z)
• Towards Generalized Multi-stage Clustering: Multi-view Self-distillation [10.368796552760571]
  Existing multi-stage clustering methods independently learn the salient features from multiple views and then perform the clustering task. This paper proposes a novel multi-stage deep MVC framework where multi-view self-distillation (DistilMVC) is introduced to distill dark knowledge of label distribution.
  arXiv  Detail & Related papers  (2023-10-29T03:35:34Z)
• Noise-Tolerant Few-Shot Unsupervised Adapter for Vision-Language Models [8.59772105902647]
  We design NtUA, a Noise-tolerant Unsupervised Adapter that allows the learning of effective target models with few unlabelled target samples. NtUA works as a key-value cache that formulates visual features and predicted pseudo-labels of the few unlabelled target samples as key-value pairs. NtUA achieves superior performance consistently across multiple widely adopted benchmarks.
  arXiv  Detail & Related papers  (2023-09-26T13:35:31Z)
• DualCoOp++: Fast and Effective Adaptation to Multi-Label Recognition with Limited Annotations [79.433122872973]
  Multi-label image recognition in the low-label regime is a task of great challenge and practical significance. We leverage the powerful alignment between textual and visual features pretrained with millions of auxiliary image-text pairs. We introduce an efficient and effective framework called Evidence-guided Dual Context Optimization (DualCoOp++)
  arXiv  Detail & Related papers  (2023-08-03T17:33:20Z)
• Semantic Contrastive Bootstrapping for Single-positive Multi-label Recognition [36.3636416735057]
  We present a semantic contrastive bootstrapping (Scob) approach to gradually recover the cross-object relationships. We then propose a recurrent semantic masked transformer to extract iconic object-level representations. Extensive experimental results demonstrate that the proposed joint learning framework surpasses the state-of-the-art models.
  arXiv  Detail & Related papers  (2023-07-15T01:59:53Z)
• A Simple Long-Tailed Recognition Baseline via Vision-Language Model [92.2866546058082]
  The visual world naturally exhibits a long-tailed distribution of open classes, which poses great challenges to modern visual systems. Recent advances in contrastive visual-language pretraining shed light on a new pathway for visual recognition. We propose BALLAD to leverage contrastive vision-language models for long-tailed recognition.
  arXiv  Detail & Related papers  (2021-11-29T17:49:24Z)
• Dense Contrastive Visual-Linguistic Pretraining [53.61233531733243]
  Several multimodal representation learning approaches have been proposed that jointly represent image and text. These approaches achieve superior performance by capturing high-level semantic information from large-scale multimodal pretraining. We propose unbiased Dense Contrastive Visual-Linguistic Pretraining to replace the region regression and classification with cross-modality region contrastive learning.
  arXiv  Detail & Related papers  (2021-09-24T07:20:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.