Enabling Small Models for Zero-Shot Classification through Model Label Learning

• URL: http://arxiv.org/abs/2408.11449v1
• Date: Wed, 21 Aug 2024 09:08:26 GMT
• Title: Enabling Small Models for Zero-Shot Classification through Model Label Learning
• Authors: Jia Zhang, Zhi Zhou, Lan-Zhe Guo, Yu-Feng Li,
• Abstract summary: We introduce a novel paradigm, Model Label Learning (MLL), which bridges the gap between models and their functionalities. Experiments on seven real-world datasets validate the effectiveness and efficiency of MLL.
• Score: 50.68074833512999
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-language models (VLMs) like CLIP have demonstrated impressive zero-shot ability in image classification tasks by aligning text and images but suffer inferior performance compared with task-specific expert models. On the contrary, expert models excel in their specialized domains but lack zero-shot ability for new tasks. How to obtain both the high performance of expert models and zero-shot ability is an important research direction. In this paper, we attempt to demonstrate that by constructing a model hub and aligning models with their functionalities using model labels, new tasks can be solved in a zero-shot manner by effectively selecting and reusing models in the hub. We introduce a novel paradigm, Model Label Learning (MLL), which bridges the gap between models and their functionalities through a Semantic Directed Acyclic Graph (SDAG) and leverages an algorithm, Classification Head Combination Optimization (CHCO), to select capable models for new tasks. Compared with the foundation model paradigm, it is less costly and more scalable, i.e., the zero-shot ability grows with the sizes of the model hub. Experiments on seven real-world datasets validate the effectiveness and efficiency of MLL, demonstrating that expert models can be effectively reused for zero-shot tasks. Our code will be released publicly.

Related papers

• NegMerge: Consensual Weight Negation for Strong Machine Unlearning [21.081262106431506]
  Machine unlearning aims to selectively remove specific knowledge from a model. Current methods rely on fine-tuning models on the forget set, generating a task vector, and subtracting it from the original model. We propose a novel method that leverages all given fine-tuned models rather than selecting a single one.
  arXiv  Detail & Related papers  (2024-10-08T00:50:54Z)
• Investigating Self-Supervised Methods for Label-Efficient Learning [27.029542823306866]
  We study different self supervised pretext tasks, namely contrastive learning, clustering, and masked image modelling for their low-shot capabilities. We introduce a framework involving both mask image modelling and clustering as pretext tasks, which performs better across all low-shot downstream tasks. When testing the model on full scale datasets, we show performance gains in multi-class classification, multi-label classification and semantic segmentation.
  arXiv  Detail & Related papers  (2024-06-25T10:56:03Z)
• EMR-Merging: Tuning-Free High-Performance Model Merging [55.03509900949149]
  We show that Elect, Mask & Rescale-Merging (EMR-Merging) shows outstanding performance compared to existing merging methods. EMR-Merging is tuning-free, thus requiring no data availability or any additional training while showing impressive performance.
  arXiv  Detail & Related papers  (2024-05-23T05:25:45Z)
• Data-efficient Large Vision Models through Sequential Autoregression [58.26179273091461]
  We develop an efficient, autoregression-based vision model on a limited dataset. We demonstrate how this model achieves proficiency in a spectrum of visual tasks spanning both high-level and low-level semantic understanding. Our empirical evaluations underscore the model's agility in adapting to various tasks, heralding a significant reduction in the parameter footprint.
  arXiv  Detail & Related papers  (2024-02-07T13:41:53Z)
• Building a Winning Team: Selecting Source Model Ensembles using a Submodular Transferability Estimation Approach [20.86345962679122]
  Estimating the transferability of publicly available pretrained models to a target task has assumed an important place for transfer learning tasks. We propose a novel Optimal tranSport-based suBmOdular tRaNsferability metric (OSBORN) to estimate the transferability of an ensemble of models to a downstream task.
  arXiv  Detail & Related papers  (2023-09-05T17:57:31Z)
• Revealing the Underlying Patterns: Investigating Dataset Similarity, Performance, and Generalization [0.0]
  Supervised deep learning models require significant amount of labeled data to achieve an acceptable performance on a specific task. We establish image-image, dataset-dataset, and image-dataset distances to gain insights into the model's behavior.
  arXiv  Detail & Related papers  (2023-08-07T13:35:53Z)
• LLM2Loss: Leveraging Language Models for Explainable Model Diagnostics [5.33024001730262]
  We propose an approach that can provide semantic insights into a model's patterns of failures and biases. We show that an ensemble of such lightweight models can be used to generate insights on the performance of the black-box model.
  arXiv  Detail & Related papers  (2023-05-04T23:54:37Z)
• Dataless Knowledge Fusion by Merging Weights of Language Models [51.8162883997512]
  Fine-tuning pre-trained language models has become the prevalent paradigm for building downstream NLP models. This creates a barrier to fusing knowledge across individual models to yield a better single model. We propose a dataless knowledge fusion method that merges models in their parameter space.
  arXiv  Detail & Related papers  (2022-12-19T20:46:43Z)
• Composing Ensembles of Pre-trained Models via Iterative Consensus [95.10641301155232]
  We propose a unified framework for composing ensembles of different pre-trained models. We use pre-trained models as "generators" or "scorers" and compose them via closed-loop iterative consensus optimization. We demonstrate that consensus achieved by an ensemble of scorers outperforms the feedback of a single scorer.
  arXiv  Detail & Related papers  (2022-10-20T18:46:31Z)
• Improving Label Quality by Jointly Modeling Items and Annotators [68.8204255655161]
  We propose a fully Bayesian framework for learning ground truth labels from noisy annotators. Our framework ensures scalability by factoring a generative, Bayesian soft clustering model over label distributions into the classic David and Skene joint annotator-data model.
  arXiv  Detail & Related papers  (2021-06-20T02:15:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.