Pre-Trained Vision-Language Model Selection and Reuse for Downstream Tasks

• URL: http://arxiv.org/abs/2501.18271v1
• Date: Thu, 30 Jan 2025 11:10:46 GMT
• Title: Pre-Trained Vision-Language Model Selection and Reuse for Downstream Tasks
• Authors: Hao-Zhe Tan, Zhi Zhou, Lan-Zhe Guo, Yu-Feng Li,
• Abstract summary: This paper proposes a novel paradigm to select and reuse VLM for downstream tasks, called Model Label Learning (MLL) The proposal is highly computationally efficient and growable since the model labeling process is completed target task independent.
• Score: 48.67303250592189
• License:
• Abstract: Pre-trained Vision-Language Models (VLMs) are becoming increasingly popular across various visual tasks, and several open-sourced VLM variants have been released. However, selecting the best-performing pre-trained VLM for a specific downstream task is challenging since no single VLM can achieve promising performance on all downstream tasks, and evaluating all available VLMs is impossible due to time and data limitations. To address this problem, this paper proposes a novel paradigm to select and reuse VLM for downstream tasks, called Model Label Learning (MLL). The proposal contains three key modules: \emph{model labeling}, which assigns labels to each VLM to describe their specialty and utility; \emph{model selection}, which matches the requirements of the target task with model labels; and \emph{model reuse}, which applies selected VLMs to the target task in an ensemble manner. The proposal is highly computationally efficient and growable since the model labeling process is completed target task independent and the ability could grow with the number of candidate VLMs. We also introduce a new benchmark for evaluating VLM selection methods, including 49 VLMs and 17 target task datasets. Experimental results clearly demonstrate the effectiveness of the proposed method for selecting and reusing VLMs.

Related papers

• Sparse Attention Vectors: Generative Multimodal Model Features Are Discriminative Vision-Language Classifiers [79.45405711339322]
  Generative Large Multimodal Models (LMMs) excel at a wide variety of vision-language (VL) tasks such as image captioning or visual question answering. We propose an approach for finding features in the model's latent space to more effectively leverage LMMs for discriminative tasks.
  arXiv  Detail & Related papers  (2024-11-28T18:55:41Z)
• Rethinking VLMs and LLMs for Image Classification [6.550471260627169]
  Large Language Models (LLMs) are increasingly being merged with Visual Language Models (VLMs) to enable new capabilities. We show that, for object and scene recognition, VLMs that do not leverage LLMs can achieve better performance than VLMs that do. We propose a pragmatic solution: a lightweight fix involving a relatively small LLM that efficiently routes visual tasks to the most suitable model for the task.
  arXiv  Detail & Related papers  (2024-10-03T23:40:21Z)
• SELF-GUIDE: Better Task-Specific Instruction Following via Self-Synthetic Finetuning [70.21358720599821]
  Large language models (LLMs) hold the promise of solving diverse tasks when provided with appropriate natural language prompts. We propose SELF-GUIDE, a multi-stage mechanism in which we synthesize task-specific input-output pairs from the student LLM. We report an absolute improvement of approximately 15% for classification tasks and 18% for generation tasks in the benchmark's metrics.
  arXiv  Detail & Related papers  (2024-07-16T04:41:58Z)
• The Solution for CVPR2024 Foundational Few-Shot Object Detection Challenge [14.330962576584446]
  This report introduces an enhanced method for the Foundational Few-Shot Object Detection (FSOD) task, leveraging the vision-language model (VLM) for object detection. We propose the VLM+ framework, which integrates the multimodal large language model (MM-LLM) We use these referential expressions to generate pseudo-labels for all images in the training set and then combine them with the original labeled data to fine-tune the VLM.
  arXiv  Detail & Related papers  (2024-06-18T03:03:02Z)
• Bridge the Modality and Capability Gaps in Vision-Language Model Selection [62.26769826687365]
  Vision Language Models (VLMs) excel in zero-shot image classification by pairing images with textual category names. To better reuse the VLM resource, a promising strategy is selecting appropriate Pre-Trained VLMs from the VLM Zoo. We analyze two inherent challenges in assessing the ability of a VLM in this Language-Only VLM selection. We propose VLM Selection With gAp Bridging to mitigate the negative impact of two gaps.
  arXiv  Detail & Related papers  (2024-03-20T17:54:58Z)
• Your Vision-Language Model Itself Is a Strong Filter: Towards High-Quality Instruction Tuning with Data Selection [59.11430077029321]
  We introduce a novel dataset selection method, Self-Filter, for vision-language models (VLMs) In the first stage, we devise a scoring network to evaluate the difficulty of training instructions, which is co-trained with the VLM. In the second stage, we use the trained score net to measure the difficulty of each instruction, select the most challenging samples, and penalize similar samples to encourage diversity.
  arXiv  Detail & Related papers  (2024-02-19T20:08:48Z)
• LOVM: Language-Only Vision Model Selection [13.857583570058392]
  We introduce a new task LOVM: Language-Only Vision Model Selection, where methods are expected to perform both model selection and performance prediction. We then introduced an extensive LOVM benchmark consisting of ground-truth evaluations of 35 pre-trained VLMs and 23 datasets.
  arXiv  Detail & Related papers  (2023-06-15T06:53:05Z)
• Test-Time Adaptation with CLIP Reward for Zero-Shot Generalization in Vision-Language Models [76.410400238974]
  We propose TTA with feedback to rectify the model output and prevent the model from becoming blindly confident. A CLIP model is adopted as the reward model during TTA and provides feedback for the VLM. The proposed textitreinforcement learning with CLIP feedback(RLCF) framework is highly flexible and universal.
  arXiv  Detail & Related papers  (2023-05-29T11:03:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.