Unlock the Power: Competitive Distillation for Multi-Modal Large
Language Models
- URL: http://arxiv.org/abs/2311.08213v1
- Date: Tue, 14 Nov 2023 14:49:46 GMT
- Title: Unlock the Power: Competitive Distillation for Multi-Modal Large
Language Models
- Authors: Xinwei Li, Li Lin, Shuai Wang, Chen Qian
- Abstract summary: Competitive Multi-modal Distillation framework (CoMD) captures bidirectional feedback between teacher and student models.
Our experimental analysis of diverse datasets shows that our knowledge transfer method consistently improves the capabilities of the student model.
- Score: 17.25135606956287
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Recently, multi-modal content generation has attracted lots of attention from
researchers by investigating the utilization of visual instruction tuning based
on large language models (LLMs). To enhance the performance and generalization
ability of such LLMs, the practice of distilling knowledge from pretrained
multi-modal models (a.k.a. teachers) to more compact multi-modal LLMs
(students) has gained considerable interest. However, the prevailing paradigm
of instructiontuning in multi-modal LLMs knowledge distillation is
resource-intensive and unidirectional, neglecting the potential for mutual
feedback between the student and teacher models. Thus, we propose an innovative
Competitive Multi-modal Distillation framework (CoMD), which captures
bidirectional feedback between teacher and student models and continually
updates the multi-modal capabilities that the student model has learned. It
comprises two stages: multi-modal pre-training and multi-modal competitive
distillation. The first stage pre-trains the student model on a large number of
filtered multi-modal datasets. The second stage facilitates a bidirectional
knowledge transfer between the student and teacher models. Our experimental
analysis of diverse datasets shows that our knowledge transfer method
consistently improves the capabilities of the student model. Finally, the
7B-sized student model after four distillations surpassed the current
state-of-the-art model LLaVA-13B on the ScienceQA and LLaVA Test dataset, also
outperforms other strong baselines in the zero-shot setting.
Related papers
- Generative Multimodal Models are In-Context Learners [60.50927925426832]
We introduce Emu2, a generative multimodal model with 37 billion parameters, trained on large-scale multimodal sequences.
Emu2 exhibits strong multimodal in-context learning abilities, even emerging to solve tasks that require on-the-fly reasoning.
arXiv Detail & Related papers (2023-12-20T18:59:58Z) - Improving Discriminative Multi-Modal Learning with Large-Scale
Pre-Trained Models [51.5543321122664]
This paper investigates how to better leverage large-scale pre-trained uni-modal models to enhance discriminative multi-modal learning.
We introduce Multi-Modal Low-Rank Adaptation learning (MMLoRA)
arXiv Detail & Related papers (2023-10-08T15:01:54Z) - VideoAdviser: Video Knowledge Distillation for Multimodal Transfer
Learning [6.379202839994046]
Multimodal transfer learning aims to transform pretrained representations of diverse modalities into a common domain space for effective multimodal fusion.
We propose VideoAdviser, a video knowledge distillation method to transfer multimodal knowledge of video-enhanced prompts from a multimodal fundamental model to a specific modal fundamental model.
We evaluate our method in two challenging multimodal tasks: video-level sentiment analysis and audio-visual retrieval.
arXiv Detail & Related papers (2023-09-27T08:44:04Z) - On Uni-Modal Feature Learning in Supervised Multi-Modal Learning [21.822251958013737]
We abstract the features (i.e. learned representations) of multi-modal data into 1) uni-modal features, which can be learned from uni-modal training, and 2) paired features, which can only be learned from cross-modal interactions.
We demonstrate that, under a simple guiding strategy, we can achieve comparable results to other complex late-fusion or intermediate-fusion methods on various multi-modal datasets.
arXiv Detail & Related papers (2023-05-02T07:15:10Z) - Efficient Multimodal Fusion via Interactive Prompting [62.08292938484994]
Large-scale pre-training has brought unimodal fields such as computer vision and natural language processing to a new era.
We propose an efficient and flexible multimodal fusion method, namely PMF, tailored for fusing unimodally pre-trained transformers.
arXiv Detail & Related papers (2023-04-13T07:31:51Z) - Multimodal Knowledge Expansion [14.332957885505547]
We propose a knowledge distillation-based framework to utilize multimodal data without requiring labels.
We show that a multimodal student model consistently denoises pseudo labels and generalizes better than its teacher.
arXiv Detail & Related papers (2021-03-26T12:32:07Z) - Reinforced Multi-Teacher Selection for Knowledge Distillation [54.72886763796232]
knowledge distillation is a popular method for model compression.
Current methods assign a fixed weight to a teacher model in the whole distillation.
Most of the existing methods allocate an equal weight to every teacher model.
In this paper, we observe that, due to the complexity of training examples and the differences in student model capability, learning differentially from teacher models can lead to better performance of student models distilled.
arXiv Detail & Related papers (2020-12-11T08:56:39Z) - Relating by Contrasting: A Data-efficient Framework for Multimodal
Generative Models [86.9292779620645]
We develop a contrastive framework for generative model learning, allowing us to train the model not just by the commonality between modalities, but by the distinction between "related" and "unrelated" multimodal data.
Under our proposed framework, the generative model can accurately identify related samples from unrelated ones, making it possible to make use of the plentiful unlabeled, unpaired multimodal data.
arXiv Detail & Related papers (2020-07-02T15:08:11Z) - InterBERT: Vision-and-Language Interaction for Multi-modal Pretraining [76.32065400614162]
We propose a novel model, namely InterBERT (BERT for Interaction), which is the first model of our series of multimodal pretraining methods M6.
The model owns strong capability of modeling interaction between the information flows of different modalities.
We propose a large-scale dataset for multi-modal pretraining in Chinese, and we develop the Chinese InterBERT which is the first Chinese multi-modal pretrained model.
arXiv Detail & Related papers (2020-03-30T03:13:22Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.