Towards Robust Instruction Tuning on Multimodal Large Language Models

• URL: http://arxiv.org/abs/2402.14492v2
• Date: Fri, 14 Jun 2024 13:38:14 GMT
• Title: Towards Robust Instruction Tuning on Multimodal Large Language Models
• Authors: Wei Han, Hui Chen, Soujanya Poria,
• Abstract summary: In this work, we introduce an automatic instruction augmentation method named INSTRAUG in multimodal tasks. Results on two popular multimodal instructionfollowing benchmarks show that INSTRAUG can significantly improve the alignment of multimodal large language models (MLLMs) across 12 multimodal tasks.
• Score: 25.506776502317436
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Fine-tuning large language models (LLMs) on multi-task instruction-following data has been proven to be a powerful learning paradigm for improving their zero-shot capabilities on new tasks. Recent works about high-quality instruction-following data generation and selection require amounts of human labor to conceive model-understandable instructions for the given tasks and carefully filter the LLM-generated data. In this work, we introduce an automatic instruction augmentation method named INSTRAUG in multimodal tasks. It starts from a handful of basic and straightforward meta instructions but can expand an instruction-following dataset by 30 times. Results on two popular multimodal instructionfollowing benchmarks MULTIINSTRUCT and InstructBLIP show that INSTRAUG can significantly improve the alignment of multimodal large language models (MLLMs) across 12 multimodal tasks, which is even equivalent to the benefits of scaling up training data multiple times.

Related papers

• MLAN: Language-Based Instruction Tuning Improves Zero-Shot Generalization of Multimodal Large Language Models [79.0546136194314]
  We present a novel instruction tuning recipe to improve the zero-shot task generalization of multimodal large language models. We evaluate the performance of the proposed approach on 9 unseen datasets across both language and vision modalities.
  arXiv  Detail & Related papers  (2024-11-15T20:09:59Z)
• Layer by Layer: Uncovering Where Multi-Task Learning Happens in Instruction-Tuned Large Language Models [22.676688441884465]
  Fine-tuning pre-trained large language models (LLMs) on a diverse array of tasks has become a common approach for building models. This study investigates the task-specific information encoded in pre-trained LLMs and the effects of instruction tuning on their representations.
  arXiv  Detail & Related papers  (2024-10-25T23:38:28Z)
• Align$^2$LLaVA: Cascaded Human and Large Language Model Preference Alignment for Multi-modal Instruction Curation [56.75665429851673]
  This paper introduces a novel instruction curation algorithm, derived from two unique perspectives, human and LLM preference alignment. Experiments demonstrate that we can maintain or even improve model performance by compressing synthetic multimodal instructions by up to 90%.
  arXiv  Detail & Related papers  (2024-09-27T08:20:59Z)
• MMInstruct: A High-Quality Multi-Modal Instruction Tuning Dataset with Extensive Diversity [80.02202386597138]
  We construct a high-quality, diverse visual instruction tuning dataset MMInstruct, which consists of 973K instructions from 24 domains. Our instruction generation engine enables semi-automatic, low-cost, and multi-domain instruction generation at the cost of manual construction.
  arXiv  Detail & Related papers  (2024-07-22T17:55:22Z)
• SELF-GUIDE: Better Task-Specific Instruction Following via Self-Synthetic Finetuning [70.21358720599821]
  Large language models (LLMs) hold the promise of solving diverse tasks when provided with appropriate natural language prompts. We propose SELF-GUIDE, a multi-stage mechanism in which we synthesize task-specific input-output pairs from the student LLM. We report an absolute improvement of approximately 15% for classification tasks and 18% for generation tasks in the benchmark's metrics.
  arXiv  Detail & Related papers  (2024-07-16T04:41:58Z)
• Mosaic-IT: Free Compositional Data Augmentation Improves Instruction Tuning [30.82220015525281]
  Mosaic Instruction Tuning (Mosaic-IT) is a human/model-free compositional data augmentation method. Mosaic-IT randomly creates rich and diverse augmentations from existing instruction tuning data. Our evaluations demonstrate a superior performance and training efficiency of Mosaic-IT.
  arXiv  Detail & Related papers  (2024-05-22T04:08:20Z)
• INTERS: Unlocking the Power of Large Language Models in Search with Instruction Tuning [59.07490387145391]
  Large language models (LLMs) have demonstrated impressive capabilities in various natural language processing tasks. Their application to information retrieval (IR) tasks is still challenging due to the infrequent occurrence of many IR-specific concepts in natural language. We introduce a novel instruction tuning dataset, INTERS, encompassing 20 tasks across three fundamental IR categories.
  arXiv  Detail & Related papers  (2024-01-12T12:10:28Z)
• Diffusion Model is an Effective Planner and Data Synthesizer for Multi-Task Reinforcement Learning [101.66860222415512]
  Multi-Task Diffusion Model (textscMTDiff) is a diffusion-based method that incorporates Transformer backbones and prompt learning for generative planning and data synthesis. For generative planning, we find textscMTDiff outperforms state-of-the-art algorithms across 50 tasks on Meta-World and 8 maps on Maze2D.
  arXiv  Detail & Related papers  (2023-05-29T05:20:38Z)
• MultiInstruct: Improving Multi-Modal Zero-Shot Learning via Instruction Tuning [24.741736629886564]
  Instruction tuning is a new learning paradigm that fine-tunes pre-trained language models on tasks specified through instructions. We introduce MUL-TIINSTRUCT, the first multimodal instruction tuning benchmark dataset. We show strong zero-shot performance on various unseen multimodal tasks and the benefit of transfer learning from a text-only instruction dataset.
  arXiv  Detail & Related papers  (2022-12-21T05:17:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.