LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset,
Framework, and Benchmark
- URL: http://arxiv.org/abs/2306.06687v3
- Date: Mon, 6 Nov 2023 07:02:19 GMT
- Title: LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset,
Framework, and Benchmark
- Authors: Zhenfei Yin, Jiong Wang, Jianjian Cao, Zhelun Shi, Dingning Liu, Mukai
Li, Lu Sheng, Lei Bai, Xiaoshui Huang, Zhiyong Wang, Jing Shao, Wanli Ouyang
- Abstract summary: We present Language-Assisted Multi-Modal instruction tuning dataset, framework, and benchmark.
Our aim is to establish LAMM as a growing ecosystem for training and evaluating MLLMs.
We present a comprehensive dataset and benchmark, which cover a wide range of vision tasks for 2D and 3D vision.
- Score: 81.42376626294812
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Large language models have emerged as a promising approach towards achieving
general-purpose AI agents. The thriving open-source LLM community has greatly
accelerated the development of agents that support human-machine dialogue
interaction through natural language processing. However, human interaction
with the world extends beyond only text as a modality, and other modalities
such as vision are also crucial. Recent works on multi-modal large language
models, such as GPT-4V and Bard, have demonstrated their effectiveness in
handling visual modalities. However, the transparency of these works is limited
and insufficient to support academic research. To the best of our knowledge, we
present one of the very first open-source endeavors in the field, LAMM,
encompassing a Language-Assisted Multi-Modal instruction tuning dataset,
framework, and benchmark. Our aim is to establish LAMM as a growing ecosystem
for training and evaluating MLLMs, with a specific focus on facilitating AI
agents capable of bridging the gap between ideas and execution, thereby
enabling seamless human-AI interaction. Our main contribution is three-fold: 1)
We present a comprehensive dataset and benchmark, which cover a wide range of
vision tasks for 2D and 3D vision. Extensive experiments validate the
effectiveness of our dataset and benchmark. 2) We outline the detailed
methodology of constructing multi-modal instruction tuning datasets and
benchmarks for MLLMs, enabling rapid scaling and extension of MLLM research to
diverse domains, tasks, and modalities. 3) We provide a primary but potential
MLLM training framework optimized for modality extension. We also provide
baseline models, comprehensive experimental observations, and analysis to
accelerate future research. Our baseline model is trained within 24 A100 GPU
hours, framework supports training with V100 and RTX3090 is available thanks to
the open-source society.
Related papers
- Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs [56.391404083287235]
We introduce Cambrian-1, a family of multimodal LLMs (MLLMs) designed with a vision-centric approach.
Our study uses LLMs and visual instruction tuning as an interface to evaluate various visual representations.
We provide model weights, code, supporting tools, datasets, and detailed instruction-tuning and evaluation recipes.
arXiv Detail & Related papers (2024-06-24T17:59:42Z) - Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions [11.786387517781328]
Vision-Language Models (VLMs) are advanced models that can tackle more intricate tasks such as image captioning and visual question answering.
Our classification organizes VLMs into three distinct categories: models dedicated to vision-language understanding, models that process multimodal inputs to generate unimodal (textual) outputs and models that both accept and produce multimodal inputs and outputs.
We meticulously dissect each model, offering an extensive analysis of its foundational architecture, training data sources, as well as its strengths and limitations wherever possible.
arXiv Detail & Related papers (2024-02-20T18:57:34Z) - Generative Multi-Modal Knowledge Retrieval with Large Language Models [75.70313858231833]
We propose an innovative end-to-end generative framework for multi-modal knowledge retrieval.
Our framework takes advantage of the fact that large language models (LLMs) can effectively serve as virtual knowledge bases.
We demonstrate significant improvements ranging from 3.0% to 14.6% across all evaluation metrics when compared to strong baselines.
arXiv Detail & Related papers (2024-01-16T08:44:29Z) - Small LLMs Are Weak Tool Learners: A Multi-LLM Agent [73.54562551341454]
Large Language Model (LLM) agents significantly extend the capabilities of standalone LLMs.
We propose a novel approach that decomposes the aforementioned capabilities into a planner, caller, and summarizer.
This modular framework facilitates individual updates and the potential use of smaller LLMs for building each capability.
arXiv Detail & Related papers (2024-01-14T16:17:07Z) - ModaVerse: Efficiently Transforming Modalities with LLMs [25.49713745405194]
We introduce ModaVerse, a Multi-modal Large Language Model capable of comprehending and transforming content across various modalities.
We propose a novel Input/Output (I/O) alignment mechanism that operates directly at the level of natural language.
arXiv Detail & Related papers (2024-01-12T06:28:54Z) - LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language
Models [56.25156596019168]
This paper introduces the LMRL-Gym benchmark for evaluating multi-turn RL for large language models (LLMs)
Our benchmark consists of 8 different language tasks, which require multiple rounds of language interaction and cover a range of tasks in open-ended dialogue and text games.
arXiv Detail & Related papers (2023-11-30T03:59:31Z) - u-LLaVA: Unifying Multi-Modal Tasks via Large Language Model [18.100947750831885]
u-LLaVA is an innovative unifying multi-task framework that integrates pixel, regional, and global features to refine the perceptual faculties of MLLMs.
This work contributes a novel mask-based multi-task dataset comprising 277K samples, crafted to challenge and assess the fine-grained perception capabilities of MLLMs.
arXiv Detail & Related papers (2023-11-09T13:18:27Z) - Sight Beyond Text: Multi-Modal Training Enhances LLMs in Truthfulness
and Ethics [32.123919380959485]
Multi-modal large language models (MLLMs) are trained based on large language models (LLM)
While they excel in multi-modal tasks, the pure NLP abilities of MLLMs are often underestimated and left untested.
We show that visual instruction tuning, a prevailing strategy for transitioning LLMs into MLLMs, unexpectedly and interestingly helps models attain both improved truthfulness and ethical alignment.
arXiv Detail & Related papers (2023-09-13T17:57:21Z) - Macaw-LLM: Multi-Modal Language Modeling with Image, Audio, Video, and
Text Integration [50.94902442781148]
We propose a novel multi-modal large language model (LLM) that seamlessly integrates visual, audio, and textual information.
Macaw-LLM consists of three main components: a modality module for encoding multi-modal data, a cognitive module for harnessing pretrained LLMs, and an alignment module for harmonizing diverse representations.
We construct a large-scale multi-modal instruction dataset in terms of multi-turn dialogue, including 69K image instances and 50K video instances.
arXiv Detail & Related papers (2023-06-15T12:45:25Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.