Virgo: A Preliminary Exploration on Reproducing o1-like MLLM

• URL: http://arxiv.org/abs/2501.01904v2
• Date: Wed, 05 Feb 2025 09:17:01 GMT
• Title: Virgo: A Preliminary Exploration on Reproducing o1-like MLLM
• Authors: Yifan Du, Zikang Liu, Yifan Li, Wayne Xin Zhao, Yuqi Huo, Bingning Wang, Weipeng Chen, Zheng Liu, Zhongyuan Wang, Ji-Rong Wen,
• Abstract summary: Slow-thinking reasoning systems have garnered widespread attention by scaling the thinking time during inference. There is also growing interest in adapting this capability to multimodal large language models (MLLMs) In this paper, we explore a straightforward approach by fine-tuning a capable MLLM with a small amount of textual long-form thought data. We find that these long-form reasoning processes, expressed in natural language, can be effectively transferred to MLLMs.
• Score: 89.50691075011429
• License:
• Abstract: Recently, slow-thinking reasoning systems, built upon large language models (LLMs), have garnered widespread attention by scaling the thinking time during inference. There is also growing interest in adapting this capability to multimodal large language models (MLLMs). Given that MLLMs handle more complex data semantics across different modalities, it is intuitively more challenging to implement multimodal slow-thinking systems. To address this issue, in this paper, we explore a straightforward approach by fine-tuning a capable MLLM with a small amount of textual long-form thought data, resulting in a multimodal slow-thinking system, Virgo (Visual reasoning with long thought). We find that these long-form reasoning processes, expressed in natural language, can be effectively transferred to MLLMs. Moreover, it seems that such textual reasoning data can be even more effective than visual reasoning data in eliciting the slow-thinking capacities of MLLMs. While this work is preliminary, it demonstrates that slow-thinking capacities are fundamentally associated with the language model component, which can be transferred across modalities or domains. This finding can be leveraged to guide the development of more powerful slow-thinking reasoning systems. We release our resources at https://github.com/RUCAIBox/Virgo.

Related papers

• DriVLM: Domain Adaptation of Vision-Language Models in Autonomous Driving [20.644133177870852]
  multimodal large language models (MLLM) can combine multiple modalities such as pictures, videos, sounds, texts, etc. Most MLLMs require very high computational resources, which is a major challenge for most researchers and developers. In this paper, we explored the utility of small-scale MLLMs and applied small-scale MLLMs to the field of autonomous driving.
  arXiv  Detail & Related papers  (2025-01-09T09:02:41Z)
• Can Multimodal Large Language Model Think Analogically? [9.517193263050228]
  Multimodal Large Language Model (MLLM) has recently sparked considerable discussion due to its emergent capabilities. We explore two facets: textitMLLM as an explainer and textitMLLM as a predictor We propose a unified prompt template and a method for harnessing the comprehension capabilities of MLLM to augment existing models.
  arXiv  Detail & Related papers  (2024-11-02T16:59:49Z)
• A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Tasks [74.52259252807191]
  Multimodal Large Language Models (MLLMs) address the complexities of real-world applications far beyond the capabilities of single-modality systems. This paper systematically sorts out the applications of MLLM in multimodal tasks such as natural language, vision, and audio.
  arXiv  Detail & Related papers  (2024-08-02T15:14:53Z)
• The Synergy between Data and Multi-Modal Large Language Models: A Survey from Co-Development Perspective [53.48484062444108]
  We find that the development of models and data is not two separate paths but rather interconnected. On the one hand, vaster and higher-quality data contribute to better performance of MLLMs; on the other hand, MLLMs can facilitate the development of data. To promote the data-model co-development for MLLM community, we systematically review existing works related to MLLMs from the data-model co-development perspective.
  arXiv  Detail & Related papers  (2024-07-11T15:08:11Z)
• Efficient Multimodal Large Language Models: A Survey [60.7614299984182]
  Multimodal Large Language Models (MLLMs) have demonstrated remarkable performance in tasks such as visual question answering, visual understanding and reasoning. The extensive model size and high training and inference costs have hindered the widespread application of MLLMs in academia and industry. This survey provides a comprehensive and systematic review of the current state of efficient MLLMs.
  arXiv  Detail & Related papers  (2024-05-17T12:37:10Z)
• The Curious Case of Nonverbal Abstract Reasoning with Multi-Modal Large Language Models [19.213774611556]
  Multi-modal large language models (MLLMs) integrate verbal and visual information. Despite the revolutionizing prospect of MLLMs, our understanding of their reasoning abilities is limited. In this study, we assess the nonverbal abstract reasoning abilities of open-source and closed-source MLLMs.
  arXiv  Detail & Related papers  (2024-01-22T16:57:05Z)
• A Survey on Multimodal Large Language Models [71.63375558033364]
  Multimodal Large Language Model (MLLM) represented by GPT-4V has been a new rising research hotspot. This paper aims to trace and summarize the recent progress of MLLMs.
  arXiv  Detail & Related papers  (2023-06-23T15:21:52Z)
• MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models [73.86954509967416]
  Multimodal Large Language Model (MLLM) relies on the powerful LLM to perform multimodal tasks. This paper presents the first comprehensive MLLM Evaluation benchmark MME. It measures both perception and cognition abilities on a total of 14 subtasks.
  arXiv  Detail & Related papers  (2023-06-23T09:22:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.