Gemini in Reasoning: Unveiling Commonsense in Multimodal Large Language
Models
- URL: http://arxiv.org/abs/2312.17661v1
- Date: Fri, 29 Dec 2023 15:57:49 GMT
- Title: Gemini in Reasoning: Unveiling Commonsense in Multimodal Large Language
Models
- Authors: Yuqing Wang, Yun Zhao
- Abstract summary: Google introduced Gemini, a cutting-edge MLLM designed specifically for multimodal integration.
Despite its advancements, preliminary benchmarks indicate that Gemini lags behind GPT models in commonsense reasoning tasks.
This study undertakes a thorough evaluation of Gemini's performance in complex reasoning tasks.
- Score: 14.30980373935713
- License: http://creativecommons.org/licenses/by-nc-sa/4.0/
- Abstract: The burgeoning interest in Multimodal Large Language Models (MLLMs), such as
OpenAI's GPT-4V(ision), has significantly impacted both academic and industrial
realms. These models enhance Large Language Models (LLMs) with advanced visual
understanding capabilities, facilitating their application in a variety of
multimodal tasks. Recently, Google introduced Gemini, a cutting-edge MLLM
designed specifically for multimodal integration. Despite its advancements,
preliminary benchmarks indicate that Gemini lags behind GPT models in
commonsense reasoning tasks. However, this assessment, based on a limited
dataset (i.e., HellaSWAG), does not fully capture Gemini's authentic
commonsense reasoning potential. To address this gap, our study undertakes a
thorough evaluation of Gemini's performance in complex reasoning tasks that
necessitate the integration of commonsense knowledge across modalities. We
carry out a comprehensive analysis of 12 commonsense reasoning datasets,
ranging from general to domain-specific tasks. This includes 11 datasets
focused solely on language, as well as one that incorporates multimodal
elements. Our experiments across four LLMs and two MLLMs demonstrate Gemini's
competitive commonsense reasoning capabilities. Additionally, we identify
common challenges faced by current LLMs and MLLMs in addressing commonsense
problems, underscoring the need for further advancements in enhancing the
commonsense reasoning abilities of these models.
Related papers
- Needle In A Multimodal Haystack [79.81804334634408]
We present the first benchmark specifically designed to evaluate the capability of existing MLLMs to comprehend long multimodal documents.
Our benchmark includes three types of evaluation tasks: multimodal retrieval, counting, and reasoning.
We observe that existing models still have significant room for improvement on these tasks, especially on vision-centric evaluation.
arXiv Detail & Related papers (2024-06-11T13:09:16Z) - Assessing the Emergent Symbolic Reasoning Abilities of Llama Large Language Models [47.129504708849446]
Large Language Models (LLMs) achieve impressive performance in a wide range of tasks.
LLMs show emergent abilities in mathematical reasoning benchmarks.
We evaluate three models of the Llama 2 family on different symbolic reasoning tasks.
arXiv Detail & Related papers (2024-06-05T12:22:43Z) - Mipha: A Comprehensive Overhaul of Multimodal Assistant with Small Language Models [25.724995114710165]
We investigate the design aspects of Multimodal Small Language Models (MSLMs) and propose an efficient multimodal assistant named Mipha.
Our Mipha-3B outperforms the state-of-the-art large MLLMs, especially LLaVA-1.5-13B, on multiple benchmarks.
arXiv Detail & Related papers (2024-03-10T12:43:27Z) - Multimodal Large Language Models to Support Real-World Fact-Checking [80.41047725487645]
Multimodal large language models (MLLMs) carry the potential to support humans in processing vast amounts of information.
While MLLMs are already being used as a fact-checking tool, their abilities and limitations in this regard are understudied.
We propose a framework for systematically assessing the capacity of current multimodal models to facilitate real-world fact-checking.
arXiv Detail & Related papers (2024-03-06T11:32:41Z) - A Challenger to GPT-4V? Early Explorations of Gemini in Visual Expertise [78.54563675327198]
Gemini is Google's newest and most capable MLLM built from the ground up for multi-modality.
Can Gemini challenge GPT-4V's leading position in multi-modal learning?
We compare Gemini Pro with the state-of-the-art GPT-4V to evaluate its upper limits, along with the latest open-sourced MLLM, Sphinx.
arXiv Detail & Related papers (2023-12-19T18:59:22Z) - LION : Empowering Multimodal Large Language Model with Dual-Level Visual
Knowledge [58.82222646803248]
Multimodal Large Language Models (MLLMs) have endowed LLMs with the ability to perceive and understand multi-modal signals.
Most of the existing MLLMs mainly adopt vision encoders pretrained on coarsely aligned image-text pairs, leading to insufficient extraction and reasoning of visual knowledge.
We propose a dual-Level vIsual knedgeOwl eNhanced Multimodal Large Language Model (LION), which empowers the MLLM by injecting visual knowledge in two levels.
arXiv Detail & Related papers (2023-11-20T15:56:44Z) - A Survey on Multimodal Large Language Models [71.63375558033364]
Multimodal Large Language Model (MLLM) represented by GPT-4V has been a new rising research hotspot.
This paper aims to trace and summarize the recent progress of MLLMs.
arXiv Detail & Related papers (2023-06-23T15:21:52Z) - D$^2$TV: Dual Knowledge Distillation and Target-oriented Vision Modeling
for Many-to-Many Multimodal Summarization [113.72253589338472]
Many-to-many multimodal summarization (M$3$S) task aims to generate summaries in any language with document inputs in any language and the corresponding image sequence.
We propose a dual knowledge distillation and target-oriented vision modeling framework for the M$3$S task.
arXiv Detail & Related papers (2023-05-22T06:47:35Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.