mPLUG-PaperOwl: Scientific Diagram Analysis with the Multimodal Large
Language Model
- URL: http://arxiv.org/abs/2311.18248v2
- Date: Tue, 9 Jan 2024 12:07:02 GMT
- Title: mPLUG-PaperOwl: Scientific Diagram Analysis with the Multimodal Large
Language Model
- Authors: Anwen Hu, Yaya Shi, Haiyang Xu, Jiabo Ye, Qinghao Ye, Ming Yan,
Chenliang Li, Qi Qian, Ji Zhang, Fei Huang
- Abstract summary: This work focuses on strengthening the multi-modal diagram analysis ability of Multimodal LLMs.
By parsing Latex source files of high-quality papers, we carefully build a multi-modal diagram understanding dataset M-Paper.
M-Paper is the first dataset to support joint comprehension of multiple scientific diagrams, including figures and tables in the format of images or Latex codes.
- Score: 73.38800189095173
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Recently, the strong text creation ability of Large Language Models(LLMs) has
given rise to many tools for assisting paper reading or even writing. However,
the weak diagram analysis abilities of LLMs or Multimodal LLMs greatly limit
their application scenarios, especially for scientific academic paper writing.
In this work, towards a more versatile copilot for academic paper writing, we
mainly focus on strengthening the multi-modal diagram analysis ability of
Multimodal LLMs. By parsing Latex source files of high-quality papers, we
carefully build a multi-modal diagram understanding dataset M-Paper. By
aligning diagrams in the paper with related paragraphs, we construct
professional diagram analysis samples for training and evaluation. M-Paper is
the first dataset to support joint comprehension of multiple scientific
diagrams, including figures and tables in the format of images or Latex codes.
Besides, to better align the copilot with the user's intention, we introduce
the `outline' as the control signal, which could be directly given by the user
or revised based on auto-generated ones. Comprehensive experiments with a
state-of-the-art Mumtimodal LLM demonstrate that training on our dataset shows
stronger scientific diagram understanding performance, including diagram
captioning, diagram analysis, and outline recommendation. The dataset, code,
and model are available at
https://github.com/X-PLUG/mPLUG-DocOwl/tree/main/PaperOwl.
Related papers
- TextHawk: Exploring Efficient Fine-Grained Perception of Multimodal Large Language Models [9.232693392690702]
TextHawk is a document-oriented Multimodal Large Language Model (MLLM)
It is designed to explore efficient fine-grained perception by designing four dedicated components.
We conduct extensive experiments on both general and document-oriented MLLM benchmarks, and show that TextHawk outperforms the state-of-the-art methods.
arXiv Detail & Related papers (2024-04-14T09:48:37Z) - ChartX & ChartVLM: A Versatile Benchmark and Foundation Model for
Complicated Chart Reasoning [56.4579228575522]
We benchmark the ability of off-the-shelf Multi-modal Large Language Models (MLLMs) in the chart domain.
We construct ChartX, a multi-modal evaluation set covering 18 chart types, 7 chart tasks, 22 disciplinary topics, and high-quality chart data.
We develop ChartVLM to offer a new perspective on handling multi-modal tasks that strongly depend on interpretable patterns.
arXiv Detail & Related papers (2024-02-19T14:48:23Z) - Large Language Models on Graphs: A Comprehensive Survey [81.7684686396014]
We provide a systematic review of scenarios and techniques related to large language models on graphs.
We first summarize potential scenarios of adopting LLMs on graphs into three categories, namely pure graphs, text-attributed graphs, and text-paired graphs.
We discuss the real-world applications of such methods and summarize open-source codes and benchmark datasets.
arXiv Detail & Related papers (2023-12-05T14:14:27Z) - Integrating Graphs with Large Language Models: Methods and Prospects [68.37584693537555]
Large language models (LLMs) have emerged as frontrunners, showcasing unparalleled prowess in diverse applications.
Merging the capabilities of LLMs with graph-structured data has been a topic of keen interest.
This paper bifurcates such integrations into two predominant categories.
arXiv Detail & Related papers (2023-10-09T07:59:34Z) - Beyond Text: A Deep Dive into Large Language Models' Ability on
Understanding Graph Data [13.524529952170672]
Large language models (LLMs) have achieved impressive performance on many natural language processing tasks.
We aim to assess whether LLMs can effectively process graph data and leverage topological structures to enhance performance.
By comparing LLMs' performance with specialized graph models, we offer insights into the strengths and limitations of employing LLMs for graph analytics.
arXiv Detail & Related papers (2023-10-07T23:25:22Z) - Sight Beyond Text: Multi-Modal Training Enhances LLMs in Truthfulness
and Ethics [32.123919380959485]
Multi-modal large language models (MLLMs) are trained based on large language models (LLM)
While they excel in multi-modal tasks, the pure NLP abilities of MLLMs are often underestimated and left untested.
We show that visual instruction tuning, a prevailing strategy for transitioning LLMs into MLLMs, unexpectedly and interestingly helps models attain both improved truthfulness and ethical alignment.
arXiv Detail & Related papers (2023-09-13T17:57:21Z) - mPLUG-DocOwl: Modularized Multimodal Large Language Model for Document
Understanding [55.4806974284156]
Document understanding refers to automatically extract, analyze and comprehend information from digital documents, such as a web page.
Existing Multi-model Large Language Models (MLLMs) have demonstrated promising zero-shot capabilities in shallow OCR-free text recognition.
arXiv Detail & Related papers (2023-07-04T11:28:07Z) - mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality [95.76661165594884]
mPLUG-Owl is a training paradigm that equips large language models (LLMs) with multi-modal abilities.
The training paradigm involves a two-stage method for aligning image and text, which learns visual knowledge with the assistance of LLM.
Experimental results show that our model outperforms existing multi-modal models.
arXiv Detail & Related papers (2023-04-27T13:27:01Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.