From Text to Pixel: Advancing Long-Context Understanding in MLLMs

• URL: http://arxiv.org/abs/2405.14213v2
• Date: Mon, 26 Aug 2024 04:59:05 GMT
• Title: From Text to Pixel: Advancing Long-Context Understanding in MLLMs
• Authors: Yujie Lu, Xiujun Li, Tsu-Jui Fu, Miguel Eckstein, William Yang Wang,
• Abstract summary: We introduce SEEKER, a multimodal large language model designed to tackle this issue. SEEKER aims to optimize the compact encoding of long text by compressing the text sequence into the visual pixel space via images. Our experiments on six long-context multimodal tasks demonstrate that SEEKER can leverage fewer image tokens to convey the same amount of textual information compared with the OCR-based approach.
• Score: 70.78454154014989
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The rapid progress in Multimodal Large Language Models (MLLMs) has significantly advanced their ability to process and understand complex visual and textual information. However, the integration of multiple images and extensive textual contexts remains a challenge due to the inherent limitation of the models' capacity to handle long input sequences efficiently. In this paper, we introduce SEEKER, a multimodal large language model designed to tackle this issue. SEEKER aims to optimize the compact encoding of long text by compressing the text sequence into the visual pixel space via images, enabling the model to handle long text within a fixed token-length budget efficiently. Our empirical experiments on six long-context multimodal tasks demonstrate that SEEKER can leverage fewer image tokens to convey the same amount of textual information compared with the OCR-based approach, and is more efficient in understanding long-form multimodal input and generating long-form textual output, outperforming all existing proprietary and open-source MLLMs by large margins.

Related papers

• Hierarchical Visual Feature Aggregation for OCR-Free Document Understanding [41.43688559565315]
  We present a novel OCR-free document understanding framework based on pretrained Multimodal Large Language Models (MLLMs) Our approach employs multi-scale visual features to handle various font sizes within document images. We introduce a novel instruction tuning task, which facilitates the model's text-reading capability by learning to predict the relative positions of input text.
  arXiv  Detail & Related papers  (2024-11-08T00:58:12Z)
• MaVEn: An Effective Multi-granularity Hybrid Visual Encoding Framework for Multimodal Large Language Model [49.931663904599205]
  MaVEn is an innovative framework designed to enhance the capabilities of Multimodal Large Language Models (MLLMs) in multi-image reasoning. We show that MaVEn significantly enhances MLLMs' understanding in complex multi-image scenarios, while also improving performance in single-image contexts.
  arXiv  Detail & Related papers  (2024-08-22T11:57:16Z)
• mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Models [71.40705814904898]
  We introduce the versatile multi-modal large language model, mPLUG-Owl3, which enhances the capability for long image-sequence understanding. Specifically, we propose novel hyper attention blocks to efficiently integrate vision and language into a common language-guided semantic space.
  arXiv  Detail & Related papers  (2024-08-09T03:25:42Z)
• SEED-Story: Multimodal Long Story Generation with Large Language Model [66.37077224696242]
  SEED-Story is a novel method that leverages a Multimodal Large Language Model (MLLM) to generate extended multimodal stories. We propose multimodal attention sink mechanism to enable the generation of stories with up to 25 sequences (only 10 for training) in a highly efficient autoregressive manner. We present a large-scale and high-resolution dataset named StoryStream for training our model and quantitatively evaluating the task of multimodal story generation in various aspects.
  arXiv  Detail & Related papers  (2024-07-11T17:21:03Z)
• DocKylin: A Large Multimodal Model for Visual Document Understanding with Efficient Visual Slimming [33.40963475653868]
  DocKylin is a document-centric MLLM that performs visual content slimming at both the pixel and token levels. We introduce an Adaptive Pixel Slimming (APS) preprocessing module to perform pixel-level slimming. We also propose a novel Dynamic Token Slimming (DTS) module to conduct token-level slimming.
  arXiv  Detail & Related papers  (2024-06-27T11:28:36Z)
• CoMM: A Coherent Interleaved Image-Text Dataset for Multimodal Understanding and Generation [20.106207598099363]
  We introduce CoMM, a high-quality dataset designed to enhance the coherence, consistency, and alignment of generated multimodal content. CoMM harnesses raw data from diverse sources, focusing on instructional content and visual storytelling. Various quality evaluation metrics are designed to prove the high quality of the filtered dataset.
  arXiv  Detail & Related papers  (2024-06-15T01:27:58Z)
• TRINS: Towards Multimodal Language Models that Can Read [61.17806538631744]
  TRINS is a Text-Rich image INStruction dataset. It contains 39,153 text-rich images, captions, and 102,437 questions. We introduce a Language-vision Reading Assistant (LaRA) which is good at understanding textual content within images.
  arXiv  Detail & Related papers  (2024-06-10T18:52:37Z)
• Generating Images with Multimodal Language Models [78.6660334861137]
  We propose a method to fuse frozen text-only large language models with pre-trained image encoder and decoder models. Our model demonstrates a wide suite of multimodal capabilities: image retrieval, novel image generation, and multimodal dialogue.
  arXiv  Detail & Related papers  (2023-05-26T19:22:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.