LogicOCR: Do Your Large Multimodal Models Excel at Logical Reasoning on Text-Rich Images?

• URL: http://arxiv.org/abs/2505.12307v1
• Date: Sun, 18 May 2025 08:39:37 GMT
• Title: LogicOCR: Do Your Large Multimodal Models Excel at Logical Reasoning on Text-Rich Images?
• Authors: Maoyuan Ye, Jing Zhang, Juhua Liu, Bo Du, Dacheng Tao,
• Abstract summary: We introduce LogicOCR, a benchmark comprising 1,100 multiple-choice questions designed to evaluate LMMs' logical reasoning abilities on text-rich images.<n>We develop a scalable, automated pipeline to convert a text corpus into multimodal samples.<n>We evaluate a range of representative open-source and proprietary LMMs under both Chain-of-Thought (CoT) and direct-answer settings.
• Score: 80.4577892387028
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Recent advances in Large Multimodal Models (LMMs) have significantly improved their reasoning and Optical Character Recognition (OCR) capabilities. However, their performance on complex logical reasoning tasks involving text-rich images remains underexplored. To bridge this gap, we introduce LogicOCR, a benchmark comprising 1,100 multiple-choice questions designed to evaluate LMMs' logical reasoning abilities on text-rich images, while minimizing reliance on domain-specific knowledge (e.g., mathematics). We construct LogicOCR by curating a text corpus from the Chinese National Civil Servant Examination and develop a scalable, automated pipeline to convert it into multimodal samples. First, we design prompt templates to steer GPT-Image-1 to generate images with diverse backgrounds, interleaved text-illustration layouts, and varied fonts, ensuring contextual relevance and visual realism. Then, the generated images are manually verified, with low-quality examples discarded. We evaluate a range of representative open-source and proprietary LMMs under both Chain-of-Thought (CoT) and direct-answer settings. Our multi-dimensional analysis reveals key insights, such as the impact of test-time scaling, input modality differences, and sensitivity to visual-text orientation. Notably, LMMs still lag in multimodal reasoning compared to text-only inputs, indicating that they have not fully bridged visual reading with reasoning. We hope LogicOCR will serve as a valuable resource for advancing multimodal reasoning research. The dataset is available at https://github.com/MiliLab/LogicOCR.

Related papers

• Towards Text-Image Interleaved Retrieval [49.96332254241075]
  We introduce the text-image interleaved retrieval (TIIR) task, where the query and document are interleaved text-image sequences.<n>We construct a TIIR benchmark based on naturally interleaved wikiHow tutorials, where a specific pipeline is designed to generate interleaved queries.<n>We propose a novel Matryoshka Multimodal Embedder (MME), which compresses the number of visual tokens at different granularity.
  arXiv  Detail & Related papers  (2025-02-18T12:00:47Z)
• Boosting Text-To-Image Generation via Multilingual Prompting in Large Multimodal Models [43.16111789538798]
  We build parallel multilingual prompts aimed at harnessing the multilingual capabilities of large multimodal models (LMMs)<n> Experiments on two LMMs across 3 benchmarks show that our method, PMT2I achieves, superior performance in general, compositional, and fine-grained assessments.
  arXiv  Detail & Related papers  (2025-01-13T06:41:23Z)
• MC-Bench: A Benchmark for Multi-Context Visual Grounding in the Era of MLLMs [61.56904387052982]
  This paper proposes a new visual grounding task called multi-context visual grounding.<n>It aims to localize instances of interest across multiple images based on open-ended text prompts.<n>We benchmark over 20 state-of-the-art MLLMs and foundation models with potential multi-context visual grounding capabilities.
  arXiv  Detail & Related papers  (2024-10-16T07:52:57Z)
• Leopard: A Vision Language Model For Text-Rich Multi-Image Tasks [62.758680527838436]
  Leopard is a vision-language model for handling vision-language tasks involving multiple text-rich images. First, we curated about one million high-quality multimodal instruction-tuning data, tailored to text-rich, multi-image scenarios. Second, we developed an adaptive high-resolution multi-image encoding module to dynamically optimize the allocation of visual sequence length.
  arXiv  Detail & Related papers  (2024-10-02T16:55:01Z)
• MMR: Evaluating Reading Ability of Large Multimodal Models [52.953316772123586]
  Large multimodal models (LMMs) have demonstrated impressive capabilities in understanding various types of image, including text-rich images. Current benchmarks fail to accurately reflect performance of different models. We propose the Multi-Modal Reading (MMR) benchmark in 11 diverse tasks to evaluate LMMs for text-rich image understanding.
  arXiv  Detail & Related papers  (2024-08-26T19:26:50Z)
• Efficient End-to-End Visual Document Understanding with Rationale Distillation [43.28272448274713]
  Rationale Distillation (RD) trains a small student model to predict both rationales and answers. RD outperforms the base model by 4-5% absolute accuracy with only 1% higher computational cost.
  arXiv  Detail & Related papers  (2023-11-16T06:50:26Z)
• MultiQG-TI: Towards Question Generation from Multi-modal Sources [4.913248451323163]
  We study the new problem of automatic question generation from multi-modal sources containing images and texts. We propose a simple solution for our new problem, called MultiQG-TI, which enables a text-only question generator to process visual input. We demonstrate that MultiQG-TI significantly outperforms ChatGPT with few-shot prompting, despite having hundred-times less trainable parameters.
  arXiv  Detail & Related papers  (2023-07-07T08:14:15Z)
• Structured Multimodal Attentions for TextVQA [57.71060302874151]
  We propose an end-to-end structured multimodal attention (SMA) neural network to mainly solve the first two issues above. SMA first uses a structural graph representation to encode the object-object, object-text and text-text relationships appearing in the image, and then designs a multimodal graph attention network to reason over it. Our proposed model outperforms the SoTA models on TextVQA dataset and two tasks of ST-VQA dataset among all models except pre-training based TAP.
  arXiv  Detail & Related papers  (2020-06-01T07:07:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.