Visual Text Processing: A Comprehensive Review and Unified Evaluation

• URL: http://arxiv.org/abs/2504.21682v1
• Date: Wed, 30 Apr 2025 14:19:29 GMT
• Title: Visual Text Processing: A Comprehensive Review and Unified Evaluation
• Authors: Yan Shu, Weichao Zeng, Fangmin Zhao, Zeyu Chen, Zhenhang Li, Xiaomeng Yang, Yu Zhou, Paolo Rota, Xiang Bai, Lianwen Jin, Xu-Cheng Yin, Nicu Sebe,
• Abstract summary: We present a comprehensive, multi-perspective analysis of recent advancements in visual text processing.<n>Our aim is to establish this work as a fundamental resource that fosters future exploration and innovation in the dynamic field of visual text processing.
• Score: 99.57846940547171
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Visual text is a crucial component in both document and scene images, conveying rich semantic information and attracting significant attention in the computer vision community. Beyond traditional tasks such as text detection and recognition, visual text processing has witnessed rapid advancements driven by the emergence of foundation models, including text image reconstruction and text image manipulation. Despite significant progress, challenges remain due to the unique properties that differentiate text from general objects. Effectively capturing and leveraging these distinct textual characteristics is essential for developing robust visual text processing models. In this survey, we present a comprehensive, multi-perspective analysis of recent advancements in visual text processing, focusing on two key questions: (1) What textual features are most suitable for different visual text processing tasks? (2) How can these distinctive text features be effectively incorporated into processing frameworks? Furthermore, we introduce VTPBench, a new benchmark that encompasses a broad range of visual text processing datasets. Leveraging the advanced visual quality assessment capabilities of multimodal large language models (MLLMs), we propose VTPScore, a novel evaluation metric designed to ensure fair and reliable evaluation. Our empirical study with more than 20 specific models reveals substantial room for improvement in the current techniques. Our aim is to establish this work as a fundamental resource that fosters future exploration and innovation in the dynamic field of visual text processing. The relevant repository is available at https://github.com/shuyansy/Visual-Text-Processing-survey.

Related papers

• TextCrafter: Accurately Rendering Multiple Texts in Complex Visual Scenes [30.947323186487885]
  Complex Visual Text Generation (CVTG) centers on generating intricate textual content distributed across diverse regions within visual images. InG, image generation models often rendering distorted and blurred visual text or missing some visual text. We propose TextCrafter, a novel multi-visual text rendering method.
  arXiv  Detail & Related papers  (2025-03-30T14:36:55Z)
• TextInVision: Text and Prompt Complexity Driven Visual Text Generation Benchmark [61.412934963260724]
  Existing diffusion-based text-to-image models often struggle to accurately embed text within images.<n>We introduce TextInVision, a large-scale, text and prompt complexity driven benchmark to evaluate the ability of diffusion models to integrate visual text into images.
  arXiv  Detail & Related papers  (2025-03-17T21:36:31Z)
• StrucTexTv3: An Efficient Vision-Language Model for Text-rich Image Perception, Comprehension, and Beyond [68.0107158115377]
  We have crafted an efficient vision-language model, StrucTexTv3, tailored to tackle various intelligent tasks for text-rich images. We enhance the perception and comprehension abilities of StrucTexTv3 through instruction learning. Our method achieved SOTA results in text-rich image perception tasks, and significantly improved performance in comprehension tasks.
  arXiv  Detail & Related papers  (2024-05-31T16:55:04Z)
• Visual Text Meets Low-level Vision: A Comprehensive Survey on Visual Text Processing [4.057550183467041]
  The field of visual text processing has experienced a surge in research, driven by the advent of fundamental generative models. We present a comprehensive, multi-perspective analysis of recent advancements in this field.
  arXiv  Detail & Related papers  (2024-02-05T15:13:20Z)
• Towards Improving Document Understanding: An Exploration on Text-Grounding via MLLMs [96.54224331778195]
  We present a text-grounding document understanding model, termed TGDoc, which enhances MLLMs with the ability to discern the spatial positioning of text within images. We formulate instruction tuning tasks including text detection, recognition, and spotting to facilitate the cohesive alignment between the visual encoder and large language model. Our method achieves state-of-the-art performance across multiple text-rich benchmarks, validating the effectiveness of our method.
  arXiv  Detail & Related papers  (2023-11-22T06:46:37Z)
• OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models [122.27878464009181]
  We conducted a comprehensive evaluation of Large Multimodal Models, such as GPT4V and Gemini, in various text-related visual tasks. OCRBench contains 29 datasets, making it the most comprehensive OCR evaluation benchmark available.
  arXiv  Detail & Related papers  (2023-05-13T11:28:37Z)
• Learning the Visualness of Text Using Large Vision-Language Models [42.75864384249245]
  Visual text evokes an image in a person's mind, while non-visual text fails to do so. A method to automatically detect visualness in text will enable text-to-image retrieval and generation models to augment text with relevant images. We curate a dataset of 3,620 English sentences and their visualness scores provided by multiple human annotators.
  arXiv  Detail & Related papers  (2023-05-11T17:45:16Z)
• Vision-Language Pre-Training for Boosting Scene Text Detectors [57.08046351495244]
  We specifically adapt vision-language joint learning for scene text detection. We propose to learn contextualized, joint representations through vision-language pre-training. The pre-trained model is able to produce more informative representations with richer semantics.
  arXiv  Detail & Related papers  (2022-04-29T03:53:54Z)
• Language Matters: A Weakly Supervised Pre-training Approach for Scene Text Detection and Spotting [69.77701325270047]
  This paper presents a weakly supervised pre-training method that can acquire effective scene text representations. Our network consists of an image encoder and a character-aware text encoder that extract visual and textual features. Experiments show that our pre-trained model improves F-score by +2.5% and +4.8% while transferring its weights to other text detection and spotting networks.
  arXiv  Detail & Related papers  (2022-03-08T08:10:45Z)
• Multi-Modal Reasoning Graph for Scene-Text Based Fine-Grained Image Classification and Retrieval [8.317191999275536]
  This paper focuses on leveraging multi-modal content in the form of visual and textual cues to tackle the task of fine-grained image classification and retrieval. We employ a Graph Convolutional Network to perform multi-modal reasoning and obtain relationship-enhanced features by learning a common semantic space between salient objects and text found in an image.
  arXiv  Detail & Related papers  (2020-09-21T12:31:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.