Emergence of Text Readability in Vision Language Models

• URL: http://arxiv.org/abs/2506.19389v1
• Date: Tue, 24 Jun 2025 07:35:32 GMT
• Title: Emergence of Text Readability in Vision Language Models
• Authors: Jaeyoo Park, Sanghyuk Chun, Wonjae Kim, Sangdoo Yun, Bohyung Han,
• Abstract summary: We investigate how the ability to recognize textual content within images emerges during the training of Vision-Language Models (VLMs)<n>Our analysis reveals a critical phenomenon: the ability to read textual information in a given image emerges abruptly after substantial training iterations.<n>This delayed emergence may reflect how contrastive learning tends to initially prioritize general semantic understanding, with text-specific symbolic processing developing later.
• Score: 69.10470691283281
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We investigate how the ability to recognize textual content within images emerges during the training of Vision-Language Models (VLMs). Our analysis reveals a critical phenomenon: the ability to read textual information in a given image \textbf{(text readability)} emerges abruptly after substantial training iterations, in contrast to semantic content understanding which develops gradually from the early stages of training. This delayed emergence may reflect how contrastive learning tends to initially prioritize general semantic understanding, with text-specific symbolic processing developing later. Interestingly, the ability to match images with rendered text develops even slower, indicating a deeper need for semantic integration. These findings highlight the need for tailored training strategies to accelerate robust text comprehension in VLMs, laying the groundwork for future research on optimizing multimodal learning.

Related papers

• Improving Fine-grained Visual Understanding in VLMs through Text-Only Training [0.0]
  We investigate the feasibility of enhancing fine-grained visual understanding in Visual-Language Models (VLMs) through text-only training.<n>We conduct comprehensive experiments on two distinct domains: fine-grained species classification and cultural visual understanding tasks.<n>Our findings demonstrate that text-only training can be comparable to conventional image-text training while significantly reducing computational costs.
  arXiv  Detail & Related papers  (2024-12-17T14:18:50Z)
• Improving In-Context Learning in Diffusion Models with Visual Context-Modulated Prompts [83.03471704115786]
  We introduce improved Prompt Diffusion (iPromptDiff) in this study. iPromptDiff integrates an end-to-end trained vision encoder that converts visual context into an embedding vector. We show that a diffusion-based vision foundation model, when equipped with this visual context-modulated text guidance and a standard ControlNet structure, exhibits versatility and robustness across a variety of training tasks.
  arXiv  Detail & Related papers  (2023-12-03T14:15:52Z)
• Rewrite Caption Semantics: Bridging Semantic Gaps for Language-Supervised Semantic Segmentation [100.81837601210597]
  We propose Concept Curation (CoCu) to bridge the gap between visual and textual semantics in pre-training data. CoCu achieves superb zero-shot transfer performance and greatly boosts language-supervised segmentation baseline by a large margin.
  arXiv  Detail & Related papers  (2023-09-24T00:05:39Z)
• Visually-Situated Natural Language Understanding with Contrastive Reading Model and Frozen Large Language Models [24.456117679941816]
  Contrastive Reading Model (Cream) is a novel neural architecture designed to enhance the language-image understanding capability of Large Language Models (LLMs) Our approach bridges the gap between vision and language understanding, paving the way for the development of more sophisticated Document Intelligence Assistants.
  arXiv  Detail & Related papers  (2023-05-24T11:59:13Z)
• Fine-Grained Semantically Aligned Vision-Language Pre-Training [151.7372197904064]
  Large-scale vision-language pre-training has shown impressive advances in a wide range of downstream tasks. Existing methods mainly model the cross-modal alignment by the similarity of the global representations of images and texts. We introduce LO, a fine-grained semantically aLigned visiOn-langUage PrE-training framework, which learns fine-grained semantic alignment from the novel perspective of game-theoretic interactions.
  arXiv  Detail & Related papers  (2022-08-04T07:51:48Z)
• Vision-Language Pre-Training for Boosting Scene Text Detectors [57.08046351495244]
  We specifically adapt vision-language joint learning for scene text detection. We propose to learn contextualized, joint representations through vision-language pre-training. The pre-trained model is able to produce more informative representations with richer semantics.
  arXiv  Detail & Related papers  (2022-04-29T03:53:54Z)
• Language Matters: A Weakly Supervised Pre-training Approach for Scene Text Detection and Spotting [69.77701325270047]
  This paper presents a weakly supervised pre-training method that can acquire effective scene text representations. Our network consists of an image encoder and a character-aware text encoder that extract visual and textual features. Experiments show that our pre-trained model improves F-score by +2.5% and +4.8% while transferring its weights to other text detection and spotting networks.
  arXiv  Detail & Related papers  (2022-03-08T08:10:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.