Reading $\neq$ Seeing: Diagnosing and Closing the Typography Gap in Vision-Language Models

• URL: http://arxiv.org/abs/2603.08497v1
• Date: Mon, 09 Mar 2026 15:31:47 GMT
• Title: Reading $\neq$ Seeing: Diagnosing and Closing the Typography Gap in Vision-Language Models
• Abstract summary: Vision-Language Models achieve near-perfect accuracy at reading text in images, yet prove largely typography-blind.<n>We systematically investigate this gap by evaluating font family, size, style, and color recognition across 26 fonts, four scripts, and three difficulty levels.<n>We release our evaluation framework, data, and fine-tuning recipe to support progress in closing the typographic gap in vision-language understanding.
• Score: 16.13083848552738
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-Language Models achieve near-perfect accuracy at reading text in images, yet prove largely typography-blind: capable of recognizing what text says, but not how it looks. We systematically investigate this gap by evaluating font family, size, style, and color recognition across 26 fonts, four scripts, and three difficulty levels. Our evaluation of 15 state-of-the-art VLMs reveals a striking perception hierarchy: color recognition is near-perfect, yet font style detection remains universally poor. We further find that model scale fails to predict performance and that accuracy is uniform across difficulty levels, together pointing to a training-data omission rather than a capacity ceiling. LoRA fine-tuning on a small set of synthetic samples substantially improves an open-source model, narrowing the gap to the best closed-source system and surpassing it on font size recognition. Font style alone remains resistant to fine-tuning, suggesting that relational visual reasoning may require architectural innovation beyond current patch-based encoders. We release our evaluation framework, data, and fine-tuning recipe to support progress in closing the typographic gap in vision-language understanding.

Related papers

• STELLAR: Scene Text Editor for Low-Resource Languages and Real-World Data [3.622341086373503]
  We propose STELLAR, a Scene Text Editor for Low-resource LAnguages and Real-world data.<n> STELLAR enables reliable multilingual editing through a language-adaptive glyph encoder and a multi-stage training strategy.<n>We also construct a new dataset, STIPLAR(Scene Text Image Pairs of Low-resource lAnguages and Real-world data), for training and evaluation.
  arXiv  Detail & Related papers  (2025-11-13T05:18:03Z)
• Visible Yet Unreadable: A Systematic Blind Spot of Vision Language Models Across Writing Systems [25.47053654117902]
  We investigate whether advanced vision language models (VLMs) share this resilience.<n>We construct two psychophysics inspired benchmarks across distinct writing systems.<n>Despite strong performance on clean text, contemporary VLMs show a severe drop under these perturbations.
  arXiv  Detail & Related papers  (2025-09-04T05:35:32Z)
• Vocabulary-free Fine-grained Visual Recognition via Enriched Contextually Grounded Vision-Language Model [52.01031460230826]
  Traditional approaches rely heavily on fixed vocabularies and closed-set classification paradigms.<n>Recent research has demonstrated that combining large language models with vision-language models (VLMs) makes open-set recognition possible.<n>We propose our training-free method, Enriched-FineR, which demonstrates state-of-the-art results in fine-grained visual recognition.
  arXiv  Detail & Related papers  (2025-07-30T20:06:01Z)
• Texture or Semantics? Vision-Language Models Get Lost in Font Recognition [53.14935624161711]
  We introduce the Font Recognition Benchmark (FRB), a compact and well-structured dataset comprising 15 commonly used fonts.<n> FRB includes two versions: (i) an easy version, where 10 sentences are rendered in different fonts, and (ii) a hard version, where each text sample consists of the names of the 15 fonts themselves.<n>Through extensive evaluation of various VLMs on font recognition tasks, we arrive at the following key findings.
  arXiv  Detail & Related papers  (2025-03-31T06:33:21Z)
• Extract Free Dense Misalignment from CLIP [7.0247398611254175]
  This work proposes a novel approach, dubbed CLIP4DM, for detecting dense misalignments from pre-trained CLIP.<n>We revamp the gradient-based attribution computation method, enabling negative gradient of individual text tokens to indicate misalignment.<n>Our method demonstrates state-of-the-art performance among zero-shot models and competitive performance with fine-tuned models.
  arXiv  Detail & Related papers  (2024-12-24T12:51:05Z)
• ASCIIEval: Benchmarking Models' Visual Perception in Text Strings via ASCII Art [83.95594027644124]
  We frame the problem as a recognition task, and construct a novel benchmark, ASCIIEval.<n>It covers over 3K samples with an elaborate categorization tree, along with a training set for further enhancement.<n>Given textual input, language models shows their visual perception ability on ASCII art concepts.<n>For image inputs, we reveal that open-source MLLMs suffer from a trade-off between fine-grained text recognition and collective visual perception.
  arXiv  Detail & Related papers  (2024-10-02T16:46:01Z)
• Capturing Style in Author and Document Representation [4.323709559692927]
  We propose a new architecture that learns embeddings for both authors and documents with a stylistic constraint.<n>We evaluate our method on three datasets: a literary corpus extracted from the Gutenberg Project, the Blog Authorship and IMDb62.
  arXiv  Detail & Related papers  (2024-07-18T10:01:09Z)
• Do LVLMs Understand Charts? Analyzing and Correcting Factual Errors in Chart Captioning [90.13978453378768]
  We introduce a comprehensive typology of factual errors in generated chart captions. A large-scale human annotation effort provides insight into the error patterns and frequencies in captions crafted by various chart captioning models. Our analysis reveals that even state-of-the-art models, including GPT-4V, frequently produce captions laced with factual inaccuracies.
  arXiv  Detail & Related papers  (2023-12-15T19:16:21Z)
• Linguistic More: Taking a Further Step toward Efficient and Accurate Scene Text Recognition [92.6211155264297]
  Vision models have gained increasing attention due to their simplicity and efficiency in Scene Text Recognition (STR) task. Recent vision models suffer from two problems: (1) the pure vision-based query results in attention drift, which usually causes poor recognition and is summarized as linguistic insensitive drift (LID) problem in this paper. We propose a $textbfL$inguistic $textbfP$erception $textbfV$ision model (LPV) which explores the linguistic capability of vision model for accurate text recognition.
  arXiv  Detail & Related papers  (2023-05-09T02:52:47Z)
• DS-Fusion: Artistic Typography via Discriminated and Stylized Diffusion [10.75789076591325]
  We introduce a novel method to automatically generate an artistic typography by stylizing one or more letter fonts. Our approach utilizes large language models to bridge texts and visual images for stylization and build an unsupervised generative model.
  arXiv  Detail & Related papers  (2023-03-16T19:12:52Z)
• Visually Grounded Compound PCFGs [65.04669567781634]
  Exploiting visual groundings for language understanding has recently been drawing much attention. We study visually grounded grammar induction and learn a constituency from both unlabeled text and its visual captions.
  arXiv  Detail & Related papers  (2020-09-25T19:07:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.