ASCIIBench: Evaluating Language-Model-Based Understanding of Visually-Oriented Text

• URL: http://arxiv.org/abs/2512.04125v1
• Date: Tue, 02 Dec 2025 20:55:42 GMT
• Title: ASCIIBench: Evaluating Language-Model-Based Understanding of Visually-Oriented Text
• Abstract summary: We introduce ASCIIBench, a novel benchmark for evaluating the generation and classification of ASCII-text images.<n>Our analysis shows that cosine similarity over CLIP embeddings fails to separate most ASCII categories.<n>These findings position ASCII art as a stress test for multimodal representations.
• Score: 3.7458289725370855
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Large language models (LLMs) have demonstrated several emergent behaviors with scale, including reasoning and fluency in long-form text generation. However, they continue to struggle with tasks requiring precise spatial and positional reasoning. ASCII art, a symbolic medium where characters encode structure and form, provides a unique probe of this limitation. We introduce ASCIIBench, a novel benchmark for evaluating both the generation and classification of ASCII-text images. ASCIIBench consists of a filtered dataset of 5,315 class-labeled ASCII images and is, to our knowledge, the first publicly available benchmark of its kind. Alongside the dataset, we release weights for a fine-tuned CLIP model adapted to capture ASCII structure, enabling the evaluation of LLM-generated ASCII art. Our analysis shows that cosine similarity over CLIP embeddings fails to separate most ASCII categories, yielding chance-level performance even for low-variance classes. In contrast, classes with high internal mean similarity exhibit clear discriminability, revealing that the bottleneck lies in representation rather than generational variance. These findings position ASCII art as a stress test for multimodal representations and motivate the development of new embedding methods or evaluation metrics tailored to symbolic visual modalities. All resources are available at https://github.com/ASCIIBench/ASCIIBench.

Related papers

• Unlocking the Latent Canvas: Eliciting and Benchmarking Symbolic Visual Expression in LLMs [24.67518539705944]
  Current approaches predominantly treat visual generation as an external process, relying on pixel rendering or code execution.<n>In this work, we unlock this potential through ASCII art, a compact, efficient, and text-native visual format.<n>We introduce SVE-ASCII, a unified framework designed to elicit and benchmark Visual Expression directly within the pure text space.
  arXiv  Detail & Related papers  (2026-03-15T17:38:10Z)
• Beyond one-hot encoding? Journey into compact encoding for large multi-class segmentation [3.731545953583865]
  We propose a family of binary encoding approaches instead of one-hot encoding to reduce the computational complexity and memory requirements to logarithmic in the number of classes.<n>We apply the methods to the use case of whole brain parcellation with 108 classes based on 3D MRI images.
  arXiv  Detail & Related papers  (2025-10-01T08:53:39Z)
• Evaluating Machine Learning Approaches for ASCII Art Generation [0.0]
  This paper explores the application of modern machine learning methods to generate structured ASCII art.<n>We focus on three key criteria: fidelity, character classification accuracy, and output quality.<n>Our results show that complex neural network architectures often fall short in producing high-quality ASCII art.
  arXiv  Detail & Related papers  (2025-03-18T16:07:29Z)
• Classification Done Right for Vision-Language Pre-Training [66.90286715149786]
  We introduce SuperClass, a super simple classification method for vision-language pre-training on image-text data. SuperClass directly utilizes tokenized raw text as supervised classification labels, without the need for additional text filtering or selection. SuperClass demonstrated superior performance on various downstream tasks, including classic computer vision benchmarks and vision language downstream tasks.
  arXiv  Detail & Related papers  (2024-11-05T18:58:15Z)
• ASCIIEval: Benchmarking Models' Visual Perception in Text Strings via ASCII Art [83.95594027644124]
  We frame the problem as a recognition task, and construct a novel benchmark, ASCIIEval.<n>It covers over 3K samples with an elaborate categorization tree, along with a training set for further enhancement.<n>Given textual input, language models shows their visual perception ability on ASCII art concepts.<n>For image inputs, we reveal that open-source MLLMs suffer from a trade-off between fine-grained text recognition and collective visual perception.
  arXiv  Detail & Related papers  (2024-10-02T16:46:01Z)
• Focus on the Whole Character: Discriminative Character Modeling for Scene Text Recognition [28.93482989766411]
  We propose a novel method that enriches the character features to enhance discriminability of characters. CACE introduces a decay matrix in each block to explicitly guide the attention region for each token. I2CL improves the discnative capability of features by learning a long-term memory unit for each character category.
  arXiv  Detail & Related papers  (2024-07-08T02:33:29Z)
• GET: Unlocking the Multi-modal Potential of CLIP for Generalized Category Discovery [50.564146730579424]
  We propose a Text Embedding Synthesizer (TES) to generate pseudo text embeddings for unlabelled samples.<n>Our method unlocks the multi-modal potentials of CLIP and outperforms the baseline methods by a large margin on all GCD benchmarks.
  arXiv  Detail & Related papers  (2024-03-15T02:40:13Z)
• Improving Zero-Shot Generalization for CLIP with Synthesized Prompts [135.4317555866831]
  Most existing methods require labeled data for all classes, which may not hold in real-world applications. We propose a plug-and-play generative approach called textbfSynttextbfHestextbfIzed textbfPrompts(textbfSHIP) to improve existing fine-tuning methods.
  arXiv  Detail & Related papers  (2023-07-14T15:15:45Z)
• GENIUS: Sketch-based Language Model Pre-training via Extreme and Selective Masking for Text Generation and Augmentation [76.7772833556714]
  We introduce GENIUS: a conditional text generation model using sketches as input. GENIUS is pre-trained on a large-scale textual corpus with a novel reconstruction from sketch objective. We show that GENIUS can be used as a strong and ready-to-use data augmentation tool for various natural language processing (NLP) tasks.
  arXiv  Detail & Related papers  (2022-11-18T16:39:45Z)
• byteSteady: Fast Classification Using Byte-Level n-Gram Embeddings [77.6701264226519]
  We introduce byteSteady, a fast model for classification using byte-level n-gram embeddings. A straightforward application of byteSteady is text classification. We also apply byteSteady to one type of non-language data -- DNA sequences for gene classification.
  arXiv  Detail & Related papers  (2021-06-24T20:14:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.