Unlocking the Latent Canvas: Eliciting and Benchmarking Symbolic Visual Expression in LLMs

• URL: http://arxiv.org/abs/2603.14505v1
• Date: Sun, 15 Mar 2026 17:38:10 GMT
• Title: Unlocking the Latent Canvas: Eliciting and Benchmarking Symbolic Visual Expression in LLMs
• Abstract summary: Current approaches predominantly treat visual generation as an external process, relying on pixel rendering or code execution.<n>In this work, we unlock this potential through ASCII art, a compact, efficient, and text-native visual format.<n>We introduce SVE-ASCII, a unified framework designed to elicit and benchmark Visual Expression directly within the pure text space.
• Score: 24.67518539705944
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Current multimodal approaches predominantly treat visual generation as an external process, relying on pixel rendering or code execution, thereby overlooking the native visual representation capabilities latent within Large Language Models (LLMs). In this work, we unlock this potential through ASCII art, a compact, efficient, and text-native visual format. We introduce SVE-ASCII, a unified framework designed to elicit and benchmark Symbolic Visual Expression directly within the pure text space. To address the scarcity of systematic resources, we construct ASCIIArt-7K, a high-quality dataset synthesized via a novel "Seed-and-Evolve" pipeline that augments human-curated anchors through in-context stylistic editing. We further implement a unified instruction-tuning strategy that jointly optimizes for both Generation (Text-to-ASCII) and Understanding (ASCII-to-Text). Crucially, our experiments reveal a critical phenomenon regarding task duality: while it is established that perception aids generation, we provide compelling evidence that generative training significantly enhances visual comprehension. This confirms a mutually reinforcing cycle in symbolic visual processing, a relationship previously hypothesized but rarely empirically demonstrated in the visual domain. We release our dataset, the ASCIIArt-Bench benchmark, and the SVE-ASCII model, establishing a robust baseline for native text-based visual intelligence.

Related papers

• Aligning Text, Code, and Vision: A Multi-Objective Reinforcement Learning Framework for Text-to-Visualization [50.13408999553116]
  We propose RL-Text2Vis, the first reinforcement learning framework for Text2Vis generation.<n>Our method uses a novel multi-objective reward that jointly optimize textual accuracy, code validity, and visualization quality.<n>Our results establish GRPO as an effective strategy for structured, multimodal reasoning in visualization generation.
  arXiv  Detail & Related papers  (2026-01-08T04:29:07Z)
• ASCIIBench: Evaluating Language-Model-Based Understanding of Visually-Oriented Text [3.7458289725370855]
  We introduce ASCIIBench, a novel benchmark for evaluating the generation and classification of ASCII-text images.<n>Our analysis shows that cosine similarity over CLIP embeddings fails to separate most ASCII categories.<n>These findings position ASCII art as a stress test for multimodal representations.
  arXiv  Detail & Related papers  (2025-12-02T20:55:42Z)
• Revisit What You See: Disclose Language Prior in Vision Tokens for LVLM Decoding [6.612630497074871]
  Large Vision-Language Models (LVLMs) achieve strong performance across multimodal tasks by integrating visual perception with language understanding.<n>We propose ReVisiT, a training-free decoding method that references vision tokens to guide text generation.
  arXiv  Detail & Related papers  (2025-06-11T08:46:55Z)
• Text Speaks Louder than Vision: ASCII Art Reveals Textual Biases in Vision-Language Models [93.46875303598577]
  Vision-language models (VLMs) have advanced rapidly in processing multimodal information, but their ability to reconcile conflicting signals remains underexplored.<n>This work investigates how VLMs process ASCII art, a unique medium where textual elements collectively form visual patterns, potentially creating semantic-visual conflicts.
  arXiv  Detail & Related papers  (2025-04-02T10:47:07Z)
• AltGen: AI-Driven Alt Text Generation for Enhancing EPUB Accessibility [5.312946761836463]
  This paper introduces AltGen, a novel AI-driven pipeline designed to automate the generation of alt text for images.<n>By integrating state-of-the-art generative models, AltGen achieves contextually relevant and linguistically coherent alt text descriptions.<n> Experimental results demonstrate the efficacy of AltGen across diverse datasets, achieving a 97.5% reduction in accessibility errors.
  arXiv  Detail & Related papers  (2024-12-30T19:23:07Z)
• Visual Lexicon: Rich Image Features in Language Space [99.94214846451347]
  ViLex simultaneously captures rich semantic content and fine visual details.<n>ViLex generates tokens optimized for reconstructing input images using a frozen text-to-image (T2I) diffusion model.<n>As an image embedding in the language space, ViLex tokens leverage the compositionality of natural languages.
  arXiv  Detail & Related papers  (2024-12-09T18:57:24Z)
• ASCIIEval: Benchmarking Models' Visual Perception in Text Strings via ASCII Art [83.95594027644124]
  We frame the problem as a recognition task, and construct a novel benchmark, ASCIIEval.<n>It covers over 3K samples with an elaborate categorization tree, along with a training set for further enhancement.<n>Given textual input, language models shows their visual perception ability on ASCII art concepts.<n>For image inputs, we reveal that open-source MLLMs suffer from a trade-off between fine-grained text recognition and collective visual perception.
  arXiv  Detail & Related papers  (2024-10-02T16:46:01Z)
• See then Tell: Enhancing Key Information Extraction with Vision Grounding [32.445618057103324]
  We introduce STNet (See then Tell Net), a novel end-to-end model designed to deliver precise answers with relevant vision grounding.<n>To enhance the model's seeing capabilities, we collect extensive structured table recognition datasets.<n>Our approach demonstrates substantial advancements in KIE performance, achieving state-of-the-art results on publicly available datasets.
  arXiv  Detail & Related papers  (2024-09-29T06:21:05Z)
• Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenization [52.935150075484074]
  We introduce a well-designed visual tokenizer to translate the non-linguistic image into a sequence of discrete tokens like a foreign language. The resulting visual tokens encompass high-level semantics worthy of a word and also support dynamic sequence length varying from the image. This unification empowers LaVIT to serve as an impressive generalist interface to understand and generate multi-modal content simultaneously.
  arXiv  Detail & Related papers  (2023-09-09T03:01:38Z)
• Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision [57.031588264841]
  We leverage a noisy dataset of over one billion image alt-text pairs, obtained without expensive filtering or post-processing steps. A simple dual-encoder architecture learns to align visual and language representations of the image and text pairs using a contrastive loss. We show that the scale of our corpus can make up for its noise and leads to state-of-the-art representations even with such a simple learning scheme.
  arXiv  Detail & Related papers  (2021-02-11T10:08:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.