Full Glyph Images Beat Token Embeddings: A Controlled Study for Transformers

• URL: http://arxiv.org/abs/2607.03994v1
• Date: Sat, 04 Jul 2026 19:30:16 GMT
• Title: Full Glyph Images Beat Token Embeddings: A Controlled Study for Transformers
• Abstract summary: We use a Vision-based model and an index-based baseline to represent Chinese characters.<n>Across all tested decoder backbones, the Vision-based model consistently outperforms the baseline.<n>The advantage requires both spatially coherent input and a ViT encoder with 2D positional encodings.
• Score: 1.920312875157821
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Modern language models generally represent text as sequences of discrete token embeddings, an assumption deeply rooted in current practice but rarely questioned. We challenge this representation, especially for Chinese, by replacing index-based token embeddings entirely with a single rasterized image of the character sequence, processed by a vision encoder composed of a shared ResNet and a shallow Vision Transformer. To isolate the role of input representation, we construct a dual-branch controlled framework in which both a Vision-based model and an index-based baseline share an identical decoder backbone, training objective, optimizer, and data curriculum. Any performance difference is therefore attributable to the input modality only. Across all tested decoder backbones, the Vision-based model consistently outperforms the baseline, reaching a peak accuracy of 0.429 versus 0.355 for the index-based baseline,that is, a 21% relative improvement, while converging in about half the number of training epochs. The advantage emerges especially within the first five epochs (under 21% of total data) and persists under moderate character corruption: the corrupted Vision model matches the clean index-based baseline. Ablation studies reveal that the advantage requires both spatially coherent input and a ViT encoder with 2D positional encodings. A cross-script comparison on English shows the advantage does not transfer directly to alphabetic writing systems, suggesting that the uniform visual density and radical structure of Chinese characters are enabling conditions. These findings suggest that transformers are more modality-agnostic than commonly assumed, and that discrete tokenization is not a fundamental requirement for Chinese language modeling.

Related papers

• Visual Token Codec: Unleashing Spatial Redundancy for ViT Feature Coding [49.540751364954666]
  Visual Token Codec separates global and patch tokens into dedicated coding paths.<n>We show that VTC consistently outperforms representative ViT feature coding baselines on classification, segmentation, and detection tasks.
  arXiv  Detail & Related papers  (2026-08-09T17:32:04Z)
• Same Semantics, Different Paths: Self-Improving Alignment for Vision-Text Compression [7.944386069136441]
  Vision-Text Compression (VTC) renders long texts into images and encodes them through the vision encoder (ViT)<n>ViT captures visual attributes (glyphs, font sizes, layout) rather than linguistic semantics, causing rendered-image representations to diverge from native-text representations.<n>We propose SPIRAL, a self-supervised alignment framework that closes this gap using only the model's own text-path behavior as supervision.
  arXiv  Detail & Related papers  (2026-08-03T12:08:56Z)
• MAViE: A Multi-scale Adaptive Vision Encoder for Fine-grained Visual Perception and Efficient Multimodal Reasoning [0.0]
  method fuses shallow, intermediate, and deep features from a vision Transformer to preserve global semantics.<n>It then performs question-conditioned token routing according to question relevance, local information content, global semantics, and spatial coverage.<n>It reduces the average number of SigLIP-SO400M visual tokens from 729 to 146 (approximately 80.0%) and improves the mean score on VQAv2, GQA, TextVQA, ScienceQA-IMG, and MMBench by 2.2 percentage points.
  arXiv  Detail & Related papers  (2026-07-27T13:36:08Z)
• RADIO1D: Elastic Representations for Condensed Vision Modeling [69.84453279129937]
  This paper challenges the assumption that vision-language models (VLMs) require fixed patch-based 2D vision features.<n>We introduce RADIO1D, which compresses images into a compact, variable-length 1D token sequence using multi-teacher knowledge distillation and an autoencoder design.<n>The resulting representations exhibit strong hierarchical summarization, enabling accurate scene understanding - even with a single token - and support improved composition-aware image retrieval.
  arXiv  Detail & Related papers  (2026-07-03T22:58:54Z)
• Entropy-Aware Structural Alignment for Zero-Shot Handwritten Chinese Character Recognition [7.632962062462334]
  Zero-shot Handwritten Chinese Character Recognition aims to recognize unseen characters by leveraging radical-based semantic compositions.<n>We propose an Entropy-Aware Structural Alignment Network that bridges the visual-semantic gap through information-theoretic modeling.<n>Our method establishes new state-of-the-art performance, achieving an accuracy of 55.04% on the ICDAR 2013 dataset.
  arXiv  Detail & Related papers  (2026-02-03T16:08:40Z)
• A Transformer Based Handwriting Recognition System Jointly Using Online and Offline Features [8.419663258260671]
  We introduce an end-to-end network that performs early fusion of offline images and online stroke data.<n>Our approach achieves state-of-the-art accuracy, exceeding previous bests by up to 1%.
  arXiv  Detail & Related papers  (2025-06-25T08:58:47Z)
• Efficient Medical Vision-Language Alignment Through Adapting Masked Vision Models [29.571937393873444]
  Cross-modal contrastive learning (CLIP) methods suffer from suboptimal visual representation capabilities.<n>We propose ALTA (ALign Through Adapting), an efficient vision-language alignment method that utilizes only about 8% of the trainable parameters.<n>ALTA superior performance in vision-language matching tasks like retrieval and zero-shot classification by adapting the pretrained vision model from masked record modeling.
  arXiv  Detail & Related papers  (2025-06-10T17:02:27Z)
• Token-Shuffle: Towards High-Resolution Image Generation with Autoregressive Models [92.18057318458528]
  Token-Shuffle is a novel method that reduces the number of image tokens in Transformer.<n>Our strategy requires no additional pretrained text-encoder and enables MLLMs to support extremely high-resolution image synthesis.<n>In GenAI-benchmark, our 2.7B model achieves 0.77 overall score on hard prompts, outperforming AR models LlamaGen by 0.18 and diffusion models LDM by 0.15.
  arXiv  Detail & Related papers  (2025-04-24T17:59:56Z)
• Translatotron-V(ison): An End-to-End Model for In-Image Machine Translation [81.45400849638347]
  In-image machine translation (IIMT) aims to translate an image containing texts in source language into an image containing translations in target language. In this paper, we propose an end-to-end IIMT model consisting of four modules. Our model achieves competitive performance compared to cascaded models with only 70.9% of parameters, and significantly outperforms the pixel-level end-to-end IIMT model.
  arXiv  Detail & Related papers  (2024-07-03T08:15:39Z)
• Do Vision and Language Encoders Represent the World Similarly? [22.70701869402434]
  Aligned text-image encoders such as CLIP have become the de facto model for vision-language tasks. We find that the representation spaces of unaligned and aligned encoders are semantically similar. In the absence of statistical similarity in aligned encoders like CLIP, we show that a possible matching of unaligned encoders exists without any training.
  arXiv  Detail & Related papers  (2024-01-10T15:51:39Z)
• Scaling Autoregressive Models for Content-Rich Text-to-Image Generation [95.02406834386814]
  Parti treats text-to-image generation as a sequence-to-sequence modeling problem. Parti uses a Transformer-based image tokenizer, ViT-VQGAN, to encode images as sequences of discrete tokens. PartiPrompts (P2) is a new holistic benchmark of over 1600 English prompts.
  arXiv  Detail & Related papers  (2022-06-22T01:11:29Z)
• Single-Stream Multi-Level Alignment for Vision-Language Pretraining [103.09776737512078]
  We propose a single stream model that aligns the modalities at multiple levels. We achieve this using two novel tasks: symmetric cross-modality reconstruction and a pseudo-labeled key word prediction. We demonstrate top performance on a set of Vision-Language downstream tasks such as zero-shot/fine-tuned image/text retrieval, referring expression, and VQA.
  arXiv  Detail & Related papers  (2022-03-27T21:16:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.