Seeing the Unseen: Visual Similarity for Pixel Language Model Adaptation

• URL: http://arxiv.org/abs/2608.30541v1
• Date: Mon, 31 Aug 2026 10:11:05 GMT
• Title: Seeing the Unseen: Visual Similarity for Pixel Language Model Adaptation
• Abstract summary: Pixel-based language models (LMs) replace traditional tokenizers by processing rendered images of text.<n>We analyze how continued pre-training of pixel-based LMs is influenced by data scale, initial script exposure, and cross-lingual transfer from languages written in other Brahmic scripts.
• Score: 7.634308270954448
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Pixel-based language models (LMs) replace traditional tokenizers by processing rendered images of text, making cross-lingual transfer heavily dependent on the visual and structural properties of writing systems. However, the dynamics of adapting these models to low-resource languages with complex morphology and written in unique scripts are not yet explored. Using Tibetan as a case study, we analyze how continued pre-training of pixel-based LMs is influenced by data scale, initial script exposure, and cross-lingual transfer from languages written in other Brahmic scripts. We introduce four rendering-level metrics to quantify visual script similarity. We evaluate downstream performance across three tasks. Our results show that higher orthographic proximity enhances semantic transfer, even under severe data constraints. Additionally, we find a performance asymmetry based on the pre-training starting point: while multilingual pre-training PIXEL-M4 has stronger initial performance, its capacity for subsequent adaptation seems to be constrained, whereas adapting a monolingual model PIXEL with mixed scripts yields more gains on sentence-level tasks. Our metrics and case study offer empirical observations that could help inform data selection and script adaptation choices when working with pixel-based models in similar low-resource settings.

Related papers

• MIXAR: Scaling Autoregressive Pixel-based Language Models to Multiple Languages and Scripts [66.21710415920332]
  MIXAR is the first generative pixel-based language model trained on eight different languages.<n>We show how MIXAR is robust to languages never seen during the training.
  arXiv  Detail & Related papers  (2026-04-13T14:53:19Z)
• Brazilian Portuguese Image Captioning with Transformers: A Study on Cross-Native-Translated Dataset [41.02122939888977]
  This study proposes a cross-native evaluation of Transformer-based vision and language models for Brazilian Portuguese IC.<n>We use a version of Flickr30K comprised of captions manually created by native Brazilian Portuguese speakers and compare it to a version with captions automatically translated from English to Portuguese.<n>Our findings show that Swin-DistilBERTimbau consistently outperforms other models, demonstrating strong generalization across datasets.
  arXiv  Detail & Related papers  (2026-01-30T23:12:03Z)
• Enhancing Robustness of Autoregressive Language Models against Orthographic Attacks via Pixel-based Approach [51.95266411355865]
  Autoregressive language models are vulnerable to orthographic attacks.<n>This vulnerability stems from the out-of-vocabulary issue inherent in subword tokenizers and their embeddings.<n>We propose a pixel-based generative language model that replaces the text-based embeddings with pixel-based representations by rendering words as individual images.
  arXiv  Detail & Related papers  (2025-08-28T20:48:38Z)
• A Case Against Implicit Standards: Homophone Normalization in Machine Translation for Languages that use the Ge'ez Script [3.5149312379702127]
  Homophone normalization is a pre-processing step applied in Amharic Natural Language Processing literature.<n>We propose a post-inference intervention in which normalization is applied to model predictions instead of training data.<n>Our work contributes to the broader discussion on technology-facilitated language change and calls for more language-aware interventions.
  arXiv  Detail & Related papers  (2025-07-20T22:35:08Z)
• ViLa-MIL: Dual-scale Vision-Language Multiple Instance Learning for Whole Slide Image Classification [52.405499816861635]
  Multiple instance learning (MIL)-based framework has become the mainstream for processing the whole slide image (WSI)<n>We propose a dual-scale vision-language multiple instance learning (ViLa-MIL) framework for whole slide image classification.
  arXiv  Detail & Related papers  (2025-02-12T13:28:46Z)
• Pixology: Probing the Linguistic and Visual Capabilities of Pixel-based Language Models [7.356870418870544]
  Pixel-based language models have emerged as a compelling alternative to subword-based language modelling. PIXEL is a vision transformer that has been pre-trained on rendered text.
  arXiv  Detail & Related papers  (2024-10-15T19:21:23Z)
• To token or not to token: A Comparative Study of Text Representations for Cross-Lingual Transfer [23.777874316083984]
  We propose a scoring Language Quotient metric capable of providing a weighted representation of both zero-shot and few-shot evaluation combined. Our analysis reveals that image-based models excel in cross-lingual transfer when languages are closely related and share visually similar scripts. In dependency parsing tasks where word relationships play a crucial role, models with their character-level focus, outperform others.
  arXiv  Detail & Related papers  (2023-10-12T06:59:10Z)
• ALIP: Adaptive Language-Image Pre-training with Synthetic Caption [78.93535202851278]
  Contrastive Language-Image Pre-training (CLIP) has significantly boosted the performance of various vision-language tasks. The presence of intrinsic noise and unmatched image-text pairs in web data can potentially affect the performance of representation learning. We propose an Adaptive Language-Image Pre-training (ALIP), a bi-path model that integrates supervision from both raw text and synthetic caption.
  arXiv  Detail & Related papers  (2023-08-16T15:19:52Z)
• CLIP-ViP: Adapting Pre-trained Image-Text Model to Video-Language Representation Alignment [146.3128011522151]
  We propose a Omni Crossmodal Learning method equipped with a Video Proxy mechanism on the basis of CLIP, namely CLIP-ViP. Our approach improves the performance of CLIP on video-text retrieval by a large margin. Our model also achieves SOTA results on a variety of datasets, including MSR-VTT, DiDeMo, LSMDC, and ActivityNet.
  arXiv  Detail & Related papers  (2022-09-14T05:47:02Z)
• Language Modelling with Pixels [29.976453396194053]
  This paper introduces PIXEL, the Pixel-based of Language, which suffers from neither of these issues. PIXEL is a pretrained language model that renders text as images, making it possible to transfer representations across languages. We evaluate on syntactic and semantic tasks in typologically diverse languages, including various non-Latin scripts.
  arXiv  Detail & Related papers  (2022-07-14T15:20:36Z)
• Unsupervised Domain Adaptation of a Pretrained Cross-Lingual Language Model [58.27176041092891]
  Recent research indicates that pretraining cross-lingual language models on large-scale unlabeled texts yields significant performance improvements. We propose a novel unsupervised feature decomposition method that can automatically extract domain-specific features from the entangled pretrained cross-lingual representations. Our proposed model leverages mutual information estimation to decompose the representations computed by a cross-lingual model into domain-invariant and domain-specific parts.
  arXiv  Detail & Related papers  (2020-11-23T16:00:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.