MIXAR: Scaling Autoregressive Pixel-based Language Models to Multiple Languages and Scripts

• URL: http://arxiv.org/abs/2604.11575v1
• Date: Mon, 13 Apr 2026 14:53:19 GMT
• Title: MIXAR: Scaling Autoregressive Pixel-based Language Models to Multiple Languages and Scripts
• Abstract summary: MIXAR is the first generative pixel-based language model trained on eight different languages.<n>We show how MIXAR is robust to languages never seen during the training.
• Score: 66.21710415920332
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Pixel-based language models are gaining momentum as alternatives to traditional token-based approaches, promising to circumvent tokenization challenges. However, the inherent perceptual diversity across languages poses a significant hurdle for multilingual generalization in pixel space. This paper introduces MIXAR, the first generative pixel-based language model trained on eight different languages utilizing a range of different scripts. We empirically evaluate MIXAR against previous pixel-based models as well as comparable tokenizer-based models, demonstrating substantial performance improvement on discriminative and generative multilingual tasks. Additionally, we show how MIXAR is robust to languages never seen during the training. These results are further strengthened when scaling the model to 0.5B parameters which not only improves its capabilities in generative tasks like LAMBADA but also its robustness when challenged with input perturbations such as orthographic attacks.

Related papers

• Seeing the Unseen: Visual Similarity for Pixel Language Model Adaptation [7.634308270954448]
  Pixel-based language models (LMs) replace traditional tokenizers by processing rendered images of text.<n>We analyze how continued pre-training of pixel-based LMs is influenced by data scale, initial script exposure, and cross-lingual transfer from languages written in other Brahmic scripts.
  arXiv  Detail & Related papers  (2026-08-31T10:11:05Z)
• Enhancing Robustness of Autoregressive Language Models against Orthographic Attacks via Pixel-based Approach [51.95266411355865]
  Autoregressive language models are vulnerable to orthographic attacks.<n>This vulnerability stems from the out-of-vocabulary issue inherent in subword tokenizers and their embeddings.<n>We propose a pixel-based generative language model that replaces the text-based embeddings with pixel-based representations by rendering words as individual images.
  arXiv  Detail & Related papers  (2025-08-28T20:48:38Z)
• Overcoming Vocabulary Constraints with Pixel-level Fallback [9.753745943931207]
  Subword tokenization requires balancing computational efficiency and vocabulary coverage.<n>We propose a vocabulary-free encoder that generates input embeddings from text rendered as pixels.
  arXiv  Detail & Related papers  (2025-04-02T20:50:31Z)
• Evaluating Pixel Language Models on Non-Standardized Languages [24.94386050975835]
  pixel-based models convert text into images that are divided into patches, enabling a continuous vocabulary representation.<n>Using German as a case study, we compare the performance of pixel-based models to token-based models across various syntactic and semantic tasks.
  arXiv  Detail & Related papers  (2024-12-12T09:11:45Z)
• Multilingual Conceptual Coverage in Text-to-Image Models [98.80343331645626]
  "Conceptual Coverage Across Languages" (CoCo-CroLa) is a technique for benchmarking the degree to which any generative text-to-image system provides multilingual parity to its training language in terms of tangible nouns. For each model we can assess "conceptual coverage" of a given target language relative to a source language by comparing the population of images generated for a series of tangible nouns in the source language to the population of images generated for each noun under translation in the target language.
  arXiv  Detail & Related papers  (2023-06-02T17:59:09Z)
• Extrapolating Multilingual Understanding Models as Multilingual Generators [82.1355802012414]
  This paper explores methods to empower multilingual understanding models the generation abilities to get a unified model. We propose a textbfSemantic-textbfGuided textbfAlignment-then-Denoising (SGA) approach to adapt an encoder to a multilingual generator with a small number of new parameters.
  arXiv  Detail & Related papers  (2023-05-22T15:33:21Z)
• Beyond Contrastive Learning: A Variational Generative Model for Multilingual Retrieval [109.62363167257664]
  We propose a generative model for learning multilingual text embeddings. Our model operates on parallel data in $N$ languages. We evaluate this method on a suite of tasks including semantic similarity, bitext mining, and cross-lingual question retrieval.
  arXiv  Detail & Related papers  (2022-12-21T02:41:40Z)
• Quark: Controllable Text Generation with Reinforced Unlearning [68.07749519374089]
  Large-scale language models often learn behaviors that are misaligned with user expectations. We introduce Quantized Reward Konditioning (Quark), an algorithm for optimizing a reward function that quantifies an (un)wanted property. For unlearning toxicity, negative sentiment, and repetition, our experiments show that Quark outperforms both strong baselines and state-of-the-art reinforcement learning methods.
  arXiv  Detail & Related papers  (2022-05-26T21:11:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.