Evaluating Pixel Language Models on Non-Standardized Languages

• URL: http://arxiv.org/abs/2412.09084v1
• Date: Thu, 12 Dec 2024 09:11:45 GMT
• Title: Evaluating Pixel Language Models on Non-Standardized Languages
• Authors: Alberto Muñoz-Ortiz, Verena Blaschke, Barbara Plank,
• Abstract summary: pixel-based models convert text into images that are divided into patches, enabling a continuous vocabulary representation. Using German as a case study, we compare the performance of pixel-based models to token-based models across various syntactic and semantic tasks.
• Score: 24.94386050975835
• License:
• Abstract: We explore the potential of pixel-based models for transfer learning from standard languages to dialects. These models convert text into images that are divided into patches, enabling a continuous vocabulary representation that proves especially useful for out-of-vocabulary words common in dialectal data. Using German as a case study, we compare the performance of pixel-based models to token-based models across various syntactic and semantic tasks. Our results show that pixel-based models outperform token-based models in part-of-speech tagging, dependency parsing and intent detection for zero-shot dialect evaluation by up to 26 percentage points in some scenarios, though not in Standard German. However, pixel-based models fall short in topic classification. These findings emphasize the potential of pixel-based models for handling dialectal data, though further research should be conducted to assess their effectiveness in various linguistic contexts.

Related papers

• Pixology: Probing the Linguistic and Visual Capabilities of Pixel-based Language Models [7.356870418870544]
  Pixel-based language models have emerged as a compelling alternative to subword-based language modelling. PIXEL is a vision transformer that has been pre-trained on rendered text.
  arXiv  Detail & Related papers  (2024-10-15T19:21:23Z)
• Reading Is Believing: Revisiting Language Bottleneck Models for Image Classification [4.1205832766381985]
  We revisit language bottleneck models as an approach to ensuring the explainability of deep learning models for image classification. We experimentally show that a language bottleneck model that combines a modern image captioner with a pre-trained language model can achieve image classification accuracy that exceeds that of black-box models.
  arXiv  Detail & Related papers  (2024-06-22T10:49:34Z)
• Modeling Orthographic Variation in Occitan's Dialects [3.038642416291856]
  Large multilingual models minimize the need for spelling normalization during pre-processing. Our findings suggest that large multilingual models minimize the need for spelling normalization during pre-processing.
  arXiv  Detail & Related papers  (2024-04-30T07:33:51Z)
• Pixel Aligned Language Models [94.32841818609914]
  We develop a vision-language model that can take locations as either inputs or outputs. When taking locations as inputs, the model performs location-conditioned captioning, which generates captions for the indicated object or region. Our model is pre-trained on the Localized Narrative dataset, which contains pixel-word-aligned captioning from human attention.
  arXiv  Detail & Related papers  (2023-12-14T18:57:58Z)
• Multilingual Conceptual Coverage in Text-to-Image Models [98.80343331645626]
  "Conceptual Coverage Across Languages" (CoCo-CroLa) is a technique for benchmarking the degree to which any generative text-to-image system provides multilingual parity to its training language in terms of tangible nouns. For each model we can assess "conceptual coverage" of a given target language relative to a source language by comparing the population of images generated for a series of tangible nouns in the source language to the population of images generated for each noun under translation in the target language.
  arXiv  Detail & Related papers  (2023-06-02T17:59:09Z)
• On The Ingredients of an Effective Zero-shot Semantic Parser [95.01623036661468]
  We analyze zero-shot learning by paraphrasing training examples of canonical utterances and programs from a grammar. We propose bridging these gaps using improved grammars, stronger paraphrasers, and efficient learning methods. Our model achieves strong performance on two semantic parsing benchmarks (Scholar, Geo) with zero labeled data.
  arXiv  Detail & Related papers  (2021-10-15T21:41:16Z)
• Language Models are Few-shot Multilingual Learners [66.11011385895195]
  We evaluate the multilingual skills of the GPT and T5 models in conducting multi-class classification on non-English languages. We show that, given a few English examples as context, pre-trained language models can predict not only English test samples but also non-English ones.
  arXiv  Detail & Related papers  (2021-09-16T03:08:22Z)
• Comparison of Interactive Knowledge Base Spelling Correction Models for Low-Resource Languages [81.90356787324481]
  Spelling normalization for low resource languages is a challenging task because the patterns are hard to predict. This work shows a comparison of a neural model and character language models with varying amounts on target language data. Our usage scenario is interactive correction with nearly zero amounts of training examples, improving models as more data is collected.
  arXiv  Detail & Related papers  (2020-10-20T17:31:07Z)
• Are Some Words Worth More than Others? [3.5598388686985354]
  We propose two new intrinsic evaluation measures within the framework of a simple word prediction task. We evaluate several commonly-used large English language models using our proposed metrics.
  arXiv  Detail & Related papers  (2020-10-12T23:12:11Z)
• Grounded Compositional Outputs for Adaptive Language Modeling [59.02706635250856]
  A language model's vocabulary$-$typically selected before training and permanently fixed later$-$affects its size. We propose a fully compositional output embedding layer for language models. To our knowledge, the result is the first word-level language model with a size that does not depend on the training vocabulary.
  arXiv  Detail & Related papers  (2020-09-24T07:21:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.