Different Tokenization Schemes Lead to Comparable Performance in Spanish Number Agreement

• URL: http://arxiv.org/abs/2403.13754v1
• Date: Wed, 20 Mar 2024 17:01:56 GMT
• Title: Different Tokenization Schemes Lead to Comparable Performance in Spanish Number Agreement
• Authors: Catherine Arnett, Pamela D. Rivière, Tyler A. Chang, Sean Trott,
• Abstract summary: We investigate how different tokenization schemes impact number agreement in Spanish plurals. We find that morphologically-aligned tokenization performs similarly to other tokenization schemes. Our results indicate that morphological tokenization is not strictly required for performance.
• Score: 5.223020867766102
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The relationship between language model tokenization and performance is an open area of research. Here, we investigate how different tokenization schemes impact number agreement in Spanish plurals. We find that morphologically-aligned tokenization performs similarly to other tokenization schemes, even when induced artificially for words that would not be tokenized that way during training. We then present exploratory analyses demonstrating that language model embeddings for different plural tokenizations have similar distributions along the embedding space axis that maximally distinguishes singular and plural nouns. Our results suggest that morphologically-aligned tokenization is a viable tokenization approach, and existing models already generalize some morphological patterns to new items. However, our results indicate that morphological tokenization is not strictly required for performance.

Related papers

• Why do language models perform worse for morphologically complex languages? [0.913127392774573]
  We find new evidence for a performance gap between agglutinative and fusional languages. We propose three possible causes for this performance gap: morphological alignment of tokenizers, tokenization quality, and disparities in dataset sizes and measurement. Results suggest that no language is harder or easier for a language model to learn on the basis of its morphological typology.
  arXiv  Detail & Related papers  (2024-11-21T15:06:51Z)
• Tokenization and Morphology in Multilingual Language Models: A Comparative Analysis of mT5 and ByT5 [4.779196219827507]
  We capture the impact of tokenization by contrasting two multilingual language models: mT5 and ByT5. Probing the morphological knowledge encoded in these models on four tasks and 17 languages, our analyses show that the models learn the morphological systems of some languages better than others.
  arXiv  Detail & Related papers  (2024-10-15T14:14:19Z)
• Tokenization with Factorized Subword Encoding [2.538209532048867]
  We propose a novel tokenization method that factorizes subwords onto discrete triplets using a VQ-VAE model. Results indicate that this method is more appropriate and robust for morphological tasks than the commonly used byte-pair encoding (BPE) tokenization algorithm.
  arXiv  Detail & Related papers  (2023-06-13T13:27:34Z)
• Language Model Tokenizers Introduce Unfairness Between Languages [98.92630681729518]
  We show how disparity in the treatment of different languages arises at the tokenization stage, well before a model is even invoked. Character-level and byte-level models also exhibit over 4 times the difference in the encoding length for some language pairs. We make the case that we should train future language models using multilingually fair subword tokenizers.
  arXiv  Detail & Related papers  (2023-05-17T14:17:57Z)
• VECO 2.0: Cross-lingual Language Model Pre-training with Multi-granularity Contrastive Learning [56.47303426167584]
  We propose a cross-lingual pre-trained model VECO2.0 based on contrastive learning with multi-granularity alignments. Specifically, the sequence-to-sequence alignment is induced to maximize the similarity of the parallel pairs and minimize the non-parallel pairs. token-to-token alignment is integrated to bridge the gap between synonymous tokens excavated via the thesaurus dictionary from the other unpaired tokens in a bilingual instance.
  arXiv  Detail & Related papers  (2023-04-17T12:23:41Z)
• Multilingual Extraction and Categorization of Lexical Collocations with Graph-aware Transformers [86.64972552583941]
  We put forward a sequence tagging BERT-based model enhanced with a graph-aware transformer architecture, which we evaluate on the task of collocation recognition in context. Our results suggest that explicitly encoding syntactic dependencies in the model architecture is helpful, and provide insights on differences in collocation typification in English, Spanish and French.
  arXiv  Detail & Related papers  (2022-05-23T16:47:37Z)
• Impact of Tokenization on Language Models: An Analysis for Turkish [2.4660652494309936]
  We train tokenizers and pretrain medium-sized language models using RoBERTa pretraining procedure on the Turkish split of the OSCAR corpus. Our experiments, supported by statistical tests, reveal that Morphological-level tokenizer has challenging performance with de facto tokenizers. We find that increasing the vocabulary size improves the performance of Morphological and Word-level tokenizers more than that of de facto tokenizers.
  arXiv  Detail & Related papers  (2022-04-19T12:01:46Z)
• Interpreting Language Models with Contrastive Explanations [99.7035899290924]
  Language models must consider various features to predict a token, such as its part of speech, number, tense, or semantics. Existing explanation methods conflate evidence for all these features into a single explanation, which is less interpretable for human understanding. We show that contrastive explanations are quantifiably better than non-contrastive explanations in verifying major grammatical phenomena.
  arXiv  Detail & Related papers  (2022-02-21T18:32:24Z)
• On The Ingredients of an Effective Zero-shot Semantic Parser [95.01623036661468]
  We analyze zero-shot learning by paraphrasing training examples of canonical utterances and programs from a grammar. We propose bridging these gaps using improved grammars, stronger paraphrasers, and efficient learning methods. Our model achieves strong performance on two semantic parsing benchmarks (Scholar, Geo) with zero labeled data.
  arXiv  Detail & Related papers  (2021-10-15T21:41:16Z)
• More Than Words: Collocation Tokenization for Latent Dirichlet Allocation Models [71.42030830910227]
  We propose a new metric for measuring the clustering quality in settings where the models differ. We show that topics trained with merged tokens result in topic keys that are clearer, more coherent, and more effective at distinguishing topics than those unmerged models.
  arXiv  Detail & Related papers  (2021-08-24T14:08:19Z)
• Recurrent Neural Network Language Models Always Learn English-Like Relative Clause Attachment [17.995905582226463]
  We compare model performance in English and Spanish to show that non-linguistic biases in RNN LMs advantageously overlap with syntactic structure in English but not Spanish. English models may appear to acquire human-like syntactic preferences, while models trained on Spanish fail to acquire comparable human-like preferences.
  arXiv  Detail & Related papers  (2020-05-01T01:21:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.