Confounding Factors in Relating Model Performance to Morphology

• URL: http://arxiv.org/abs/2511.01380v1
• Date: Mon, 03 Nov 2025 09:26:38 GMT
• Title: Confounding Factors in Relating Model Performance to Morphology
• Abstract summary: We identify confounding factors in analyses trying to answer the question of whether, and how, morphology relates to language modeling.<n>We introduce token bigram metrics as an intrinsic way to predict the difficulty of causal language modeling.
• Score: 8.325450847248467
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The extent to which individual language characteristics influence tokenization and language modeling is an open question. Differences in morphological systems have been suggested as both unimportant and crucial to consider (Cotterell et al., 2018; Gerz et al., 2018a; Park et al., 2021, inter alia). We argue this conflicting evidence is due to confounding factors in experimental setups, making it hard to compare results and draw conclusions. We identify confounding factors in analyses trying to answer the question of whether, and how, morphology relates to language modeling. Next, we re-assess three hypotheses by Arnett & Bergen (2025) for why modeling agglutinative languages results in higher perplexities than fusional languages: they look at morphological alignment of tokenization, tokenization efficiency, and dataset size. We show that each conclusion includes confounding factors. Finally, we introduce token bigram metrics as an intrinsic way to predict the difficulty of causal language modeling, and find that they are gradient proxies for morphological complexity that do not require expert annotation. Ultimately, we outline necessities to reliably answer whether, and how, morphology relates to language modeling.

Related papers

• Surprisal Theory is Tautological (without Rational Grounding) [60.917028769172795]
  Surprisal theory holds that the human processing difficulty of a linguistic unit in context is an affine function of its surprisal under some language model.<n>I argue this claim is a tautology without further constraint.<n>I conclude that breaking the tautology requires a rationalist intervention.
  arXiv  Detail & Related papers  (2026-07-23T17:54:37Z)
• On the Interplay between Positional Encodings, Morphological Complexity, and Word Order Flexibility [5.521655731616328]
  Positional encodings are a direct target to investigate the implications of the trade-off hypothesis.<n>Contrary to previous findings, we do not observe a clear interaction between position encodings and morphological complexity or word order flexibility.<n>Our results show that the choice of tasks, languages, and metrics are essential for drawing stable conclusions.
  arXiv  Detail & Related papers  (2025-11-11T11:50:21Z)
• Why do language models perform worse for morphologically complex languages? [0.913127392774573]
  We find new evidence for a performance gap between agglutinative and fusional languages. We propose three possible causes for this performance gap: morphological alignment of tokenizers, tokenization quality, and disparities in dataset sizes and measurement. Results suggest that no language is harder or easier for a language model to learn on the basis of its morphological typology.
  arXiv  Detail & Related papers  (2024-11-21T15:06:51Z)
• Holmes: A Benchmark to Assess the Linguistic Competence of Language Models [59.627729608055006]
  We introduce Holmes, a new benchmark designed to assess language models (LMs) linguistic competence. We use computation-based probing to examine LMs' internal representations regarding distinct linguistic phenomena. As a result, we meet recent calls to disentangle LMs' linguistic competence from other cognitive abilities.
  arXiv  Detail & Related papers  (2024-04-29T17:58:36Z)
• Cross-Lingual Transfer of Cognitive Processing Complexity [11.939409227407769]
  We use sentence-level eye-tracking patterns as a cognitive indicator for structural complexity. We show that the multilingual model XLM-RoBERTa can successfully predict varied patterns for 13 typologically diverse languages.
  arXiv  Detail & Related papers  (2023-02-24T15:48:23Z)
• Transparency Helps Reveal When Language Models Learn Meaning [71.96920839263457]
  Our systematic experiments with synthetic data reveal that, with languages where all expressions have context-independent denotations, both autoregressive and masked language models learn to emulate semantic relations between expressions. Turning to natural language, our experiments with a specific phenomenon -- referential opacity -- add to the growing body of evidence that current language models do not well-represent natural language semantics.
  arXiv  Detail & Related papers  (2022-10-14T02:35:19Z)
• Naturalistic Causal Probing for Morpho-Syntax [76.83735391276547]
  We suggest a naturalistic strategy for input-level intervention on real world data in Spanish. Using our approach, we isolate morpho-syntactic features from counfounders in sentences. We apply this methodology to analyze causal effects of gender and number on contextualized representations extracted from pre-trained models.
  arXiv  Detail & Related papers  (2022-05-14T11:47:58Z)
• Quantifying Synthesis and Fusion and their Impact on Machine Translation [79.61874492642691]
  In Natural Language Processing (NLP) typically labels a whole language with a strict type of morphology, e.g. fusional or agglutinative. In this work, we propose to reduce the rigidity of such claims, by quantifying morphological typology at the word and segment level. For computing literature, we test unsupervised and supervised morphological segmentation methods for English, German and Turkish, whereas for fusion, we propose a semi-automatic method using Spanish as a case study. Then, we analyse the relationship between machine translation quality and the degree of synthesis and fusion at word (nouns and verbs for English-Turkish,
  arXiv  Detail & Related papers  (2022-05-06T17:04:58Z)
• Morphology Without Borders: Clause-Level Morphological Annotation [8.559428282730021]
  We propose to view morphology as a clause-level phenomenon, rather than word-level. We deliver a novel dataset for clause-level morphology covering 4 typologically-different languages: English, German, Turkish and Hebrew. Our experiments show that the clause-level tasks are substantially harder than the respective word-level tasks, while having comparable complexity across languages.
  arXiv  Detail & Related papers  (2022-02-25T17:20:28Z)
• Interpreting Language Models with Contrastive Explanations [99.7035899290924]
  Language models must consider various features to predict a token, such as its part of speech, number, tense, or semantics. Existing explanation methods conflate evidence for all these features into a single explanation, which is less interpretable for human understanding. We show that contrastive explanations are quantifiably better than non-contrastive explanations in verifying major grammatical phenomena.
  arXiv  Detail & Related papers  (2022-02-21T18:32:24Z)
• Morphology Matters: A Multilingual Language Modeling Analysis [8.791030561752384]
  Prior studies disagree on whether inflectional morphology makes languages harder to model. We compile a larger corpus of 145 Bible translations in 92 languages and a larger number of typological features. Several morphological measures are significantly associated with higher surprisal when LSTM models are trained with BPE-segmented data.
  arXiv  Detail & Related papers  (2020-12-11T11:55:55Z)
• A Simple Joint Model for Improved Contextual Neural Lemmatization [60.802451210656805]
  We present a simple joint neural model for lemmatization and morphological tagging that achieves state-of-the-art results on 20 languages. Our paper describes the model in addition to training and decoding procedures.
  arXiv  Detail & Related papers  (2019-04-04T02:03:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.