Probing Character-level Transformers for the Spanish L-shaped Morphome

• URL: http://arxiv.org/abs/2608.03452v1
• Date: Tue, 04 Aug 2026 10:48:17 GMT
• Title: Probing Character-level Transformers for the Spanish L-shaped Morphome
• Abstract summary: Our test case is the Spanish emphL-shaped morphome, a complex irregular pattern in which the verb's stem alternates in exactly the first-person singular indicative and all subjunctive forms.<n>Prior studies have shown that character-level transformers can reproduce this pattern, but that evidence describes what models produce, not what they represent.<n>We show that the models encode the L-shaped class itself, not just its visible alternations.
• Score: 7.353970093374431
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: When a transformer learns an irregular morphological pattern, what has it learned? Our test case is the Spanish \emph{L-shaped morphome}, a complex irregular pattern in which the verb's stem alternates in exactly the first-person singular indicative and all subjunctive forms, and whose membership no phonological, semantic, or syntactic feature predicts. Prior studies have shown that character-level transformers can reproduce this pattern, but that evidence describes what models produce, not what they represent. Probing five architectures, twelve trained models each, under lemma-disjoint cross-validation with controls and surface baselines, we show that the models encode the L-shaped class itself, not just its visible alternations. It is decodable above every surface baseline, survives instances in which every form shows the same stem, and probes trained on alternating instances still classify non-alternating ones. The encoding is localized where the stem choice is made, at the stem-final consonant position of the middle decoder, before the alternant is read. And it is item-specific: which verbs a model learned matters far more than which architecture it is. The models store the morphome as an item-specific lexical abstraction, sufficient to reproduce the pattern but not to generalize it as humans do.

Related papers

• Transducing Language Models [52.080921891265255]
  We introduce a framework for language models derived from deterministic string-to-string transformations.<n>We develop algorithms that compose a language model with an FST to *marginalize* over source strings mapping to a given target.<n>We present an exact algorithm, an efficient approximation, and a theoretical analysis.
  arXiv  Detail & Related papers  (2026-03-05T14:04:08Z)
• A theoretical model of dynamical grammatical gender shifting based on set-valued set function [0.0]
  This study investigates the diverse characteristics of nouns, focusing on both semantic (e.g., countable/uncountable) and morphosyntactic (e.g., masculine/feminine) distinctions.
  arXiv  Detail & Related papers  (2026-03-03T20:32:13Z)
• Character-aware Transformers Learn an Irregular Morphological Pattern Yet None Generalize Like Humans [8.033684021402165]
  We show that encoder-decoder models can acquire irregular patterns, but evidence that they generalize these patterns like humans is mixed.<n>We investigate this using the Spanish emphL-shaped morphome, where only the first-person singular indicative shares its stem with all subjunctive forms.<n>None of the models reproduce the human pattern, highlighting the gap between statistical pattern reproduction and morphological abstraction.
  arXiv  Detail & Related papers  (2026-02-15T11:22:12Z)
• Echoes of BERT: Do Modern Language Models Rediscover the Classical NLP Pipeline? [4.991808275998526]
  Building on classic BERTology work, we analyze 25 models spanning from classical architectures (BERT, DeBERTa, GPT-2) to modern large language models (Pythia, OLMo-2, Gemma-2, Qwen2.5, Llama-3.1)<n>We find that hierarchical organization persists in modern models, with early layers capture syntax, middle layers handle semantics and entity-level information, and later layers encode discourse phenomena.<n>We find that lexical information concentrates linearly in early layers but becomes increasingly nonlinear deeper in the network, while inflectional information remains linearly accessible throughout all layers.
  arXiv  Detail & Related papers  (2025-06-02T18:01:56Z)
• Frequency matters: Modeling irregular morphological patterns in Spanish with Transformers [0.8602553195689513]
  We focus on Spanish verbal paradigms, where certain verbs follow an irregular L-shaped pattern.<n>We investigate the role of input frequency in the acquisition of regular versus irregular L-shaped patterns in transformer models.
  arXiv  Detail & Related papers  (2024-10-28T13:36:46Z)
• AWOL: Analysis WithOut synthesis using Language [57.31874938870305]
  We leverage language to control existing 3D shape models to produce novel shapes. We show that we can use text to generate new animals not present during training. This work also constitutes the first language-driven method for generating 3D trees.
  arXiv  Detail & Related papers  (2024-04-03T20:04:44Z)
• Probabilistic Transformer: A Probabilistic Dependency Model for Contextual Word Representation [52.270712965271656]
  We propose a new model of contextual word representation, not from a neural perspective, but from a purely syntactic and probabilistic perspective. We find that the graph of our model resembles transformers, with correspondences between dependencies and self-attention. Experiments show that our model performs competitively to transformers on small to medium sized datasets.
  arXiv  Detail & Related papers  (2023-11-26T06:56:02Z)
• Lexinvariant Language Models [84.2829117441298]
  Token embeddings, a mapping from discrete lexical symbols to continuous vectors, are at the heart of any language model (LM) We study textitlexinvariantlanguage models that are invariant to lexical symbols and therefore do not need fixed token embeddings in practice. We show that a lexinvariant LM can attain perplexity comparable to that of a standard language model, given a sufficiently long context.
  arXiv  Detail & Related papers  (2023-05-24T19:10:46Z)
• Mapping of attention mechanisms to a generalized Potts model [50.91742043564049]
  We show that training a neural network is exactly equivalent to solving the inverse Potts problem by the so-called pseudo-likelihood method. We also compute the generalization error of self-attention in a model scenario analytically using the replica method.
  arXiv  Detail & Related papers  (2023-04-14T16:32:56Z)
• How do we get there? Evaluating transformer neural networks as cognitive models for English past tense inflection [0.0]
  We train a set of transformer models with different settings to examine their behavior on this task. The models' performance on the regulars is heavily affected by type frequency and ratio but not token frequency and ratio, and vice versa for the irregulars. Although the transformer model exhibits some level of learning on the abstract category of verb regularity, its performance does not fit human data well.
  arXiv  Detail & Related papers  (2022-10-17T15:13:35Z)
• Unnatural Language Inference [48.45003475966808]
  We find that state-of-the-art NLI models, such as RoBERTa and BART, are invariant to, and sometimes even perform better on, examples with randomly reordered words. Our findings call into question the idea that our natural language understanding models, and the tasks used for measuring their progress, genuinely require a human-like understanding of syntax.
  arXiv  Detail & Related papers  (2020-12-30T20:40:48Z)
• Linguistically inspired morphological inflection with a sequence to sequence model [19.892441884896893]
  Our research question is whether a neural network would be capable of learning inflectional morphemes for inflection production. We are using an inflectional corpus and a single layer seq2seq model to test this hypothesis. Our character-morpheme-based model creates inflection by predicting the stem character-to-character and the inflectional affixes as character blocks.
  arXiv  Detail & Related papers  (2020-09-04T08:58:42Z)
• A Simple Joint Model for Improved Contextual Neural Lemmatization [60.802451210656805]
  We present a simple joint neural model for lemmatization and morphological tagging that achieves state-of-the-art results on 20 languages. Our paper describes the model in addition to training and decoding procedures.
  arXiv  Detail & Related papers  (2019-04-04T02:03:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.