Related papers: Evaluating the cognitive reality of Spanish irregular morphomic patterns: Humans vs. Transformers

Evaluating the cognitive reality of Spanish irregular morphomic patterns: Humans vs. Transformers

URL: http://arxiv.org/abs/2507.21556v1
Date: Tue, 29 Jul 2025 07:40:32 GMT
Title: Evaluating the cognitive reality of Spanish irregular morphomic patterns: Humans vs. Transformers
Authors: Akhilesh Kakolu Ramarao, Kevin Tang, Dinah Baer-Henney,
Abstract summary: This study investigates the cognitive plausibility of the Spanish irregular morphomic pattern.<n>Using the same analytical framework as the original human study, we evaluate whether transformer models can replicate human-like sensitivity to the morphome.
Score: 0.8602553195689513
License: http://creativecommons.org/licenses/by-sa/4.0/
Abstract: This study investigates the cognitive plausibility of the Spanish irregular morphomic pattern by directly comparing transformer-based neural networks to human behavioral data from \citet{Nevins2015TheRA}. Using the same analytical framework as the original human study, we evaluate whether transformer models can replicate human-like sensitivity to a complex linguistic phenomena, the morphome, under controlled input conditions. Our experiments focus on three frequency conditions: natural, low-frequency, and high-frequency distributions of verbs exhibiting irregular morphomic patterns. While the models outperformed humans in stem and suffix accuracy, a clear divergence emerged in response preferences. Unlike humans, who consistently favored natural responses across all test items, models' preferred irregular responses and were influenced by the proportion of irregular verbs in their training data. Additionally, models trained on the natural and low-frequency distributions, but not the high-frequency distribution, were sensitive to the phonological similarity between test items and real Spanish L-shaped verbs.

Related papers

Frequency matters: Modeling irregular morphological patterns in Spanish with Transformers [0.8602553195689513]
We focus on Spanish verbal paradigms, where certain verbs follow an irregular L-shaped pattern.<n>We investigate the role of input frequency in the acquisition of regular versus irregular L-shaped patterns in transformer models.
arXiv Detail & Related papers (2024-10-28T13:36:46Z)
Agentivit\`a e telicit\`a in GilBERTo: implicazioni cognitive [77.71680953280436]
The goal of this study is to investigate whether a Transformer-based neural language model infers lexical semantics. The semantic properties considered are telicity (also combined with definiteness) and agentivity.
arXiv Detail & Related papers (2023-07-06T10:52:22Z)
How do we get there? Evaluating transformer neural networks as cognitive models for English past tense inflection [0.0]
We train a set of transformer models with different settings to examine their behavior on this task. The models' performance on the regulars is heavily affected by type frequency and ratio but not token frequency and ratio, and vice versa for the irregulars. Although the transformer model exhibits some level of learning on the abstract category of verb regularity, its performance does not fit human data well.
arXiv Detail & Related papers (2022-10-17T15:13:35Z)
Naturalistic Causal Probing for Morpho-Syntax [76.83735391276547]
We suggest a naturalistic strategy for input-level intervention on real world data in Spanish. Using our approach, we isolate morpho-syntactic features from counfounders in sentences. We apply this methodology to analyze causal effects of gender and number on contextualized representations extracted from pre-trained models.
arXiv Detail & Related papers (2022-05-14T11:47:58Z)
Self-Normalized Importance Sampling for Neural Language Modeling [97.96857871187052]
In this work, we propose self-normalized importance sampling. Compared to our previous work, the criteria considered in this work are self-normalized and there is no need to further conduct a correction step. We show that our proposed self-normalized importance sampling is competitive in both research-oriented and production-oriented automatic speech recognition tasks.
arXiv Detail & Related papers (2021-11-11T16:57:53Z)
A Targeted Assessment of Incremental Processing in Neural LanguageModels and Humans [2.7624021966289605]
We present a scaled-up comparison of incremental processing in humans and neural language models. Data comes from a novel online experimental paradigm called the Interpolated Maze task. We find that both humans and language models show increased processing difficulty in ungrammatical sentence regions.
arXiv Detail & Related papers (2021-06-06T20:04:39Z)
Empowering Language Understanding with Counterfactual Reasoning [141.48592718583245]
We propose a Counterfactual Reasoning Model, which mimics the counterfactual thinking by learning from few counterfactual samples. In particular, we devise a generation module to generate representative counterfactual samples for each factual sample, and a retrospective module to retrospect the model prediction by comparing the counterfactual and factual samples.
arXiv Detail & Related papers (2021-06-06T06:36:52Z)
Unnatural Language Inference [48.45003475966808]
We find that state-of-the-art NLI models, such as RoBERTa and BART, are invariant to, and sometimes even perform better on, examples with randomly reordered words. Our findings call into question the idea that our natural language understanding models, and the tasks used for measuring their progress, genuinely require a human-like understanding of syntax.
arXiv Detail & Related papers (2020-12-30T20:40:48Z)
Mechanisms for Handling Nested Dependencies in Neural-Network Language Models and Humans [75.15855405318855]
We studied whether a modern artificial neural network trained with "deep learning" methods mimics a central aspect of human sentence processing. Although the network was solely trained to predict the next word in a large corpus, analysis showed the emergence of specialized units that successfully handled local and long-distance syntactic agreement. We tested the model's predictions in a behavioral experiment where humans detected violations in number agreement in sentences with systematic variations in the singular/plural status of multiple nouns.
arXiv Detail & Related papers (2020-06-19T12:00:05Z)
Do Neural Models Learn Systematicity of Monotonicity Inference in Natural Language? [41.649440404203595]
We introduce a method for evaluating whether neural models can learn systematicity of monotonicity inference in natural language. We consider four aspects of monotonicity inferences and test whether the models can systematically interpret lexical and logical phenomena on different training/test splits.
arXiv Detail & Related papers (2020-04-30T14:48:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.