Do Large Language Models Grasp The Grammar? Evidence from Grammar-Book-Guided Probing in Luxembourgish

• URL: http://arxiv.org/abs/2510.24856v1
• Date: Tue, 28 Oct 2025 18:02:51 GMT
• Title: Do Large Language Models Grasp The Grammar? Evidence from Grammar-Book-Guided Probing in Luxembourgish
• Authors: Lujun Li, Yewei Song, Lama Sleem, Yiqun Wang, Yangjie Xu, Cedric Lothritz, Niccolo Gentile, Radu State, Tegawende F. Bissyande, Jacques Klein,
• Abstract summary: Grammar refers to the system of rules that governs the structural organization and the semantic relations among linguistic units.<n>In natural language processing, there remains a notable scarcity of grammar focused evaluation protocols.<n>We propose a Grammar Book Guided evaluation pipeline to provide a systematic and generalizable framework for grammar evaluation.
• Score: 17.24445071401393
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Grammar refers to the system of rules that governs the structural organization and the semantic relations among linguistic units such as sentences, phrases, and words within a given language. In natural language processing, there remains a notable scarcity of grammar focused evaluation protocols, a gap that is even more pronounced for low-resource languages. Moreover, the extent to which large language models genuinely comprehend grammatical structure, especially the mapping between syntactic structures and meanings, remains under debate. To investigate this issue, we propose a Grammar Book Guided evaluation pipeline intended to provide a systematic and generalizable framework for grammar evaluation consisting of four key stages, and in this work we take Luxembourgish as a case study. The results show a weak positive correlation between translation performance and grammatical understanding, indicating that strong translations do not necessarily imply deep grammatical competence. Larger models perform well overall due to their semantic strength but remain weak in morphology and syntax, struggling particularly with Minimal Pair tasks, while strong reasoning ability offers a promising way to enhance their grammatical understanding.

Related papers

• Grammaticality Judgments in Humans and Language Models: Revisiting Generative Grammar with LLMs [0.0]
  In traditional generative grammar, systematic contrasts in grammaticality such as subject-auxiliary inversion and the licensing of parasitic gaps are taken as evidence for an internal, hierarchical grammar.<n>We test whether large language models (LLMs), trained only on surface forms, reproduce these contrasts in ways that imply an underlying structural representation.
  arXiv  Detail & Related papers  (2025-12-11T09:17:35Z)
• Principles of semantic and functional efficiency in grammatical patterning [1.6267479602370545]
  We show that grammatical organization provably inherits from perceptual attributes.<n>Our measurements on a diverse language sample show that grammars prioritize functional goals.
  arXiv  Detail & Related papers  (2024-10-21T10:49:54Z)
• Sparse Logistic Regression with High-order Features for Automatic Grammar Rule Extraction from Treebanks [6.390468088226495]
  We propose a new method to extract and explore significant fine-grained grammar patterns from treebanks. We extract descriptions and rules across different languages for two linguistic phenomena, agreement and word order. Our method captures both well-known and less well-known significant grammar rules in Spanish, French, and Wolof.
  arXiv  Detail & Related papers  (2024-03-26T09:39:53Z)
• CLSE: Corpus of Linguistically Significant Entities [58.29901964387952]
  We release a Corpus of Linguistically Significant Entities (CLSE) annotated by experts. CLSE covers 74 different semantic types to support various applications from airline ticketing to video games. We create a linguistically representative NLG evaluation benchmark in three languages: French, Marathi, and Russian.
  arXiv  Detail & Related papers  (2022-11-04T12:56:12Z)
• Testing the Ability of Language Models to Interpret Figurative Language [69.59943454934799]
  Figurative and metaphorical language are commonplace in discourse. It remains an open question to what extent modern language models can interpret nonliteral phrases. We introduce Fig-QA, a Winograd-style nonliteral language understanding task.
  arXiv  Detail & Related papers  (2022-04-26T23:42:22Z)
• Controlled Evaluation of Grammatical Knowledge in Mandarin Chinese Language Models [22.57309958548928]
  We investigate whether structural supervision improves language models' ability to learn grammatical dependencies in typologically different languages. We train LSTMs, Recurrent Neural Network Grammars, Transformer language models, and generative parsing models on datasets of different sizes. We find suggestive evidence that structural supervision helps with representing syntactic state across intervening content and improves performance in low-data settings.
  arXiv  Detail & Related papers  (2021-09-22T22:11:30Z)
• Dependency Induction Through the Lens of Visual Perception [81.91502968815746]
  We propose an unsupervised grammar induction model that leverages word concreteness and a structural vision-based to jointly learn constituency-structure and dependency-structure grammars. Our experiments show that the proposed extension outperforms the current state-of-the-art visually grounded models in constituency parsing even with a smaller grammar size.
  arXiv  Detail & Related papers  (2021-09-20T18:40:37Z)
• VLGrammar: Grounded Grammar Induction of Vision and Language [86.88273769411428]
  We study grounded grammar induction of vision and language in a joint learning framework. We present VLGrammar, a method that uses compound probabilistic context-free grammars (compound PCFGs) to induce the language grammar and the image grammar simultaneously.
  arXiv  Detail & Related papers  (2021-03-24T04:05:08Z)
• ERICA: Improving Entity and Relation Understanding for Pre-trained Language Models via Contrastive Learning [97.10875695679499]
  We propose a novel contrastive learning framework named ERICA in pre-training phase to obtain a deeper understanding of the entities and their relations in text. Experimental results demonstrate that our proposed ERICA framework achieves consistent improvements on several document-level language understanding tasks.
  arXiv  Detail & Related papers  (2020-12-30T03:35:22Z)
• A Benchmark for Systematic Generalization in Grounded Language Understanding [61.432407738682635]
  Humans easily interpret expressions that describe unfamiliar situations composed from familiar parts. Modern neural networks, by contrast, struggle to interpret novel compositions. We introduce a new benchmark, gSCAN, for evaluating compositional generalization in situated language understanding.
  arXiv  Detail & Related papers  (2020-03-11T08:40:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.