How Useful are LLMs for Grammar Engineering? Cantonese ParGram Resources and Controlled Experimental Evaluation with English Baselines

• URL: http://arxiv.org/abs/2608.23448v1
• Date: Mon, 24 Aug 2026 16:21:32 GMT
• Title: How Useful are LLMs for Grammar Engineering? Cantonese ParGram Resources and Controlled Experimental Evaluation with English Baselines
• Abstract summary: This paper presents new Cantonese ParGram resources and evaluates LLMs for knowledge-driven grammar engineering.<n>We investigate whether OpenAI's.<n>gpt-oss-120b and GPT-5.4 can generate machine-processable grammars.<n>from sentences and target formal structures under.<n>systematically varied prompting conditions.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper presents new Cantonese ParGram resources and evaluates LLMs for knowledge-driven grammar engineering within a controlled experimental paradigm. Using Cantonese ParGram resources as gold standards, with corresponding English baselines, we investigate whether OpenAI's gpt-oss-120b and GPT-5.4 can generate machine-processable grammars from sentences and target formal structures under systematically varied prompting conditions. GPT-5.4 outperformed gpt-oss-120b, while grammars generated from target formal structures generally outperformed those generated from sentences. Although both models could generate locally plausible phrase-structure rules, lexical entries, and templates, they often struggled to coordinate interacting formal constraints, especially in multi-construction settings. The results characterize both the capabilities and limitations of current LLMs for potential integration into AI-assisted expert workflows: LLMs may support intermediate stages of grammar development, but human linguistic expertise remains central to analysis, validation, and refinement. The study also contributes new Cantonese symbolic grammatical resources.

Related papers

• Grammar Engineering Meets LLMs: Development of Cantonese and Irish ParGram Treebanks [0.9310318514564272]
  This paper presents the development of Cantonese and Irish treebanks within the Parallel Grammar (ParGram) Project.<n>We also investigate the potential and limitations of using multilingual LLMs to support grammar engineering.
  arXiv  Detail & Related papers  (2026-08-07T14:44:19Z)
• Reasoning over Grammar: Can Synthetic Linguistic Reasoning Traces Enhance Low-Resource Machine Translation? [49.7935995447581]
  We investigate whether low-resource machine translation can benefit from structured intermediate steps of linguistic analysis and grammatical reasoning.<n>We propose a pipeline for automatically generating step-by-step linguistic reasoning traces from Universal Dependencies treebanks, dictionaries, and grammar-rule banks.<n>Our results show that linguistic reasoning traces are most effective as inference-time guidance.
  arXiv  Detail & Related papers  (2026-06-02T15:36:12Z)
• Exploiting the English Grammar Profile for L2 grammatical analysis with LLMs [3.027026792589421]
  The English Grammar Profile (EGP) is a taxonomy of grammatical constructs mapped to the proficiency levels of the Common European Framework of Reference (CEFR)<n>We propose a novel framework leveraging the EGP to detect learners' attempts at grammatical constructs and classify them as successful or unsuccessful.
  arXiv  Detail & Related papers  (2026-03-17T22:06:00Z)
• LingGym: How Far Are LLMs from Thinking Like Field Linguists? [20.482844306874743]
  This paper introduces LingGym, a new benchmark that evaluates LLMs' capacity for meta-linguistic reasoning.<n>We present a controlled evaluation task: Word-Gloss Inference, in which the model must infer a missing word and gloss from context.<n>Our results show that incorporating structured linguistic cues leads to consistent improvements in reasoning performance across all models.
  arXiv  Detail & Related papers  (2025-11-01T00:59:13Z)
• Read it in Two Steps: Translating Extremely Low-Resource Languages with Code-Augmented Grammar Books [30.608065466078227]
  This paper investigates the role of grammar books in translating extremely low-resource languages by decomposing it into two key steps: grammar rule retrieval and application.<n>Our experiments show that using code rules significantly boosts both rule retrieval and application, ultimately resulting in a 13.1% BLEU improvement in translation.
  arXiv  Detail & Related papers  (2025-06-02T15:36:37Z)
• How Proficient Are Large Language Models in Formal Languages? An In-Depth Insight for Knowledge Base Question Answering [52.86931192259096]
  Knowledge Base Question Answering (KBQA) aims to answer natural language questions based on facts in knowledge bases. Recent works leverage the capabilities of large language models (LLMs) for logical form generation to improve performance.
  arXiv  Detail & Related papers  (2024-01-11T09:27:50Z)
• Native Language Identification with Large Language Models [60.80452362519818]
  We show that GPT models are proficient at NLI classification, with GPT-4 setting a new performance record of 91.7% on the benchmark11 test set in a zero-shot setting. We also show that unlike previous fully-supervised settings, LLMs can perform NLI without being limited to a set of known classes.
  arXiv  Detail & Related papers  (2023-12-13T00:52:15Z)
• Democratizing LLMs for Low-Resource Languages by Leveraging their English Dominant Abilities with Linguistically-Diverse Prompts [75.33019401706188]
  Large language models (LLMs) are known to effectively perform tasks by simply observing few exemplars. We propose to assemble synthetic exemplars from a diverse set of high-resource languages to prompt the LLMs to translate from any language into English. Our unsupervised prompting method performs on par with supervised few-shot learning in LLMs of different sizes for translations between English and 13 Indic and 21 African low-resource languages.
  arXiv  Detail & Related papers  (2023-06-20T08:27:47Z)
• Grammar Prompting for Domain-Specific Language Generation with Large Language Models [40.831045850285776]
  Large language models (LLMs) can learn to perform a wide range of natural language tasks from just a handful of in-context examples. We propose emphgrammar prompting, a simple approach to enable LLMs to use external knowledge and domain-specific constraints.
  arXiv  Detail & Related papers  (2023-05-30T17:26:01Z)
• Document-Level Machine Translation with Large Language Models [91.03359121149595]
  Large language models (LLMs) can produce coherent, cohesive, relevant, and fluent answers for various natural language processing (NLP) tasks. This paper provides an in-depth evaluation of LLMs' ability on discourse modeling.
  arXiv  Detail & Related papers  (2023-04-05T03:49:06Z)
• Prompting Language Models for Linguistic Structure [73.11488464916668]
  We present a structured prompting approach for linguistic structured prediction tasks. We evaluate this approach on part-of-speech tagging, named entity recognition, and sentence chunking. We find that while PLMs contain significant prior knowledge of task labels due to task leakage into the pretraining corpus, structured prompting can also retrieve linguistic structure with arbitrary labels.
  arXiv  Detail & Related papers  (2022-11-15T01:13:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.