LM-Critic: Language Models for Unsupervised Grammatical Error Correction

• URL: http://arxiv.org/abs/2109.06822v1
• Date: Tue, 14 Sep 2021 17:06:43 GMT
• Title: LM-Critic: Language Models for Unsupervised Grammatical Error Correction
• Authors: Michihiro Yasunaga, Jure Leskovec, Percy Liang
• Abstract summary: We show how to leverage a pretrained language model (LM) in defining an LM-Critic, which judges a sentence to be grammatical. We apply this LM-Critic and BIFI along with a large set of unlabeled sentences to bootstrap realistic ungrammatical / grammatical pairs for training a corrector.
• Score: 128.9174409251852
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Training a model for grammatical error correction (GEC) requires a set of labeled ungrammatical / grammatical sentence pairs, but manually annotating such pairs can be expensive. Recently, the Break-It-Fix-It (BIFI) framework has demonstrated strong results on learning to repair a broken program without any labeled examples, but this relies on a perfect critic (e.g., a compiler) that returns whether an example is valid or not, which does not exist for the GEC task. In this work, we show how to leverage a pretrained language model (LM) in defining an LM-Critic, which judges a sentence to be grammatical if the LM assigns it a higher probability than its local perturbations. We apply this LM-Critic and BIFI along with a large set of unlabeled sentences to bootstrap realistic ungrammatical / grammatical pairs for training a corrector. We evaluate our approach on GEC datasets across multiple domains (CoNLL-2014, BEA-2019, GMEG-wiki and GMEG-yahoo) and show that it outperforms existing methods in both the unsupervised setting (+7.7 F0.5) and the supervised setting (+0.5 F0.5).

Related papers

• LM-Combiner: A Contextual Rewriting Model for Chinese Grammatical Error Correction [49.0746090186582]
  Over-correction is a critical problem in Chinese grammatical error correction (CGEC) task. Recent work using model ensemble methods can effectively mitigate over-correction and improve the precision of the GEC system. We propose the LM-Combiner, a rewriting model that can directly modify the over-correction of GEC system outputs without a model ensemble.
  arXiv  Detail & Related papers  (2024-03-26T06:12:21Z)
• Prompting open-source and commercial language models for grammatical error correction of English learner text [19.192210777082053]
  Large language models (LLMs) can be prompt to produce texts which are fluent and grammatical. We evaluate how well LLMs can perform at grammatical error correction (GEC) by measuring their performance on established benchmark datasets. We find that several open-source models outperform commercial ones on minimal edit benchmarks, and that in some settings zero-shot prompting is just as competitive as few-shot prompting.
  arXiv  Detail & Related papers  (2024-01-15T14:19:47Z)
• Chinese Spelling Correction as Rephrasing Language Model [63.65217759957206]
  We study Chinese Spelling Correction (CSC), which aims to detect and correct the potential spelling errors in a given sentence. Current state-of-the-art methods regard CSC as a sequence tagging task and fine-tune BERT-based models on sentence pairs. We propose Rephrasing Language Model (ReLM), where the model is trained to rephrase the entire sentence by infilling additional slots, instead of character-to-character tagging.
  arXiv  Detail & Related papers  (2023-08-17T06:04:28Z)
• Leveraging Denoised Abstract Meaning Representation for Grammatical Error Correction [53.55440811942249]
  Grammatical Error Correction (GEC) is the task of correcting errorful sentences into grammatically correct, semantically consistent, and coherent sentences. We propose the AMR-GEC, a seq-to-seq model that incorporates denoised AMR as additional knowledge.
  arXiv  Detail & Related papers  (2023-07-05T09:06:56Z)
• Are Pre-trained Language Models Useful for Model Ensemble in Chinese Grammatical Error Correction? [10.302225525539003]
  We explore several ensemble strategies based on strong PLMs with four sophisticated single models. The performance does not improve but even gets worse after the PLM-based ensemble.
  arXiv  Detail & Related papers  (2023-05-24T14:18:52Z)
• GRACE: Discriminator-Guided Chain-of-Thought Reasoning [75.35436025709049]
  We propose Guiding chain-of-thought ReAsoning with a CorrectnEss Discriminator (GRACE) to steer the decoding process towards producing correct reasoning steps. GRACE employs a discriminator trained with a contrastive loss over correct and incorrect steps, which is used during decoding to score next-step candidates.
  arXiv  Detail & Related papers  (2023-05-24T09:16:51Z)
• A BERT-based Unsupervised Grammatical Error Correction Framework [9.431453382607845]
  Grammatical error correction (GEC) is a challenging task of natural language processing techniques. In low-resource languages, the current unsupervised GEC based on language model scoring performs well. This study proposes a BERT-based unsupervised GEC framework, where GEC is viewed as multi-class classification task.
  arXiv  Detail & Related papers  (2023-03-30T13:29:49Z)
• A Syntax-Guided Grammatical Error Correction Model with Dependency Tree Correction [83.14159143179269]
  Grammatical Error Correction (GEC) is a task of detecting and correcting grammatical errors in sentences. We propose a syntax-guided GEC model (SG-GEC) which adopts the graph attention mechanism to utilize the syntactic knowledge of dependency trees. We evaluate our model on public benchmarks of GEC task and it achieves competitive results.
  arXiv  Detail & Related papers  (2021-11-05T07:07:48Z)
• Adversarial Grammatical Error Correction [2.132096006921048]
  We present an adversarial learning approach to Grammatical Error Correction (GEC) using the generator-discriminator framework. We pre-train both the discriminator and the generator on parallel texts and then fine-tune them further using a policy gradient method. Experimental results on FCE, CoNLL-14, and BEA-19 datasets show that Adversarial-GEC can achieve competitive GEC quality compared to NMT-based baselines.
  arXiv  Detail & Related papers  (2020-10-06T00:31:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.