Stronger Baselines for Grammatical Error Correction Using Pretrained Encoder-Decoder Model

• URL: http://arxiv.org/abs/2005.11849v2
• Date: Wed, 30 Sep 2020 02:57:04 GMT
• Title: Stronger Baselines for Grammatical Error Correction Using Pretrained Encoder-Decoder Model
• Authors: Satoru Katsumata and Mamoru Komachi
• Abstract summary: We explore the utility of bidirectional and auto-regressive transformers (BART) as a generic pretrained encoder-decoder model for grammatical error correction (GEC) We find that monolingual and multilingual BART models achieve high performance in GEC, with one of the results being comparable to the current strong results in English GEC.
• Score: 24.51571980021599
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Studies on grammatical error correction (GEC) have reported the effectiveness of pretraining a Seq2Seq model with a large amount of pseudodata. However, this approach requires time-consuming pretraining for GEC because of the size of the pseudodata. In this study, we explore the utility of bidirectional and auto-regressive transformers (BART) as a generic pretrained encoder-decoder model for GEC. With the use of this generic pretrained model for GEC, the time-consuming pretraining can be eliminated. We find that monolingual and multilingual BART models achieve high performance in GEC, with one of the results being comparable to the current strong results in English GEC. Our implementations are publicly available at GitHub (https://github.com/Katsumata420/generic-pretrained-GEC).

Related papers

• LM-Combiner: A Contextual Rewriting Model for Chinese Grammatical Error Correction [49.0746090186582]
  Over-correction is a critical problem in Chinese grammatical error correction (CGEC) task. Recent work using model ensemble methods can effectively mitigate over-correction and improve the precision of the GEC system. We propose the LM-Combiner, a rewriting model that can directly modify the over-correction of GEC system outputs without a model ensemble.
  arXiv  Detail & Related papers  (2024-03-26T06:12:21Z)
• GEC-DePenD: Non-Autoregressive Grammatical Error Correction with Decoupled Permutation and Decoding [52.14832976759585]
  Grammatical error correction (GEC) is an important NLP task that is usually solved with autoregressive sequence-to-sequence models. We propose a novel non-autoregressive approach to GEC that decouples the architecture into a permutation network. We show that the resulting network improves over previously known non-autoregressive methods for GEC.
  arXiv  Detail & Related papers  (2023-11-14T14:24:36Z)
• Improving Seq2Seq Grammatical Error Correction via Decoding Interventions [40.52259641181596]
  We propose a unified decoding intervention framework that employs an external critic to assess the appropriateness of the token to be generated incrementally. We discover and investigate two types of critics: a pre-trained left-to-right language model critic and an incremental target-side grammatical error detector critic. Our framework consistently outperforms strong baselines and achieves results competitive with state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-10-23T03:36:37Z)
• Leveraging Denoised Abstract Meaning Representation for Grammatical Error Correction [53.55440811942249]
  Grammatical Error Correction (GEC) is the task of correcting errorful sentences into grammatically correct, semantically consistent, and coherent sentences. We propose the AMR-GEC, a seq-to-seq model that incorporates denoised AMR as additional knowledge.
  arXiv  Detail & Related papers  (2023-07-05T09:06:56Z)
• Are Pre-trained Language Models Useful for Model Ensemble in Chinese Grammatical Error Correction? [10.302225525539003]
  We explore several ensemble strategies based on strong PLMs with four sophisticated single models. The performance does not improve but even gets worse after the PLM-based ensemble.
  arXiv  Detail & Related papers  (2023-05-24T14:18:52Z)
• Advancements in Arabic Grammatical Error Detection and Correction: An Empirical Investigation [12.15509670220182]
  Grammatical error correction (GEC) is a well-explored problem in English. Research on GEC in morphologically rich languages has been limited due to challenges such as data scarcity and language complexity. We present the first results on Arabic GEC using two newly developed Transformer-based pretrained sequence-to-sequence models.
  arXiv  Detail & Related papers  (2023-05-24T05:12:58Z)
• A Unified Strategy for Multilingual Grammatical Error Correction with Pre-trained Cross-Lingual Language Model [100.67378875773495]
  We propose a generic and language-independent strategy for multilingual Grammatical Error Correction. Our approach creates diverse parallel GEC data without any language-specific operations. It achieves the state-of-the-art results on the NLPCC 2018 Task 2 dataset (Chinese) and obtains competitive performance on Falko-Merlin (German) and RULEC-GEC (Russian)
  arXiv  Detail & Related papers  (2022-01-26T02:10:32Z)
• ErAConD : Error Annotated Conversational Dialog Dataset for Grammatical Error Correction [30.917993017459615]
  We present a novel parallel grammatical error correction (GEC) dataset drawn from open-domain conversations. This dataset is, to our knowledge, the first GEC dataset targeted to a conversational setting. To demonstrate the utility of the dataset, we use our annotated data to fine-tune a state-of-the-art GEC model.
  arXiv  Detail & Related papers  (2021-12-15T20:27:40Z)
• Do Grammatical Error Correction Models Realize Grammatical Generalization? [8.569720582920416]
  This study explores to what extent GEC models generalize grammatical knowledge required for correcting errors. We found that a current standard Transformer-based GEC model fails to realize grammatical generalization even in simple settings.
  arXiv  Detail & Related papers  (2021-06-06T04:59:29Z)
• Neural Quality Estimation with Multiple Hypotheses for Grammatical Error Correction [98.31440090585376]
  Grammatical Error Correction (GEC) aims to correct writing errors and help language learners improve their writing skills. Existing GEC models tend to produce spurious corrections or fail to detect lots of errors. This paper presents the Neural Verification Network (VERNet) for GEC quality estimation with multiple hypotheses.
  arXiv  Detail & Related papers  (2021-05-10T15:04:25Z)
• Encoder-Decoder Models Can Benefit from Pre-trained Masked Language Models in Grammatical Error Correction [54.569707226277735]
  Previous methods have potential drawbacks when applied to an EncDec model. Our proposed method fine-tune a corpus and then use the output fine-tuned as additional features in the GEC model. The best-performing model state-of-the-art performances on the BEA 2019 and CoNLL-2014 benchmarks.
  arXiv  Detail & Related papers  (2020-05-03T04:49:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.