Organic Data-Driven Approach for Turkish Grammatical Error Correction and LLMs

• URL: http://arxiv.org/abs/2405.15320v1
• Date: Fri, 24 May 2024 08:00:24 GMT
• Title: Organic Data-Driven Approach for Turkish Grammatical Error Correction and LLMs
• Authors: Asım Ersoy, Olcay Taner Yıldız,
• Abstract summary: We introduce a new organic data-driven approach, clean insertions, to build parallel Turkish Grammatical Error Correction datasets. We achieve state-of-the-art results on two Turkish Grammatical Error Correction test sets out of the three publicly available ones.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Grammatical Error Correction has seen significant progress with the recent advancements in deep learning. As those methods require huge amounts of data, synthetic datasets are being built to fill this gap. Unfortunately, synthetic datasets are not organic enough in some cases and even require clean data to start with. Furthermore, most of the work that has been done is focused mostly on English. In this work, we introduce a new organic data-driven approach, clean insertions, to build parallel Turkish Grammatical Error Correction datasets from any organic data, and to clean the data used for training Large Language Models. We achieve state-of-the-art results on two Turkish Grammatical Error Correction test sets out of the three publicly available ones. We also show the effectiveness of our method on the training losses of training language models.

Related papers

• Physics of Language Models: Part 2.2, How to Learn From Mistakes on Grade-School Math Problems [47.753284211200665]
  We focus on understanding the usefulness of incorporating "error-correction" data directly into the pretraining stage. This data consists of erroneous solution steps immediately followed by their corrections. We show promising results: this type of pretrain data can help language models achieve higher reasoning accuracy.
  arXiv  Detail & Related papers  (2024-08-29T06:49:20Z)
• Do Not Worry if You Do Not Have Data: Building Pretrained Language Models Using Translationese [47.45957604683302]
  Pre-training requires vast amounts of monolingual data, which is mostly unavailable for languages other than English. We take the case of English and Indic languages and translate web-crawled monolingual documents (clean) into the target language. Then, we train language models containing 28M and 85M parameters on this translationese data (synthetic) We show that their performance on downstream natural language understanding and generative tasks is only 3.56% poorer on NLU tasks and 1.51% on NLG tasks than LMs pre-trained on clean data.
  arXiv  Detail & Related papers  (2024-03-20T14:41:01Z)
• Efficient Grammatical Error Correction Via Multi-Task Training and Optimized Training Schedule [55.08778142798106]
  We propose auxiliary tasks that exploit the alignment between the original and corrected sentences. We formulate each task as a sequence-to-sequence problem and perform multi-task training. We find that the order of datasets used for training and even individual instances within a dataset may have important effects on the final performance.
  arXiv  Detail & Related papers  (2023-11-20T14:50:12Z)
• Let's Synthesize Step by Step: Iterative Dataset Synthesis with Large Language Models by Extrapolating Errors from Small Models [69.76066070227452]
  *Data Synthesis* is a promising way to train a small model with very little labeled data. We propose *Synthesis Step by Step* (**S3**), a data synthesis framework that shrinks this distribution gap. Our approach improves the performance of a small model by reducing the gap between the synthetic dataset and the real data.
  arXiv  Detail & Related papers  (2023-10-20T17:14:25Z)
• GECTurk: Grammatical Error Correction and Detection Dataset for Turkish [1.804922416527064]
  Grammatical Error Detection and Correction (GEC) tools have proven useful for native speakers and second language learners. Synthetic data generation is a common practice to overcome the scarcity of such data. We present a flexible and synthetic data generation pipeline for Turkish covering more than 20 expert-curated grammar and spelling rules.
  arXiv  Detail & Related papers  (2023-09-20T14:25:44Z)
• ASDOT: Any-Shot Data-to-Text Generation with Pretrained Language Models [82.63962107729994]
  Any-Shot Data-to-Text (ASDOT) is a new approach flexibly applicable to diverse settings. It consists of two steps, data disambiguation and sentence fusion. Experimental results show that ASDOT consistently achieves significant improvement over baselines.
  arXiv  Detail & Related papers  (2022-10-09T19:17:43Z)
• Towards Zero-Label Language Learning [20.28186484098947]
  This paper explores zero-label learning in Natural Language Processing (NLP) No human-annotated data is used anywhere during training and models are trained purely on synthetic data. Inspired by the recent success of few-shot inference on GPT-3, we present a training data creation procedure named Unsupervised Data Generation.
  arXiv  Detail & Related papers  (2021-09-19T19:00:07Z)
• Synthetic Data Generation for Grammatical Error Correction with Tagged Corruption Models [15.481446439370343]
  We use error type tags from automatic annotation tools such as ERRANT to guide synthetic data generation. We build a new, large synthetic pre-training data set with error tag frequency distributions matching a given development set. Our approach is particularly effective in adapting a GEC system, trained on mixed native and non-native English, to a native English test set.
  arXiv  Detail & Related papers  (2021-05-27T17:17:21Z)
• Comparison of Interactive Knowledge Base Spelling Correction Models for Low-Resource Languages [81.90356787324481]
  Spelling normalization for low resource languages is a challenging task because the patterns are hard to predict. This work shows a comparison of a neural model and character language models with varying amounts on target language data. Our usage scenario is interactive correction with nearly zero amounts of training examples, improving models as more data is collected.
  arXiv  Detail & Related papers  (2020-10-20T17:31:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.