Learning from Mistakes: Self-correct Adversarial Training for Chinese Unnatural Text Correction

• URL: http://arxiv.org/abs/2412.17279v1
• Date: Mon, 23 Dec 2024 04:58:58 GMT
• Title: Learning from Mistakes: Self-correct Adversarial Training for Chinese Unnatural Text Correction
• Authors: Xuan Feng, Tianlong Gu, Xiaoli Liu, Liang Chang,
• Abstract summary: Unnatural text correction aims to automatically detect and correct spelling errors or adversarial perturbation errors in sentences. Existing methods rely on fine-tuning or adversarial training to correct errors. We propose a self-correct adversarial training framework for textbfLearntextbfIng from textbfMIstextbfTakes.
• Score: 6.426690600216749
• License:
• Abstract: Unnatural text correction aims to automatically detect and correct spelling errors or adversarial perturbation errors in sentences. Existing methods typically rely on fine-tuning or adversarial training to correct errors, which have achieved significant success. However, these methods exhibit poor generalization performance due to the difference in data distribution between training data and real-world scenarios, known as the exposure bias problem. In this paper, we propose a self-correct adversarial training framework for \textbf{L}earn\textbf{I}ng from \textbf{MI}s\textbf{T}akes (\textbf{LIMIT}), which is a task- and model-independent framework to correct unnatural errors or mistakes. Specifically, we fully utilize errors generated by the model that are actively exposed during the inference phase, i.e., predictions that are inconsistent with the target. This training method not only simulates potential errors in real application scenarios, but also mitigates the exposure bias of the traditional training process. Meanwhile, we design a novel decoding intervention strategy to maintain semantic consistency. Extensive experimental results on Chinese unnatural text error correction datasets show that our proposed method can correct multiple forms of errors and outperforms the state-of-the-art text correction methods. In addition, extensive results on Chinese and English datasets validate that LIMIT can serve as a plug-and-play defense module and can extend to new models and datasets without further training.

Related papers

• Subtle Errors Matter: Preference Learning via Error-injected Self-editing [59.405145971637204]
  We propose a novel preference learning framework called eRror-Injected Self-Editing (RISE) RISE injects predefined subtle errors into partial tokens of correct solutions to construct hard pairs for error mitigation. Experiments validate the effectiveness of RISE, with preference learning on Qwen2-7B-Instruct yielding notable improvements of 3.0% on GSM8K and 7.9% on MATH.
  arXiv  Detail & Related papers  (2024-10-09T07:43:38Z)
• Physics of Language Models: Part 2.2, How to Learn From Mistakes on Grade-School Math Problems [47.753284211200665]
  We focus on understanding the usefulness of incorporating "error-correction" data directly into the pretraining stage. This data consists of erroneous solution steps immediately followed by their corrections. We show promising results: this type of pretrain data can help language models achieve higher reasoning accuracy.
  arXiv  Detail & Related papers  (2024-08-29T06:49:20Z)
• Contextual Spelling Correction with Language Model for Low-resource Setting [0.0]
  A small-scale word-based transformer LM is trained to provide the SC model with contextual understanding. Probability of error happening(error model) is extracted from the corpus. Combination of LM and error model is used to develop the SC model through the well-known noisy channel framework.
  arXiv  Detail & Related papers  (2024-04-28T05:29:35Z)
• Towards Fine-Grained Information: Identifying the Type and Location of Translation Errors [80.22825549235556]
  Existing approaches can not synchronously consider error position and type. We build an FG-TED model to predict the textbf addition and textbfomission errors. Experiments show that our model can identify both error type and position concurrently, and gives state-of-the-art results.
  arXiv  Detail & Related papers  (2023-02-17T16:20:33Z)
• uChecker: Masked Pretrained Language Models as Unsupervised Chinese Spelling Checkers [23.343006562849126]
  We propose a framework named textbfuChecker to conduct unsupervised spelling error detection and correction. Masked pretrained language models such as BERT are introduced as the backbone model. Benefiting from the various and flexible MASKing operations, we propose a Confusionset-guided masking strategy to fine-train the masked language model.
  arXiv  Detail & Related papers  (2022-09-15T05:57:12Z)
• Improving Pre-trained Language Models with Syntactic Dependency Prediction Task for Chinese Semantic Error Recognition [52.55136323341319]
  Existing Chinese text error detection mainly focuses on spelling and simple grammatical errors. Chinese semantic errors are understudied and more complex that humans cannot easily recognize.
  arXiv  Detail & Related papers  (2022-04-15T13:55:32Z)
• Tail-to-Tail Non-Autoregressive Sequence Prediction for Chinese Grammatical Error Correction [49.25830718574892]
  We present a new framework named Tail-to-Tail (textbfTtT) non-autoregressive sequence prediction. Considering that most tokens are correct and can be conveyed directly from source to target, and the error positions can be estimated and corrected. Experimental results on standard datasets, especially on the variable-length datasets, demonstrate the effectiveness of TtT in terms of sentence-level Accuracy, Precision, Recall, and F1-Measure.
  arXiv  Detail & Related papers  (2021-06-03T05:56:57Z)
• On the Robustness of Language Encoders against Grammatical Errors [66.05648604987479]
  We collect real grammatical errors from non-native speakers and conduct adversarial attacks to simulate these errors on clean text data. Results confirm that the performance of all tested models is affected but the degree of impact varies.
  arXiv  Detail & Related papers  (2020-05-12T11:01:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.