Beyond Hard Samples: Robust and Effective Grammatical Error Correction with Cycle Self-Augmenting

• URL: http://arxiv.org/abs/2310.13321v2
• Date: Mon, 23 Oct 2023 07:41:09 GMT
• Title: Beyond Hard Samples: Robust and Effective Grammatical Error Correction with Cycle Self-Augmenting
• Authors: Zecheng Tang, Kaifeng Qi, Juntao Li, Min Zhang
• Abstract summary: We propose a Cycle Self-Augmenting (CSA) method to enhance the robustness of grammatical error correction (GEC) models in adversarial attacks. By leveraging the augmenting data from the GEC models themselves in the post-training process and introducing regularization data for cycle training, our proposed method can effectively improve the model robustness of well-trained GEC models. Experiments on four benchmark datasets and seven strong models indicate that our proposed training method can significantly enhance the robustness of four types of attacks without using purposely built adversarial examples in training.
• Score: 28.84445227362245
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent studies have revealed that grammatical error correction methods in the sequence-to-sequence paradigm are vulnerable to adversarial attack, and simply utilizing adversarial examples in the pre-training or post-training process can significantly enhance the robustness of GEC models to certain types of attack without suffering too much performance loss on clean data. In this paper, we further conduct a thorough robustness evaluation of cutting-edge GEC methods for four different types of adversarial attacks and propose a simple yet very effective Cycle Self-Augmenting (CSA) method accordingly. By leveraging the augmenting data from the GEC models themselves in the post-training process and introducing regularization data for cycle training, our proposed method can effectively improve the model robustness of well-trained GEC models with only a few more training epochs as an extra cost. More concretely, further training on the regularization data can prevent the GEC models from over-fitting on easy-to-learn samples and thus can improve the generalization capability and robustness towards unseen data (adversarial noise/samples). Meanwhile, the self-augmented data can provide more high-quality pseudo pairs to improve model performance on the original testing data. Experiments on four benchmark datasets and seven strong models indicate that our proposed training method can significantly enhance the robustness of four types of attacks without using purposely built adversarial examples in training. Evaluation results on clean data further confirm that our proposed CSA method significantly improves the performance of four baselines and yields nearly comparable results with other state-of-the-art models. Our code is available at https://github.com/ZetangForward/CSA-GEC.

Related papers

• Smart Cuts: Enhance Active Learning for Vulnerability Detection by Pruning Bad Seeds [15.490968013867562]
  Vulnerability detection is crucial for identifying security weaknesses in software systems.<n>This paper proposes a novel dataset maps-empowered approach that identifies and mitigates hard-to-learn outliers.<n>Our approach can categorize training examples based on learning difficulty and integrate this information into an active learning framework.
  arXiv  Detail & Related papers  (2025-06-25T13:50:21Z)
• LEAD: Iterative Data Selection for Efficient LLM Instruction Tuning [22.242445543184264]
  We propose LEAD, an efficient iterative data selection framework that accurately estimates sample utility entirely within the standard training loop.<n>Experiments show that LEAD significantly outperforms state-of-the-art methods, improving average model performance by 6.1%-10.8% while using only 2.5% of the training data and reducing overall training time by 5-10x.
  arXiv  Detail & Related papers  (2025-05-12T10:57:51Z)
• Systematic Evaluation of Synthetic Data Augmentation for Multi-class NetFlow Traffic [2.5182419298876857]
  Multi-class classification models can identify specific types of attacks, allowing for more targeted and effective incident responses. Recent advances suggest that generative models can assist in data augmentation, claiming to offer superior solutions for imbalanced datasets. Our experiments indicate that resampling methods for balancing training data do not reliably improve classification performance.
  arXiv  Detail & Related papers  (2024-08-28T12:44:07Z)
• Adversarial Robustification via Text-to-Image Diffusion Models [56.37291240867549]
  Adrial robustness has been conventionally believed as a challenging property to encode for neural networks. We develop a scalable and model-agnostic solution to achieve adversarial robustness without using any data.
  arXiv  Detail & Related papers  (2024-07-26T10:49:14Z)
• PairCFR: Enhancing Model Training on Paired Counterfactually Augmented Data through Contrastive Learning [49.60634126342945]
  Counterfactually Augmented Data (CAD) involves creating new data samples by applying minimal yet sufficient modifications to flip the label of existing data samples to other classes. Recent research reveals that training with CAD may lead models to overly focus on modified features while ignoring other important contextual information. We employ contrastive learning to promote global feature alignment in addition to learning counterfactual clues.
  arXiv  Detail & Related papers  (2024-06-09T07:29:55Z)
• Improving Data Augmentation for Robust Visual Question Answering with Effective Curriculum Learning [12.647353699551081]
  We design an Effective Curriculum Learning strategy ECL to enhance DA-based VQA methods. ECL trains VQA models on relatively easy'' samples first, and then gradually changes to harder'' samples, and less-valuable samples are dynamically removed. Compared to training on the entire augmented dataset, our ECL strategy can further enhance VQA models' performance with fewer training samples.
  arXiv  Detail & Related papers  (2024-01-28T12:48:16Z)
• Efficient Grammatical Error Correction Via Multi-Task Training and Optimized Training Schedule [55.08778142798106]
  We propose auxiliary tasks that exploit the alignment between the original and corrected sentences. We formulate each task as a sequence-to-sequence problem and perform multi-task training. We find that the order of datasets used for training and even individual instances within a dataset may have important effects on the final performance.
  arXiv  Detail & Related papers  (2023-11-20T14:50:12Z)
• Boosting Model Inversion Attacks with Adversarial Examples [26.904051413441316]
  We propose a new training paradigm for a learning-based model inversion attack that can achieve higher attack accuracy in a black-box setting. First, we regularize the training process of the attack model with an added semantic loss function. Second, we inject adversarial examples into the training data to increase the diversity of the class-related parts.
  arXiv  Detail & Related papers  (2023-06-24T13:40:58Z)
• Universal Domain Adaptation from Foundation Models: A Baseline Study [58.51162198585434]
  We make empirical studies of state-of-the-art UniDA methods using foundation models. We introduce textitCLIP distillation, a parameter-free method specifically designed to distill target knowledge from CLIP models. Although simple, our method outperforms previous approaches in most benchmark tasks.
  arXiv  Detail & Related papers  (2023-05-18T16:28:29Z)
• Exploration and Exploitation: Two Ways to Improve Chinese Spelling Correction Models [51.744357472072416]
  We propose a method, which continually identifies the weak spots of a model to generate more valuable training instances. Experimental results show that such an adversarial training method combined with the pretraining strategy can improve both the generalization and robustness of multiple CSC models.
  arXiv  Detail & Related papers  (2021-05-31T09:17:33Z)
• A Self-Refinement Strategy for Noise Reduction in Grammatical Error Correction [54.569707226277735]
  Existing approaches for grammatical error correction (GEC) rely on supervised learning with manually created GEC datasets. There is a non-negligible amount of "noise" where errors were inappropriately edited or left uncorrected. We propose a self-refinement method where the key idea is to denoise these datasets by leveraging the prediction consistency of existing models.
  arXiv  Detail & Related papers  (2020-10-07T04:45:09Z)
• Data Weighted Training Strategies for Grammatical Error Correction [8.370770440898454]
  We show how to incorporate delta-log-perplexity, a type of example scoring, into a training schedule for Grammatical Error Correction (GEC) Models trained on scored data achieve state-of-the-art results on common GEC test sets.
  arXiv  Detail & Related papers  (2020-08-07T03:30:14Z)
• Generative Data Augmentation for Commonsense Reasoning [75.26876609249197]
  G-DAUGC is a novel generative data augmentation method that aims to achieve more accurate and robust learning in the low-resource setting. G-DAUGC consistently outperforms existing data augmentation methods based on back-translation. Our analysis demonstrates that G-DAUGC produces a diverse set of fluent training examples, and that its selection and training approaches are important for performance.
  arXiv  Detail & Related papers  (2020-04-24T06:12:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.