CoDA: Contrast-enhanced and Diversity-promoting Data Augmentation for Natural Language Understanding

• URL: http://arxiv.org/abs/2010.08670v1
• Date: Fri, 16 Oct 2020 23:57:03 GMT
• Title: CoDA: Contrast-enhanced and Diversity-promoting Data Augmentation for Natural Language Understanding
• Authors: Yanru Qu, Dinghan Shen, Yelong Shen, Sandra Sajeev, Jiawei Han, Weizhu Chen
• Abstract summary: We propose a novel data augmentation framework dubbed CoDA. CoDA synthesizes diverse and informative augmented examples by integrating multiple transformations organically. A contrastive regularization objective is introduced to capture the global relationship among all the data samples.
• Score: 67.61357003974153
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Data augmentation has been demonstrated as an effective strategy for improving model generalization and data efficiency. However, due to the discrete nature of natural language, designing label-preserving transformations for text data tends to be more challenging. In this paper, we propose a novel data augmentation framework dubbed CoDA, which synthesizes diverse and informative augmented examples by integrating multiple transformations organically. Moreover, a contrastive regularization objective is introduced to capture the global relationship among all the data samples. A momentum encoder along with a memory bank is further leveraged to better estimate the contrastive loss. To verify the effectiveness of the proposed framework, we apply CoDA to Transformer-based models on a wide range of natural language understanding tasks. On the GLUE benchmark, CoDA gives rise to an average improvement of 2.2% while applied to the RoBERTa-large model. More importantly, it consistently exhibits stronger results relative to several competitive data augmentation and adversarial training base-lines (including the low-resource settings). Extensive experiments show that the proposed contrastive objective can be flexibly combined with various data augmentation approaches to further boost their performance, highlighting the wide applicability of the CoDA framework.

Related papers

• A Simple Background Augmentation Method for Object Detection with Diffusion Model [53.32935683257045]
  In computer vision, it is well-known that a lack of data diversity will impair model performance. We propose a simple yet effective data augmentation approach by leveraging advancements in generative models. Background augmentation, in particular, significantly improves the models' robustness and generalization capabilities.
  arXiv  Detail & Related papers  (2024-08-01T07:40:00Z)
• AdaAugment: A Tuning-Free and Adaptive Approach to Enhance Data Augmentation [12.697608744311122]
  AdaAugment is a tuning-free Adaptive Augmentation method. It dynamically adjusts augmentation magnitudes for individual training samples based on real-time feedback from the target network. It consistently outperforms other state-of-the-art DA methods in effectiveness while maintaining remarkable efficiency.
  arXiv  Detail & Related papers  (2024-05-19T06:54:03Z)
• MixEdit: Revisiting Data Augmentation and Beyond for Grammatical Error Correction [24.370610646959907]
  We propose MixEdit, a data augmentation approach that strategically and dynamically augments realistic data, without requiring extra monolingual corpora. The results show that MixEdit substantially improves GEC models and is complementary to traditional data augmentation methods.
  arXiv  Detail & Related papers  (2023-10-18T02:45:51Z)
• DualAug: Exploiting Additional Heavy Augmentation with OOD Data Rejection [77.6648187359111]
  We propose a novel data augmentation method, named textbfDualAug, to keep the augmentation in distribution as much as possible at a reasonable time and computational cost. Experiments on supervised image classification benchmarks show that DualAug improve various automated data augmentation method.
  arXiv  Detail & Related papers  (2023-10-12T08:55:10Z)
• Implicit Counterfactual Data Augmentation for Robust Learning [24.795542869249154]
  This study proposes an Implicit Counterfactual Data Augmentation method to remove spurious correlations and make stable predictions. Experiments have been conducted across various biased learning scenarios covering both image and text datasets.
  arXiv  Detail & Related papers  (2023-04-26T10:36:40Z)
• Contrastive Learning with Consistent Representations [8.364383223740097]
  This paper proposes Contrastive Learning with Consistent Representations CoCor. At the heart of CoCor is a novel consistency metric termed DA consistency. Experimental results demonstrate that CoCor notably enhances the generalizability and transferability of learned representations.
  arXiv  Detail & Related papers  (2023-02-03T04:34:00Z)
• CAFE: Learning to Condense Dataset by Aligning Features [72.99394941348757]
  We propose a novel scheme to Condense dataset by Aligning FEatures (CAFE) At the heart of our approach is an effective strategy to align features from the real and synthetic data across various scales. We validate the proposed CAFE across various datasets, and demonstrate that it generally outperforms the state of the art.
  arXiv  Detail & Related papers  (2022-03-03T05:58:49Z)
• DivAug: Plug-in Automated Data Augmentation with Explicit Diversity Maximization [41.82120128496555]
  Two factors regarding the diversity of augmented data are still missing: 1) the explicit definition (and thus measurement) of diversity and 2) the quantifiable relationship between diversity and its regularization effects. We propose a diversity measure called Variance Diversity and theoretically show that the regularization effect of data augmentation is promised by Variance Diversity. An unsupervised sampling-based framework, DivAug, is designed to directly maximize Variance Diversity and hence strengthen the regularization effect.
  arXiv  Detail & Related papers  (2021-03-26T16:00:01Z)
• SDA: Improving Text Generation with Self Data Augmentation [88.24594090105899]
  We propose to improve the standard maximum likelihood estimation (MLE) paradigm by incorporating a self-imitation-learning phase for automatic data augmentation. Unlike most existing sentence-level augmentation strategies, our method is more general and could be easily adapted to any MLE-based training procedure.
  arXiv  Detail & Related papers  (2021-01-02T01:15:57Z)
• Generative Data Augmentation for Commonsense Reasoning [75.26876609249197]
  G-DAUGC is a novel generative data augmentation method that aims to achieve more accurate and robust learning in the low-resource setting. G-DAUGC consistently outperforms existing data augmentation methods based on back-translation. Our analysis demonstrates that G-DAUGC produces a diverse set of fluent training examples, and that its selection and training approaches are important for performance.
  arXiv  Detail & Related papers  (2020-04-24T06:12:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.