Enhancing NLP Robustness and Generalization through LLM-Generated Contrast Sets: A Scalable Framework for Systematic Evaluation and Adversarial Training

• URL: http://arxiv.org/abs/2503.06648v1
• Date: Sun, 09 Mar 2025 14:52:53 GMT
• Title: Enhancing NLP Robustness and Generalization through LLM-Generated Contrast Sets: A Scalable Framework for Systematic Evaluation and Adversarial Training
• Authors: Hender Lin,
• Abstract summary: We create a 3,000-example contrast set to evaluate and improve model robustness.<n>Fine-tuning on these contrast sets enhanced performance on systematically perturbed examples, maintained standard test accuracy, and modestly improved generalization to novel perturbations.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Standard NLP benchmarks often fail to capture vulnerabilities stemming from dataset artifacts and spurious correlations. Contrast sets address this gap by challenging models near decision boundaries but are traditionally labor-intensive to create and limited in diversity. This study leverages large language models to automate the generation of diverse contrast sets. Using the SNLI dataset, we created a 3,000-example contrast set to evaluate and improve model robustness. Fine-tuning on these contrast sets enhanced performance on systematically perturbed examples, maintained standard test accuracy, and modestly improved generalization to novel perturbations. This automated approach offers a scalable solution for evaluating and improving NLP models, addressing systematic generalization challenges, and advancing robustness in real-world applications.

Related papers

• SALAD: Improving Robustness and Generalization through Contrastive Learning with Structure-Aware and LLM-Driven Augmented Data [15.366930934639838]
  We propose SALAD, a novel approach to enhance model robustness and generalization. Our method generates structure-aware and counterfactually augmented data for contrastive learning. We validate our approach through experiments on three tasks: Sentiment Classification, Sexism Detection, and Natural Language Inference.
  arXiv  Detail & Related papers  (2025-04-16T15:40:10Z)
• Unified Enhancement of the Generalization and Robustness of Language Models via Bi-Stage Optimization [2.502393972789905]
  We propose a bi-stage optimization framework to uniformly enhance both the generalization and robustness of LMs. We show that our method significantly improves the generalization and robustness of LMs compared to other existing methods.
  arXiv  Detail & Related papers  (2025-03-19T13:50:36Z)
• Few-Shot Optimized Framework for Hallucination Detection in Resource-Limited NLP Systems [1.0124625066746595]
  We introduce DeepSeek Few-shot optimization to enhance weak label generation through iterative prompt engineering.<n>We achieve high-quality annotations that considerably enhanced the performance of downstream models.<n>We further fine-tuned the Mistral-7B-Instruct-v0.3 model on these optimized annotations, enabling it to accurately detect hallucinations in resource-limited settings.
  arXiv  Detail & Related papers  (2025-01-28T01:26:22Z)
• Synthetic Feature Augmentation Improves Generalization Performance of Language Models [8.463273762997398]
  Training and fine-tuning deep learning models on limited and imbalanced datasets poses substantial challenges.<n>We propose augmenting features in the embedding space by generating synthetic samples using a range of techniques.<n>We validate the effectiveness of this approach across multiple open-source text classification benchmarks.
  arXiv  Detail & Related papers  (2025-01-11T04:31:18Z)
• Preserving Multi-Modal Capabilities of Pre-trained VLMs for Improving Vision-Linguistic Compositionality [69.76121008898677]
  Fine-grained Selective Calibrated CLIP integrates local hard negative loss and selective calibrated regularization. Our evaluations show that FSC-CLIP not only achieves compositionality on par with state-of-the-art models but also retains strong multi-modal capabilities.
  arXiv  Detail & Related papers  (2024-10-07T17:16:20Z)
• Dynamic Post-Hoc Neural Ensemblers [55.15643209328513]
  In this study, we explore employing neural networks as ensemble methods. Motivated by the risk of learning low-diversity ensembles, we propose regularizing the model by randomly dropping base model predictions. We demonstrate this approach lower bounds the diversity within the ensemble, reducing overfitting and improving generalization capabilities.
  arXiv  Detail & Related papers  (2024-10-06T15:25:39Z)
• Uncertainty Aware Learning for Language Model Alignment [97.36361196793929]
  We propose uncertainty-aware learning (UAL) to improve the model alignment of different task scenarios. We implement UAL in a simple fashion -- adaptively setting the label smoothing value of training according to the uncertainty of individual samples. Experiments on widely used benchmarks demonstrate that our UAL significantly and consistently outperforms standard supervised fine-tuning.
  arXiv  Detail & Related papers  (2024-06-07T11:37:45Z)
• Don't Forget Your Reward Values: Language Model Alignment via Value-based Calibration [26.467379188463028]
  We propose a novel textbfValue-based textbfCalitextbfBration (VCB) method to better align Large Language Models with human preferences. Experimental results demonstrate that VCB surpasses existing alignment methods on AI assistant and summarization datasets.
  arXiv  Detail & Related papers  (2024-02-25T08:45:10Z)
• Tackling Diverse Minorities in Imbalanced Classification [80.78227787608714]
  Imbalanced datasets are commonly observed in various real-world applications, presenting significant challenges in training classifiers. We propose generating synthetic samples iteratively by mixing data samples from both minority and majority classes. We demonstrate the effectiveness of our proposed framework through extensive experiments conducted on seven publicly available benchmark datasets.
  arXiv  Detail & Related papers  (2023-08-28T18:48:34Z)
• Consistency Regularization for Generalizable Source-free Domain Adaptation [62.654883736925456]
  Source-free domain adaptation (SFDA) aims to adapt a well-trained source model to an unlabelled target domain without accessing the source dataset. Existing SFDA methods ONLY assess their adapted models on the target training set, neglecting the data from unseen but identically distributed testing sets. We propose a consistency regularization framework to develop a more generalizable SFDA method.
  arXiv  Detail & Related papers  (2023-08-03T07:45:53Z)
• Modeling the Q-Diversity in a Min-max Play Game for Robust Optimization [61.39201891894024]
  Group distributionally robust optimization (group DRO) can minimize the worst-case loss over pre-defined groups. We reformulate the group DRO framework by proposing Q-Diversity. Characterized by an interactive training mode, Q-Diversity relaxes the group identification from annotation into direct parameterization.
  arXiv  Detail & Related papers  (2023-05-20T07:02:27Z)
• Jacobian Ensembles Improve Robustness Trade-offs to Adversarial Attacks [5.70772577110828]
  We propose a novel approach, Jacobian Ensembles, to increase the robustness against UAPs. Our results show that Jacobian Ensembles achieves previously unseen levels of accuracy and robustness.
  arXiv  Detail & Related papers  (2022-04-19T08:04:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.