Healing Powers of BERT: How Task-Specific Fine-Tuning Recovers Corrupted Language Models

• URL: http://arxiv.org/abs/2406.14459v2
• Date: Tue, 08 Jul 2025 10:13:07 GMT
• Title: Healing Powers of BERT: How Task-Specific Fine-Tuning Recovers Corrupted Language Models
• Authors: Shijie Han, Zhenyu Zhang, Andrei Arsene Simion,
• Abstract summary: We look at what happens if a language model is "broken", in the sense that some of its parameters are corrupted and then recovered by fine-tuning.<n>We find corrupted models struggle to fully recover their original performance, with higher corruption causing more severe degradation.<n>Our insights contribute to understanding language model robustness and adaptability under adverse conditions, informing strategies for developing resilient NLP systems.
• Score: 4.793753685154721
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Language models like BERT excel at sentence classification tasks due to extensive pre-training on general data, but their robustness to parameter corruption is unexplored. To understand this better, we look at what happens if a language model is "broken", in the sense that some of its parameters are corrupted and then recovered by fine-tuning. Strategically corrupting BERT variants at different levels, we find corrupted models struggle to fully recover their original performance, with higher corruption causing more severe degradation. Notably, bottom-layer corruption affecting fundamental linguistic features is more detrimental than top-layer corruption. Our insights contribute to understanding language model robustness and adaptability under adverse conditions, informing strategies for developing resilient NLP systems against parameter perturbations.

Related papers

• Stress-Testing ML Pipelines with Adversarial Data Corruption [11.91482648083998]
  Regulators now demand evidence that high-stakes systems can withstand realistic, interdependent errors.<n>We introduce SAVAGE, a framework that formally models data-quality issues through dependency graphs and flexible corruption templates.<n>Savanage employs a bi-level optimization approach to efficiently identify vulnerable data subpopulations and fine-tune corruption severity.
  arXiv  Detail & Related papers  (2025-06-02T00:41:24Z)
• Corrupted but Not Broken: Rethinking the Impact of Corrupted Data in Visual Instruction Tuning [85.58172296577506]
  We study how corrupted data affects Multimodal Large Language Models (MLLMs) We find that while corrupted data degrades the performance of MLLMs, its effects are largely superficial. We propose a corruption-robust training paradigm combining self-validation and post-training, which significantly outperforms existing corruption mitigation strategies.
  arXiv  Detail & Related papers  (2025-02-18T08:28:29Z)
• Towards Robust Pruning: An Adaptive Knowledge-Retention Pruning Strategy for Language Models [35.58379464827462]
  We introduce a post-training pruning strategy designed to faithfully replicate the embedding space and feature space of dense language models. Compared to other state-of-art baselines, our approach demonstrates a superior balance between accuracy, sparsity, robustness, and pruning cost with BERT on datasets SST2, IMDB, and AGNews.
  arXiv  Detail & Related papers  (2023-10-19T23:02:29Z)
• Towards preserving word order importance through Forced Invalidation [80.33036864442182]
  We show that pre-trained language models are insensitive to word order. We propose Forced Invalidation to help preserve the importance of word order. Our experiments demonstrate that Forced Invalidation significantly improves the sensitivity of the models to word order.
  arXiv  Detail & Related papers  (2023-04-11T13:42:10Z)
• How Does Data Corruption Affect Natural Language Understanding Models? A Study on GLUE datasets [4.645287693363387]
  We show that performance remains high for most GLUE tasks when the models are fine-tuned or tested on corrupted data. Our proposed data transformations can be used as a diagnostic tool for assessing the extent to which a specific dataset constitutes a proper testbed for evaluating models' language understanding capabilities.
  arXiv  Detail & Related papers  (2022-01-12T13:35:53Z)
• Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Models [86.02610674750345]
  Adversarial GLUE (AdvGLUE) is a new multi-task benchmark to explore and evaluate the vulnerabilities of modern large-scale language models under various types of adversarial attacks. We apply 14 adversarial attack methods to GLUE tasks to construct AdvGLUE, which is further validated by humans for reliable annotations. All the language models and robust training methods we tested perform poorly on AdvGLUE, with scores lagging far behind the benign accuracy.
  arXiv  Detail & Related papers  (2021-11-04T12:59:55Z)
• NoiER: An Approach for Training more Reliable Fine-TunedDownstream Task Models [54.184609286094044]
  We propose noise entropy regularisation (NoiER) as an efficient learning paradigm that solves the problem without auxiliary models and additional data. The proposed approach improved traditional OOD detection evaluation metrics by 55% on average compared to the original fine-tuned models.
  arXiv  Detail & Related papers  (2021-08-29T06:58:28Z)
• NLI Data Sanity Check: Assessing the Effect of Data Corruption on Model Performance [3.7024660695776066]
  We propose a new diagnostics test suite which allows to assess whether a dataset constitutes a good testbed for evaluating the models' meaning understanding capabilities. We specifically apply controlled corruption transformations to widely used benchmarks (MNLI and ANLI) A large decrease in model accuracy indicates that the original dataset provides a proper challenge to the models' reasoning capabilities.
  arXiv  Detail & Related papers  (2021-04-10T12:28:07Z)
• Improving Translation Robustness with Visual Cues and Error Correction [58.97421756225425]
  We introduce the idea of visual context to improve translation robustness against noisy texts. We also propose a novel error correction training regime by treating error correction as an auxiliary task.
  arXiv  Detail & Related papers  (2021-03-12T15:31:34Z)
• Sentence Boundary Augmentation For Neural Machine Translation Robustness [11.290581889247983]
  We show that sentence boundary segmentation has the largest impact on quality, and we develop a simple data augmentation strategy to improve segmentation robustness. We show that sentence boundary segmentation has the largest impact on quality, and we develop a simple data augmentation strategy to improve segmentation robustness.
  arXiv  Detail & Related papers  (2020-10-21T16:44:48Z)
• BERT-ATTACK: Adversarial Attack Against BERT Using BERT [77.82947768158132]
  Adrial attacks for discrete data (such as texts) are more challenging than continuous data (such as images) We propose textbfBERT-Attack, a high-quality and effective method to generate adversarial samples. Our method outperforms state-of-the-art attack strategies in both success rate and perturb percentage.
  arXiv  Detail & Related papers  (2020-04-21T13:30:02Z)
• Adv-BERT: BERT is not robust on misspellings! Generating nature adversarial samples on BERT [95.88293021131035]
  It is unclear, however, how the models will perform in realistic scenarios where textitnatural rather than malicious adversarial instances often exist. This work systematically explores the robustness of BERT, the state-of-the-art Transformer-style model in NLP, in dealing with noisy data.
  arXiv  Detail & Related papers  (2020-02-27T22:07:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.