Exploring the Limits of Domain-Adaptive Training for Detoxifying Large-Scale Language Models

• URL: http://arxiv.org/abs/2202.04173v3
• Date: Fri, 21 Oct 2022 23:01:28 GMT
• Title: Exploring the Limits of Domain-Adaptive Training for Detoxifying Large-Scale Language Models
• Authors: Boxin Wang, Wei Ping, Chaowei Xiao, Peng Xu, Mostofa Patwary, Mohammad Shoeybi, Bo Li, Anima Anandkumar, Bryan Catanzaro
• Abstract summary: We explore domain-adaptive training to reduce the toxicity of language models. For the training corpus, we propose to leverage the generative power of LMs. We then comprehensively study LMs with parameter sizes ranging from 126M up to 530B, a scale that has never been studied before.
• Score: 84.30718841659531
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Pre-trained language models (LMs) are shown to easily generate toxic language. In this work, we systematically explore domain-adaptive training to reduce the toxicity of language models. We conduct this study on three dimensions: training corpus, model size, and parameter efficiency. For the training corpus, we propose to leverage the generative power of LMs and generate nontoxic datasets for domain-adaptive training, which mitigates the exposure bias and is shown to be more data-efficient than using a curated pre-training corpus. We demonstrate that the self-generation method consistently outperforms the existing baselines across various model sizes on both automatic and human evaluations, even when it uses a 1/3 smaller training corpus. We then comprehensively study detoxifying LMs with parameter sizes ranging from 126M up to 530B (3x larger than GPT-3), a scale that has never been studied before. We find that i) large LMs have similar toxicity levels as smaller ones given the same pre-training corpus, and ii) large LMs require more endeavor to detoxify. We also explore parameter-efficient training methods for detoxification. We demonstrate that adding and training adapter-only layers in LMs not only saves a lot of parameters but also achieves a better trade-off between toxicity and perplexity than whole model adaptation for the large-scale models.

Related papers

• Training Compute-Optimal Protein Language Models [48.79416103951816]
  Most protein language models are trained with extensive compute resources until performance gains plateau. Our investigation is grounded in a massive dataset consisting of 939 million protein sequences. We trained over 300 models ranging from 3.5 million to 10.7 billion parameters on 5 to 200 billion unique tokens.
  arXiv  Detail & Related papers  (2024-11-04T14:58:37Z)
• Large Language Models can be Strong Self-Detoxifiers [82.6594169242814]
  Self-disciplined Autoregressive Sampling (SASA) is a lightweight controlled decoding algorithm for toxicity reduction of large language models (LLMs) SASA tracks the margin of the current output to steer the generation away from the toxic subspace, by adjusting the autoregressive sampling strategy. evaluated on LLMs of different scale and nature, namely Llama-3.1-Instruct (8B), Llama-2 (7B), and GPT2-L models with the RealToxicityPrompts, BOLD, and AttaQ benchmarks.
  arXiv  Detail & Related papers  (2024-10-04T17:45:15Z)
• Are Protein Language Models Compute Optimal? [0.0]
  We investigate the optimal ratio between model parameters and training tokens within a fixed compute budget. Our study reveals that pLM sizes scale sublinearly with compute budget, showing diminishing returns in performance as model size increases. This work paves the way towards more compute-efficient pLMs, democratizing their training and practical application in computational biology.
  arXiv  Detail & Related papers  (2024-06-11T13:32:11Z)
• Emergent Abilities in Reduced-Scale Generative Language Models [10.51168925267033]
  Large language models can solve new tasks without task-specific fine-tuning. This ability is considered an emergent ability and is primarily seen in large language models with billions of parameters. This study investigates if such emergent properties are strictly tied to model size or can be demonstrated by smaller models trained on reduced-scale data.
  arXiv  Detail & Related papers  (2024-04-02T18:00:28Z)
• Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models [115.501751261878]
  Fine-tuning language models(LMs) on human-generated data remains a prevalent practice. We investigate whether we can go beyond human data on tasks where we have access to scalar feedback. We find that ReST$EM$ scales favorably with model size and significantly surpasses fine-tuning only on human data.
  arXiv  Detail & Related papers  (2023-12-11T18:17:43Z)
• Large GPT-like Models are Bad Babies: A Closer Look at the Relationship between Linguistic Competence and Psycholinguistic Measures [25.210837736795565]
  We train a series of GPT-like language models of different sizes on the strict version of the BabyLM pretraining corpus. We find a positive correlation between LM size and performance on all three challenge tasks, with different preferences for model width and depth in each of the tasks. This suggests that modelling processing effort and linguistic competence may require an approach different from training GPT-like LMs on a developmentally plausible corpus.
  arXiv  Detail & Related papers  (2023-11-08T09:26:27Z)
• Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning [52.29522018586365]
  We study structured pruning as an effective means to develop smaller LLMs from pre-trained, larger models. Our approach employs two key techniques: (1) targeted structured pruning, which prunes a larger model to a specified target shape by removing layers, heads, and intermediate and hidden dimensions in an end-to-end manner, and (2) dynamic batch loading, which dynamically updates the composition of sampled data in each training batch based on varying losses across different domains.
  arXiv  Detail & Related papers  (2023-10-10T15:13:30Z)
• Honey, I Shrunk the Language: Language Model Behavior at Reduced Scale [5.759319006531332]
  We show the benefits of pre-training with masked language modeling (MLM) objective in models as small as 1.25M parameters. We examine downscaling effects, extending scaling laws to models as small as 1M parameters.
  arXiv  Detail & Related papers  (2023-05-26T21:22:10Z)
• SPDF: Sparse Pre-training and Dense Fine-tuning for Large Language Models [4.114555639014612]
  We show the benefits of using unstructured weight sparsity to train only a subset of weights during pre-training. We demonstrate that we can induce up to 75% sparsity into a 1.3B parameter GPT-3 XL model resulting in a 2.5x reduction in pre-training FLOPs.
  arXiv  Detail & Related papers  (2023-03-18T17:56:01Z)
• METRO: Efficient Denoising Pretraining of Large Scale Autoencoding Language Models with Model Generated Signals [151.3601429216877]
  We present an efficient method of pretraining large-scale autoencoding language models using training signals generated by an auxiliary model. We propose a recipe, namely "Model generated dEnoising TRaining Objective" (METRO) The resultant models, METRO-LM, consisting of up to 5.4 billion parameters, achieve new state-of-the-art on the GLUE, SuperGLUE, and SQuAD benchmarks.
  arXiv  Detail & Related papers  (2022-04-13T21:39:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.