Contrastive Perplexity for Controlled Generation: An Application in Detoxifying Large Language Models

• URL: http://arxiv.org/abs/2401.08491v3
• Date: Fri, 30 May 2025 09:37:59 GMT
• Title: Contrastive Perplexity for Controlled Generation: An Application in Detoxifying Large Language Models
• Authors: Tassilo Klein, Moin Nabi,
• Abstract summary: The generation of toxic content by large language models (LLMs) remains a critical challenge for the safe deployment of language technology.<n>We propose a novel framework for implicit knowledge editing and controlled text generation by fine-tuning LLMs with a prototype-based contrastive perplexity objective.
• Score: 21.341749351654453
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The generation of toxic content by large language models (LLMs) remains a critical challenge for the safe deployment of language technology. We propose a novel framework for implicit knowledge editing and controlled text generation by fine-tuning LLMs with a prototype-based contrastive perplexity objective. Central to our method is the construction of hard negatives - toxic outputs that are generated through adversarial paraphrasing to be semantically similar and model probability to their non-toxic counterparts. By training on these challenging and realistic pairs, our approach ensures robust and stable contrastive optimization. Experimental results in the domain of detoxification demonstrate that our method significantly reduces toxic generation while maintaining strong performance on downstream tasks such as commonsense reasoning and reading comprehension. Our findings highlight the effectiveness of exploiting hard negatives for attribute-aware fine-tuning.

Related papers

• Text Detoxification: Data Efficiency, Semantic Preservation and Model Generalization [23.328207651816957]
  The dissemination of toxic content on social media poses a serious threat to online environments and public discourse.<n>Existing approaches often struggle to simultaneously achieve strong detoxification performance, semantic preservation, and to out-of-distribution data.<n>We propose a two-stage training framework that jointly optimize for data efficiency, semantic preservation, and model generalization.
  arXiv  Detail & Related papers  (2025-06-23T05:48:10Z)
• Adaptive Detoxification: Safeguarding General Capabilities of LLMs through Toxicity-Aware Knowledge Editing [49.85884082568318]
  ToxEdit is a toxicity-aware knowledge editing approach.<n>It dynamically detects toxic activation patterns during forward propagation.<n>It then routes computations through adaptive inter-layer pathways to mitigate toxicity effectively.
  arXiv  Detail & Related papers  (2025-05-28T12:37:06Z)
• Prompting Forgetting: Unlearning in GANs via Textual Guidance [4.3562145620596215]
  We propose Text-to-Unlearn, a novel framework that selectively unlearns concepts from pre-trained GANs using only text prompts. Our approach guides the unlearning process without requiring additional datasets or supervised fine-tuning. To our knowledge, Text-to-Unlearn is the first cross-modal unlearning framework for GANs.
  arXiv  Detail & Related papers  (2025-04-01T22:18:40Z)
• LLM Post-Training: A Deep Dive into Reasoning Large Language Models [131.10969986056]
  Large Language Models (LLMs) have transformed the natural language processing landscape and brought to life diverse applications. Post-training methods enable LLMs to refine their knowledge, improve reasoning, enhance factual accuracy, and align more effectively with user intents and ethical considerations.
  arXiv  Detail & Related papers  (2025-02-28T18:59:54Z)
• Semantic Consistency Regularization with Large Language Models for Semi-supervised Sentiment Analysis [20.503153899462323]
  We propose a framework for semi-supervised sentiment analysis. We introduce two prompting strategies to semantically enhance unlabeled text. Experiments show our method achieves remarkable performance over prior semi-supervised methods.
  arXiv  Detail & Related papers  (2025-01-29T12:03:11Z)
• Learning-to-Defer for Extractive Question Answering [3.6787328174619254]
  We introduce an adapted two-stage Learning-to-Defer mechanism that enhances decision-making by enabling selective deference to human experts or larger models without retraining language models in the context of question-answering. Our results demonstrate that deferring a minimal number of queries allows the smaller model to achieve performance comparable to their larger counterparts while preserving computing efficiency.
  arXiv  Detail & Related papers  (2024-10-21T08:21:00Z)
• Large Language Models can be Strong Self-Detoxifiers [82.6594169242814]
  Self-disciplined Autoregressive Sampling (SASA) is a lightweight controlled decoding algorithm for toxicity reduction of large language models (LLMs) SASA tracks the margin of the current output to steer the generation away from the toxic subspace, by adjusting the autoregressive sampling strategy. evaluated on LLMs of different scale and nature, namely Llama-3.1-Instruct (8B), Llama-2 (7B), and GPT2-L models with the RealToxicityPrompts, BOLD, and AttaQ benchmarks.
  arXiv  Detail & Related papers  (2024-10-04T17:45:15Z)
• Self-training Large Language Models through Knowledge Detection [26.831873737733737]
  Large language models (LLMs) often necessitate extensive labeled datasets and training compute to achieve impressive performance across downstream tasks. This paper explores a self-training paradigm, where the LLM autonomously curates its own labels and selectively trains on unknown data samples. Empirical evaluations demonstrate significant improvements in reducing hallucination in generation across multiple subjects.
  arXiv  Detail & Related papers  (2024-06-17T07:25:09Z)
• ALMol: Aligned Language-Molecule Translation LLMs through Offline Preference Contrastive Optimisation [2.296475290901356]
  We focus on machine language-molecule translation and deploy a novel training approach called contrastive preference optimisation. Our results demonstrate that our models achieve up to a 32% improvement compared to counterpart models.
  arXiv  Detail & Related papers  (2024-05-14T13:59:24Z)
• Debiasing Multimodal Large Language Models [61.6896704217147]
  Large Vision-Language Models (LVLMs) have become indispensable tools in computer vision and natural language processing. Our investigation reveals a noteworthy bias in the generated content, where the output is primarily influenced by the underlying Large Language Models (LLMs) prior to the input image. To rectify these biases and redirect the model's focus toward vision information, we introduce two simple, training-free strategies.
  arXiv  Detail & Related papers  (2024-03-08T12:35:07Z)
• DPP-Based Adversarial Prompt Searching for Lanugage Models [56.73828162194457]
  Auto-regressive Selective Replacement Ascent (ASRA) is a discrete optimization algorithm that selects prompts based on both quality and similarity with determinantal point process (DPP) Experimental results on six different pre-trained language models demonstrate the efficacy of ASRA for eliciting toxic content.
  arXiv  Detail & Related papers  (2024-03-01T05:28:06Z)
• Unveiling the Implicit Toxicity in Large Language Models [77.90933074675543]
  The open-endedness of large language models (LLMs) combined with their impressive capabilities may lead to new safety issues when being exploited for malicious use. We show that LLMs can generate diverse implicit toxic outputs that are exceptionally difficult to detect via simply zero-shot prompting. We propose a reinforcement learning (RL) based attacking method to further induce the implicit toxicity in LLMs.
  arXiv  Detail & Related papers  (2023-11-29T06:42:36Z)
• Self-Detoxifying Language Models via Toxification Reversal [11.238212967733165]
  Language model detoxification aims to minimize the risk of generating offensive or harmful content in pretrained language models (PLMs) We propose a more lightweight approach that enables the PLM itself to achieve "self-detoxification" Our method is built upon the observation that prepending a negative steering prompt can effectively induce PLMs to generate toxic content.
  arXiv  Detail & Related papers  (2023-10-14T12:51:38Z)
• Are Large Language Models Really Robust to Word-Level Perturbations? [68.60618778027694]
  We propose a novel rational evaluation approach that leverages pre-trained reward models as diagnostic tools. Longer conversations manifest the comprehensive grasp of language models in terms of their proficiency in understanding questions. Our results demonstrate that LLMs frequently exhibit vulnerability to word-level perturbations that are commonplace in daily language usage.
  arXiv  Detail & Related papers  (2023-09-20T09:23:46Z)
• CMD: a framework for Context-aware Model self-Detoxification [22.842468869653818]
  Text detoxification aims to minimize the risk of language models producing toxic content. Existing detoxification methods fail to achieve a decent balance between detoxification effectiveness and generation quality. We introduce a Context-aware Model self-Detoxification(CMD) framework that pays attention to both the context and the detoxification process.
  arXiv  Detail & Related papers  (2023-08-16T11:50:38Z)
• Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategies [104.32199881187607]
  Large language models (LLMs) have demonstrated remarkable performance across a wide array of NLP tasks. A promising approach to rectify these flaws is self-correction, where the LLM itself is prompted or guided to fix problems in its own output. This paper presents a comprehensive review of this emerging class of techniques.
  arXiv  Detail & Related papers  (2023-08-06T18:38:52Z)
• Large Language Models with Controllable Working Memory [64.71038763708161]
  Large language models (LLMs) have led to a series of breakthroughs in natural language processing (NLP) What further sets these models apart is the massive amounts of world knowledge they internalize during pretraining. How the model's world knowledge interacts with the factual information presented in the context remains under explored.
  arXiv  Detail & Related papers  (2022-11-09T18:58:29Z)
• Language Detoxification with Attribute-Discriminative Latent Space [59.167432249229584]
  Transformer-based Language Models (LMs) have achieved impressive results on natural language understanding tasks. They can also generate toxic text such as insults, threats, and profanity, limiting their real-world applications. We propose an effective yet efficient method for language detoxification using an attribute-discriminative latent space.
  arXiv  Detail & Related papers  (2022-10-19T06:54:42Z)
• Unified Detoxifying and Debiasing in Language Generation via Inference-time Adaptive Optimization [32.50246008433889]
  Pre-trained language models (PLMs) have prospered in various natural language generation (NLG) tasks due to their ability to generate fairly fluent text. These models are observed to capture and reproduce harmful contents in training corpora, typically toxic language and social biases, raising severe moral issues. We propose the first unified framework of detoxifying and debiasing called UDDIA, which jointly formalizes these two problems as rectifying the output space.
  arXiv  Detail & Related papers  (2022-10-10T08:45:25Z)
• Leashing the Inner Demons: Self-Detoxification for Language Models [13.576289320208511]
  Language models (LMs) can reproduce (or amplify) toxic language seen during training. We analyze the impact of prompts, decoding strategies and training corpora on the output. We propose a simple yet effective method for language models to "detoxify" themselves without an additional large corpus or external discriminator.
  arXiv  Detail & Related papers  (2022-03-06T23:55:12Z)
• Reward Modeling for Mitigating Toxicity in Transformer-based Language Models [0.0]
  Transformer-based language models are able to generate fluent text and be efficiently adapted across various natural language generation tasks. Language models that are pretrained on large unlabeled web text corpora have been shown to suffer from degenerating toxic content and social bias behaviors. We propose Reinforce-Detoxify; A reinforcement learning-based method for mitigating toxicity in language models.
  arXiv  Detail & Related papers  (2022-02-19T19:26:22Z)
• A Simple but Tough-to-Beat Data Augmentation Approach for Natural Language Understanding and Generation [53.8171136907856]
  We introduce a set of simple yet effective data augmentation strategies dubbed cutoff. cutoff relies on sampling consistency and thus adds little computational overhead. cutoff consistently outperforms adversarial training and achieves state-of-the-art results on the IWSLT2014 German-English dataset.
  arXiv  Detail & Related papers  (2020-09-29T07:08:35Z)
• RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models [93.151822563361]
  Pretrained neural language models (LMs) are prone to generating racist, sexist, or otherwise toxic language which hinders their safe deployment. We investigate the extent to which pretrained LMs can be prompted to generate toxic language, and the effectiveness of controllable text generation algorithms at preventing such toxic degeneration.
  arXiv  Detail & Related papers  (2020-09-24T03:17:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.