Alternate Preference Optimization for Unlearning Factual Knowledge in Large Language Models

• URL: http://arxiv.org/abs/2409.13474v1
• Date: Fri, 20 Sep 2024 13:05:07 GMT
• Title: Alternate Preference Optimization for Unlearning Factual Knowledge in Large Language Models
• Authors: Anmol Mekala, Vineeth Dorna, Shreya Dubey, Abhishek Lalwani, David Koleczek, Mukund Rungta, Sadid Hasan, Elita Lobo,
• Abstract summary: Machine unlearning aims to efficiently eliminate the influence of specific training data, known as the forget set, from the model. Existing unlearning methods rely solely on negative feedback to suppress responses related to the forget set. We propose a novel approach called Alternate Preference Optimization (AltPO), which combines negative feedback with in-domain positive feedback on the forget set.
• Score: 2.0962367975513496
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Machine unlearning aims to efficiently eliminate the influence of specific training data, known as the forget set, from the model. However, existing unlearning methods for Large Language Models (LLMs) face a critical challenge: they rely solely on negative feedback to suppress responses related to the forget set, which often results in nonsensical or inconsistent outputs, diminishing model utility and posing potential privacy risks. To address this limitation, we propose a novel approach called Alternate Preference Optimization (AltPO), which combines negative feedback with in-domain positive feedback on the forget set. Additionally, we introduce new evaluation metrics to assess the quality of responses related to the forget set. Extensive experiments show that our approach not only enables effective unlearning but also avoids undesirable model behaviors while maintaining overall model performance.

Related papers

• Negative-Prompt-driven Alignment for Generative Language Model [34.191590966148816]
  We propose NEgative-prompt-driven AlignmenT to guide language models away from undesirable behaviors. NEAT explicitly penalizes the model for producing harmful outputs, guiding it not only toward desirable behaviors but also steering it away from generating undesirable, biased responses. Extensive experiments validate NEAT's effectiveness in significantly enhancing language models' alignment with human values and preferences.
  arXiv  Detail & Related papers  (2024-10-16T03:30:09Z)
• Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearning [27.991291785091736]
  We address the problem of large language model (LLM) unlearning, aiming to remove unwanted data influences and associated model capabilities. We propose a simple yet effective unlearning optimization framework, called SimNPO, showing that'simplicity' in removing the reliance on a reference model benefits unlearning.
  arXiv  Detail & Related papers  (2024-10-09T17:58:12Z)
• Just Say What You Want: Only-prompting Self-rewarding Online Preference Optimization [64.34767799614328]
  Current self-rewarding approaches rely heavily on the discriminator's judgment capabilities. We propose a novel, only-prompting self-rewarding online algorithm that generates preference datasets without relying on judgment capabilities.
  arXiv  Detail & Related papers  (2024-09-26T04:41:08Z)
• Learn while Unlearn: An Iterative Unlearning Framework for Generative Language Models [49.043599241803825]
  Iterative Contrastive Unlearning (ICU) framework consists of three core components. A Knowledge Unlearning Induction module removes specific knowledge through an unlearning loss. A Contrastive Learning Enhancement module to preserve the model's expressive capabilities against the pure unlearning goal. And an Iterative Unlearning Refinement module that dynamically assess the unlearning extent on specific data pieces and make iterative update.
  arXiv  Detail & Related papers  (2024-07-25T07:09:35Z)
• Learning to Unlearn for Robust Machine Unlearning [6.488418950340473]
  We introduce a novel Learning-to-Unlearn (LTU) framework to optimize the unlearning process. LTU includes a meta-optimization scheme that facilitates models to effectively preserve generalizable knowledge. We also introduce a Gradient Harmonization strategy to align the optimization trajectories for remembering and forgetting.
  arXiv  Detail & Related papers  (2024-07-15T07:36:00Z)
• Unlearning with Control: Assessing Real-world Utility for Large Language Model Unlearning [97.2995389188179]
  Recent research has begun to approach large language models (LLMs) unlearning via gradient ascent (GA) Despite their simplicity and efficiency, we suggest that GA-based methods face the propensity towards excessive unlearning. We propose several controlling methods that can regulate the extent of excessive unlearning.
  arXiv  Detail & Related papers  (2024-06-13T14:41:00Z)
• RLVF: Learning from Verbal Feedback without Overgeneralization [94.19501420241188]
  We study the problem of incorporating verbal feedback without such overgeneralization. We develop a new method Contextualized Critiques with Constrained Preference Optimization (C3PO) Our approach effectively applies verbal feedback to relevant scenarios while preserving existing behaviors for other contexts.
  arXiv  Detail & Related papers  (2024-02-16T18:50:24Z)
• Active learning with biased non-response to label requests [5.940553820027303]
  Non-response to label requests can impact active learning's effectiveness in real-world contexts. We conceptualise this degradation by considering the type of non-response present in the data. We propose a cost-based correction to the sampling strategy to mitigate the impact of biased non-response.
  arXiv  Detail & Related papers  (2023-12-13T14:01:58Z)
• Re-thinking Data Availablity Attacks Against Deep Neural Networks [53.64624167867274]
  In this paper, we re-examine the concept of unlearnable examples and discern that the existing robust error-minimizing noise presents an inaccurate optimization objective. We introduce a novel optimization paradigm that yields improved protection results with reduced computational time requirements.
  arXiv  Detail & Related papers  (2023-05-18T04:03:51Z)
• Rethinking Missing Data: Aleatoric Uncertainty-Aware Recommendation [59.500347564280204]
  We propose a new Aleatoric Uncertainty-aware Recommendation (AUR) framework. AUR consists of a new uncertainty estimator along with a normal recommender model. As the chance of mislabeling reflects the potential of a pair, AUR makes recommendations according to the uncertainty.
  arXiv  Detail & Related papers  (2022-09-22T04:32:51Z)
• WSLRec: Weakly Supervised Learning for Neural Sequential Recommendation Models [24.455665093145818]
  We propose a novel model-agnostic training approach called WSLRec, which adopts a three-stage framework: pre-training, top-$k$ mining, intrinsic and fine-tuning. WSLRec resolves the incompleteness problem by pre-training models on extra weak supervisions from model-free methods like BR and ItemCF, while resolving the inaccuracy problem by leveraging the top-$k$ mining to screen out reliable user-item relevance from weak supervisions for fine-tuning.
  arXiv  Detail & Related papers  (2022-02-28T08:55:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.