Alternate Preference Optimization for Unlearning Factual Knowledge in Large Language Models

• URL: http://arxiv.org/abs/2409.13474v3
• Date: Tue, 17 Dec 2024 17:45:07 GMT
• Title: Alternate Preference Optimization for Unlearning Factual Knowledge in Large Language Models
• Authors: Anmol Mekala, Vineeth Dorna, Shreya Dubey, Abhishek Lalwani, David Koleczek, Mukund Rungta, Sadid Hasan, Elita Lobo,
• Abstract summary: Machine unlearning aims to efficiently eliminate the influence of specific training data, known as the forget set, from the model. Existing unlearning methods rely solely on negative feedback to suppress responses related to the forget set. We propose a novel approach called Alternate Preference Optimization (AltPO), which combines negative feedback with in-domain positive feedback on the forget set.
• Score: 2.0962367975513496
• License:
• Abstract: Machine unlearning aims to efficiently eliminate the influence of specific training data, known as the forget set, from the model. However, existing unlearning methods for Large Language Models (LLMs) face a critical challenge: they rely solely on negative feedback to suppress responses related to the forget set, which often results in nonsensical or inconsistent outputs, diminishing model utility and posing potential privacy risks. To address this limitation, we propose a novel approach called Alternate Preference Optimization (AltPO), which combines negative feedback with in-domain positive feedback on the forget set. Additionally, we introduce new evaluation metrics to assess the quality of responses related to the forget set. Extensive experiments show that our approach not only enables effective unlearning but also avoids undesirable model behaviors while maintaining overall model performance. Our implementation can be found at https://github.com/molereddy/Alternate-Preference-Optimization.

Related papers

• Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearning [27.991291785091736]
  This work studies the problem of large language model (LLM) unlearning, aiming to remove unwanted data influences. Despite the increasing demand for unlearning, a technically-grounded optimization framework is lacking. We propose a simple yet effective unlearning optimization framework, called SimNPO, showing that simplicity' in removing the reliance on a reference model benefits unlearning.
  arXiv  Detail & Related papers  (2024-10-09T17:58:12Z)
• Self-supervised Preference Optimization: Enhance Your Language Model with Preference Degree Awareness [27.43137305486112]
  We propose a novel Self-supervised Preference Optimization (SPO) framework, which constructs a self-supervised preference degree loss combined with the alignment loss. The results demonstrate that SPO can be seamlessly integrated with existing preference optimization methods to achieve state-of-the-art performance.
  arXiv  Detail & Related papers  (2024-09-26T12:37:26Z)
• Just Say What You Want: Only-prompting Self-rewarding Online Preference Optimization [64.34767799614328]
  Current self-rewarding approaches rely heavily on the discriminator's judgment capabilities. We propose a novel, only-prompting self-rewarding online algorithm that generates preference datasets without relying on judgment capabilities.
  arXiv  Detail & Related papers  (2024-09-26T04:41:08Z)
• Unlearning with Control: Assessing Real-world Utility for Large Language Model Unlearning [97.2995389188179]
  Recent research has begun to approach large language models (LLMs) unlearning via gradient ascent (GA) Despite their simplicity and efficiency, we suggest that GA-based methods face the propensity towards excessive unlearning. We propose several controlling methods that can regulate the extent of excessive unlearning.
  arXiv  Detail & Related papers  (2024-06-13T14:41:00Z)
• Self-Augmented Preference Optimization: Off-Policy Paradigms for Language Model Alignment [104.18002641195442]
  We introduce Self-Augmented Preference Optimization (SAPO), an effective and scalable training paradigm that does not require existing paired data. Building on the self-play concept, which autonomously generates negative responses, we further incorporate an off-policy learning pipeline to enhance data exploration and exploitation.
  arXiv  Detail & Related papers  (2024-05-31T14:21:04Z)
• Challenging Forgets: Unveiling the Worst-Case Forget Sets in Machine Unlearning [9.998859702421417]
  Machine unlearning (MU) aims to eliminate the influence of chosen data points on model performance. Despite various MU methods for data influence erasure, evaluations have largely focused on random data forgetting. We propose identifying the data subset that presents the most significant challenge for influence erasure, pinpointing the worst-case forget set.
  arXiv  Detail & Related papers  (2024-03-12T06:50:32Z)
• RLVF: Learning from Verbal Feedback without Overgeneralization [94.19501420241188]
  We study the problem of incorporating verbal feedback without such overgeneralization. We develop a new method Contextualized Critiques with Constrained Preference Optimization (C3PO) Our approach effectively applies verbal feedback to relevant scenarios while preserving existing behaviors for other contexts.
  arXiv  Detail & Related papers  (2024-02-16T18:50:24Z)
• Netflix and Forget: Efficient and Exact Machine Unlearning from Bi-linear Recommendations [15.789980605221672]
  This paper focuses on simple but widely deployed bi-linear models for recommendations based on matrix completion. We develop Unlearn-ALS by making a few key modifications to the fine-tuning procedure under Alternating Least Squares. We show that Unlearn-ALS is consistent with retraining without emphany model degradation and exhibits rapid convergence.
  arXiv  Detail & Related papers  (2023-02-13T20:27:45Z)
• Rethinking Missing Data: Aleatoric Uncertainty-Aware Recommendation [59.500347564280204]
  We propose a new Aleatoric Uncertainty-aware Recommendation (AUR) framework. AUR consists of a new uncertainty estimator along with a normal recommender model. As the chance of mislabeling reflects the potential of a pair, AUR makes recommendations according to the uncertainty.
  arXiv  Detail & Related papers  (2022-09-22T04:32:51Z)
• WSLRec: Weakly Supervised Learning for Neural Sequential Recommendation Models [24.455665093145818]
  We propose a novel model-agnostic training approach called WSLRec, which adopts a three-stage framework: pre-training, top-$k$ mining, intrinsic and fine-tuning. WSLRec resolves the incompleteness problem by pre-training models on extra weak supervisions from model-free methods like BR and ItemCF, while resolving the inaccuracy problem by leveraging the top-$k$ mining to screen out reliable user-item relevance from weak supervisions for fine-tuning.
  arXiv  Detail & Related papers  (2022-02-28T08:55:12Z)
• Machine Unlearning of Features and Labels [72.81914952849334]
  We propose first scenarios for unlearning and labels in machine learning models. Our approach builds on the concept of influence functions and realizes unlearning through closed-form updates of model parameters.
  arXiv  Detail & Related papers  (2021-08-26T04:42:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.