SUV: Scalable Large Language Model Copyright Compliance with Regularized Selective Unlearning

• URL: http://arxiv.org/abs/2503.22948v1
• Date: Sat, 29 Mar 2025 02:33:26 GMT
• Title: SUV: Scalable Large Language Model Copyright Compliance with Regularized Selective Unlearning
• Authors: Tianyang Xu, Xiaoze Liu, Feijie Wu, Xiaoqian Wang, Jing Gao,
• Abstract summary: SUV is a selective unlearning framework designed to prevent Large Language Models from memorizing copyrighted content.<n>We replace verbatim copyrighted content with plausible and coherent alternatives.<n>We validate our approach using a large-scale dataset of 500 famous books.
• Score: 22.76025238218253
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Models (LLMs) have transformed natural language processing by learning from massive datasets, yet this rapid progress has also drawn legal scrutiny, as the ability to unintentionally generate copyrighted content has already prompted several prominent lawsuits. In this work, we introduce SUV (Selective Unlearning for Verbatim data), a selective unlearning framework designed to prevent LLM from memorizing copyrighted content while preserving its overall utility. In detail, the proposed method constructs a dataset that captures instances of copyrighted infringement cases by the targeted LLM. With the dataset, we unlearn the content from the LLM by means of Direct Preference Optimization (DPO), which replaces the verbatim copyrighted content with plausible and coherent alternatives. Since DPO may hinder the LLM's performance in other unrelated tasks, we integrate gradient projection and Fisher information regularization to mitigate the degradation. We validate our approach using a large-scale dataset of 500 famous books (predominantly copyrighted works) and demonstrate that SUV significantly reduces verbatim memorization with negligible impact on the performance on unrelated tasks. Extensive experiments on both our dataset and public benchmarks confirm the scalability and efficacy of our approach, offering a promising solution for mitigating copyright risks in real-world LLM applications.

Related papers

• LLM-Lasso: A Robust Framework for Domain-Informed Feature Selection and Regularization [59.75242204923353]
  We introduce LLM-Lasso, a framework that leverages large language models (LLMs) to guide feature selection in Lasso regression.<n>LLMs generate penalty factors for each feature, which are converted into weights for the Lasso penalty using a simple, tunable model.<n>Features identified as more relevant by the LLM receive lower penalties, increasing their likelihood of being retained in the final model.
  arXiv  Detail & Related papers  (2025-02-15T02:55:22Z)
• Investigating the Feasibility of Mitigating Potential Copyright Infringement via Large Language Model Unlearning [0.0]
  Pre-trained Large Language Models (LLMs) have demonstrated remarkable capabilities but also pose risks by learning and generating copyrighted material.<n>We propose Stable Sequential Unlearning (SSU), a novel framework designed to unlearn copyrighted content from LLMs over multiple time steps.<n>SSU sometimes achieves an effective trade-off between unlearning efficacy and general-purpose language abilities, outperforming existing baselines, but it's not a cure-all for unlearning copyrighted material.
  arXiv  Detail & Related papers  (2024-12-16T20:01:06Z)
• Can Watermarking Large Language Models Prevent Copyrighted Text Generation and Hide Training Data? [62.72729485995075]
  We investigate the effectiveness of watermarking as a deterrent against the generation of copyrighted texts.<n>We find that watermarking adversely affects the success rate of Membership Inference Attacks (MIAs)<n>We propose an adaptive technique to improve the success rate of a recent MIA under watermarking.
  arXiv  Detail & Related papers  (2024-07-24T16:53:09Z)
• Evaluating Copyright Takedown Methods for Language Models [100.38129820325497]
  Language models (LMs) derive their capabilities from extensive training on diverse data, including potentially copyrighted material. This paper introduces the first evaluation of the feasibility and side effects of copyright takedowns for LMs. We examine several strategies, including adding system prompts, decoding-time filtering interventions, and unlearning approaches.
  arXiv  Detail & Related papers  (2024-06-26T18:09:46Z)
• Avoiding Copyright Infringement via Large Language Model Unlearning [24.050754626661124]
  We propose a novel framework designed to unlearn copyrighted content from Large Language Models over multiple time steps.<n>We improve unlearning efficacy by introducing random labeling loss and ensuring the model retains its general-purpose knowledge.<n> Experimental results show that SSU achieves an effective trade-off between unlearning efficacy and general-purpose language abilities.
  arXiv  Detail & Related papers  (2024-06-16T14:12:37Z)
• One Token Can Help! Learning Scalable and Pluggable Virtual Tokens for Retrieval-Augmented Large Language Models [67.49462724595445]
  Retrieval-augmented generation (RAG) is a promising way to improve large language models (LLMs)<n>We propose a novel method that involves learning scalable and pluggable virtual tokens for RAG.
  arXiv  Detail & Related papers  (2024-05-30T03:44:54Z)
• Purifying Large Language Models by Ensembling a Small Language Model [39.57304668057076]
  We propose a simple and easily implementable method for purifying LLMs from the negative effects caused by uncurated data. We empirically confirm the efficacy of ensembling LLMs with benign and small language models (SLMs)
  arXiv  Detail & Related papers  (2024-02-19T14:00:39Z)
• UNDIAL: Self-Distillation with Adjusted Logits for Robust Unlearning in Large Language Models [12.45822383965784]
  We introduce UnDIAL (Unlearning via Self-Distillation on Adjusted Logits), a novel and robust unlearning method. Our approach leverages self-distillation to adjust logits and selectively reduce the influence of targeted tokens.
  arXiv  Detail & Related papers  (2024-02-15T16:21:14Z)
• SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore [159.21914121143885]
  We present SILO, a new language model that manages this risk-performance tradeoff during inference. SILO is built by (1) training a parametric LM on Open License Corpus (OLC), a new corpus we curate with 228B tokens of public domain and permissively licensed text. Access to the datastore greatly improves out of domain performance, closing 90% of the performance gap with an LM trained on the Pile.
  arXiv  Detail & Related papers  (2023-08-08T17:58:15Z)
• Are You Copying My Model? Protecting the Copyright of Large Language Models for EaaS via Backdoor Watermark [58.60940048748815]
  Companies have begun to offer Embedding as a Service (E) based on large language models (LLMs) E is vulnerable to model extraction attacks, which can cause significant losses for the owners of LLMs. We propose an Embedding Watermark method called EmbMarker that implants backdoors on embeddings.
  arXiv  Detail & Related papers  (2023-05-17T08:28:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.