Scrub It Out! Erasing Sensitive Memorization in Code Language Models via Machine Unlearning

• URL: http://arxiv.org/abs/2509.13755v1
• Date: Wed, 17 Sep 2025 07:12:35 GMT
• Title: Scrub It Out! Erasing Sensitive Memorization in Code Language Models via Machine Unlearning
• Abstract summary: Code Language Models (CLMs) exhibit unintended memorization of sensitive training data, enabling verbatim reproduction of confidential information when specifically prompted.<n>We introduce CodeEraser, an advanced variant that selectively unlearns sensitive memorized segments in code while preserving the structural integrity and functional correctness of the surrounding code.
• Score: 50.45435841411193
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While Code Language Models (CLMs) have demonstrated superior performance in software engineering tasks such as code generation and summarization, recent empirical studies reveal a critical privacy vulnerability: these models exhibit unintended memorization of sensitive training data, enabling verbatim reproduction of confidential information when specifically prompted. To address this issue, several approaches, including training data de-duplication and differential privacy augmentation, have been proposed. However, these methods require full-model retraining for deployed CLMs, which incurs substantial computational costs. In this paper, we aim to answer the following research question: Can sensitive information memorized by CLMs be erased effectively and efficiently? We conduct a pioneering investigation into erasing sensitive memorization in CLMs through machine unlearning - a post-hoc modification method that removes specific information from trained models without requiring full retraining. Specifically, we first quantify the memorization risks of sensitive data within CLM training datasets and curate a high-risk dataset of 50,000 sensitive memorized samples as unlearning targets. We study two widely used gradient ascent-based unlearning approaches: the vanilla and constraint-based methods, and introduce CodeEraser, an advanced variant that selectively unlearns sensitive memorized segments in code while preserving the structural integrity and functional correctness of the surrounding code. Extensive experiments on three families of CLMs, i.e., CodeParrot, CodeGen-Mono, and Qwen2.5-Coder, validate the effectiveness and efficiency of CodeEraser in erasing targeted sensitive memorization while maintaining model utility.

Related papers

• Towards Privacy-Preserving Code Generation: Differentially Private Code Language Models [2.4216414826638353]
  This study systematically evaluates the effectiveness of Differential Privacy (DP) in CodeLLMs.<n>DP substantially reduces memorization in CodeLLMs across all the tested snippet types.<n>DP slightly increases perplexity but preserves, and can even enhance, the code generation capabilities of CodeLLMs.
  arXiv  Detail & Related papers  (2025-12-12T11:31:13Z)
• LLM Unlearning on Noisy Forget Sets: A Study of Incomplete, Rewritten, and Watermarked Data [69.5099112089508]
  Large language models (LLMs) exhibit remarkable generative capabilities but raise ethical and security concerns by memorizing sensitive data.<n>This work presents the first study of unlearning under perturbed or low-fidelity forget data, referred to as noisy forget sets.<n>We find that unlearning remains surprisingly robust to perturbations, provided that core semantic signals are preserved.
  arXiv  Detail & Related papers  (2025-10-10T05:10:49Z)
• Large Language Model Unlearning for Source Code [65.42425213605114]
  PROD is a novel unlearning approach that enables LLMs to forget undesired code content while preserving their code generation capabilities.<n>Our evaluation demonstrates that PROD achieves superior balance between forget quality and model utility compared to existing unlearning approaches.
  arXiv  Detail & Related papers  (2025-06-20T16:27:59Z)
• Mitigating Sensitive Information Leakage in LLMs4Code through Machine Unlearning [5.974072640812041]
  Large Language Models for Code (LLMs4Code) excel at code generation tasks, yielding promise to release developers from huge software development burdens.<n>These models have been shown to suffer from the significant privacy risks due to the potential leakage of sensitive information embedded during training.<n>Machine unlearning has emerged as a promising solution by enabling models to "forget" sensitive information without full retraining.
  arXiv  Detail & Related papers  (2025-02-09T01:50:34Z)
• Decoding Secret Memorization in Code LLMs Through Token-Level Characterization [6.92858396995673]
  Code Large Language Models (LLMs) have demonstrated remarkable capabilities in generating, understanding, and manipulating programming code.<n>LLMs inadvertently lead to the memorization of sensitive information, posing severe privacy risks.<n>We present a novel approach to characterize real and fake secrets generated by Code LLMs based on token probabilities.
  arXiv  Detail & Related papers  (2024-10-11T14:39:24Z)
• CodeUnlearn: Amortized Zero-Shot Machine Unlearning in Language Models Using Discrete Concept [5.345828824625758]
  We propose a novel amortized unlearning approach using codebook features and Sparse Autoencoders (SAEs) By leveraging a bottleneck to decompose the activation space and regulate information flow, our method efficiently unlearns targeted information while preserving the model's performance on unrelated data.
  arXiv  Detail & Related papers  (2024-10-08T10:26:22Z)
• Learn while Unlearn: An Iterative Unlearning Framework for Generative Language Models [52.40798352740857]
  We introduce the Iterative Contrastive Unlearning (ICU) framework, which consists of three core components.<n>A Knowledge Unlearning Induction module targets specific knowledge for removal using an unlearning loss.<n>A Contrastive Learning Enhancement module preserves the model's expressive capabilities against the pure unlearning goal.<n>An Iterative Unlearning Refinement module dynamically adjusts the unlearning process through ongoing evaluation and updates.
  arXiv  Detail & Related papers  (2024-07-25T07:09:35Z)
• REVS: Unlearning Sensitive Information in Language Models via Rank Editing in the Vocabulary Space [40.25037054636284]
  Language models (LMs) risk inadvertently memorizing and divulging sensitive or personally identifiable information (PII) seen in training data, causing privacy concerns.<n>We propose REVS, a novel non-gradient-based method for unlearning sensitive information from LMs.
  arXiv  Detail & Related papers  (2024-06-13T17:02:32Z)
• The Frontier of Data Erasure: Machine Unlearning for Large Language Models [56.26002631481726]
  Large Language Models (LLMs) are foundational to AI advancements. LLMs pose risks by potentially memorizing and disseminating sensitive, biased, or copyrighted information. Machine unlearning emerges as a cutting-edge solution to mitigate these concerns.
  arXiv  Detail & Related papers  (2024-03-23T09:26:15Z)
• Unlearn What You Want to Forget: Efficient Unlearning for LLMs [92.51670143929056]
  Large language models (LLMs) have achieved significant progress from pre-training on and memorizing a wide range of textual data. This process might suffer from privacy issues and violations of data protection regulations. We propose an efficient unlearning framework that could efficiently update LLMs without having to retrain the whole model after data removals.
  arXiv  Detail & Related papers  (2023-10-31T03:35:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.