Randomized Masked Finetuning: An Efficient Way to Mitigate Memorization of PIIs in LLMs

• URL: http://arxiv.org/abs/2512.03310v1
• Date: Tue, 02 Dec 2025 23:46:42 GMT
• Title: Randomized Masked Finetuning: An Efficient Way to Mitigate Memorization of PIIs in LLMs
• Authors: Kunj Joshi, David A. Smith,
• Abstract summary: We introduce Randomized Masked Fine-Tuning (RMFT), a privacy-preserving fine-tuning technique that reduces memorization while minimizing performance impact.<n>We demonstrate that RMFT achieves an 80.81% reduction in Total Extraction Rate and 80.17% reduction in Seen Extraction Rate compared to baseline fine-tuning.
• Score: 2.9506547907696006
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The current literature on memorization in Natural Language Models, especially Large Language Models (LLMs), poses severe security and privacy risks, as models tend to memorize personally identifying information (PIIs) from training data. We introduce Randomized Masked Fine-Tuning (RMFT), a novel privacy-preserving fine-tuning technique that reduces PII memorization while minimizing performance impact. Using the Enron Email Dataset, we demonstrate that RMFT achieves an 80.81% reduction in Total Extraction Rate and 80.17% reduction in Seen Extraction Rate compared to baseline fine-tuning, outperforming deduplication methods while maintaining only a 5.73% increase in perplexity. We present MaxTER, a Pareto-optimal evaluation framework for assessing privacy-utility tradeoffs, and show the performance of RMFT vs Deduplication by Area Under The Response Curve (AURC) metric.

Related papers

• Assessing and Mitigating Data Memorization Risks in Fine-Tuned Large Language Models [0.0]
  Large Language Models (LLMs) have demonstrated remarkable capabilities across diverse natural language processing tasks.<n>Their tendency to memorize training data poses significant privacy risks, particularly during fine-tuning processes.<n>This paper presents a comprehensive empirical analysis of data in fine-tuned LLMs and introduces a novel multi-layered privacy protection framework.
  arXiv  Detail & Related papers  (2025-08-10T10:26:55Z)
• Investigating Structural Pruning and Recovery Techniques for Compressing Multimodal Large Language Models: An Empirical Study [64.26593350748401]
  Multimodal Large Language Models (MLLMs) demonstrate impressive capabilities.<n>Current parameter reduction techniques primarily involve training MLLMs from Small Language Models (SLMs)<n>We propose to directly compress existing MLLMs through structural pruning combined with efficient recovery training.
  arXiv  Detail & Related papers  (2025-07-28T11:57:52Z)
• SecP-Tuning: Efficient Privacy-Preserving Prompt Tuning for Large Language Models via MPC [45.27141187385785]
  Large Language Models (LLMs) have revolutionized numerous fields, yet their adaptation to specialized tasks in privacy-sensitive domains such as healthcare and finance remains constrained.<n>Secure Multi-party Computation (MPC)-based privacy-preserving machine learning provides theoretical guarantees for the privacy of model parameters and data.<n>We propose SecP-Tuning, the first MPC-based framework designed for efficient, privacy-preserving prompt tuning of LLMs.
  arXiv  Detail & Related papers  (2025-06-18T09:36:57Z)
• Minifinetuning: Low-Data Generation Domain Adaptation through Corrective Self-Distillation [48.52666201053625]
  Minifinetuning reduces the effects of overfitting-induced degeneralization in low-data settings.<n>MFT demonstrates 2-10x more favourable specialization-to-degeneralization ratios than standard finetuning.
  arXiv  Detail & Related papers  (2025-05-30T01:54:12Z)
• Efficient and Private: Memorisation under differentially private parameter-efficient fine-tuning in language models [2.3281513013731145]
  Fine-tuning large language models (LLMs) for specific tasks introduces privacy risks, as models may inadvertently memorise and leak sensitive training data. Differential Privacy (DP) offers a solution to mitigate these risks, but introduces significant computational and performance trade-offs. We show that PEFT methods achieve comparable performance to standard fine-tuning while requiring fewer parameters and significantly reducing privacy leakage.
  arXiv  Detail & Related papers  (2024-11-24T13:17:36Z)
• A deeper look at depth pruning of LLMs [49.30061112976263]
  Large Language Models (LLMs) are resource-intensive to train but more costly to deploy in production. Recent work has attempted to prune blocks of LLMs based on cheap proxies for estimating block importance. We show that adaptive metrics exhibit a trade-off in performance between tasks.
  arXiv  Detail & Related papers  (2024-07-23T08:40:27Z)
• Protecting Privacy Through Approximating Optimal Parameters for Sequence Unlearning in Language Models [37.172662930947446]
  Language models (LMs) are potentially vulnerable to extraction attacks, which represent a significant privacy risk. We propose Privacy Protection via Optimal Parameters (POP), a novel unlearning method that effectively forgets the target token sequences from the pretrained LM. POP exhibits remarkable retention performance post-unlearning across 9 classification and 4 dialogue benchmarks, outperforming the state-of-the-art by a large margin.
  arXiv  Detail & Related papers  (2024-06-20T08:12:49Z)
• MLAE: Masked LoRA Experts for Visual Parameter-Efficient Fine-Tuning [45.93128932828256]
  Masked LoRA Experts (MLAE) is an innovative approach that applies the concept of masking to visual PEFT. Our method incorporates a cellular decomposition strategy that transforms a low-rank matrix into independent rank-1 submatrices. We show that MLAE achieves new state-of-the-art (SOTA) performance with an average accuracy score of 78.8% on the VTAB-1k benchmark and 90.9% on the FGVC benchmark.
  arXiv  Detail & Related papers  (2024-05-29T08:57:23Z)
• Low-rank finetuning for LLMs: A fairness perspective [54.13240282850982]
  Low-rank approximation techniques have become the de facto standard for fine-tuning Large Language Models. This paper investigates the effectiveness of these methods in capturing the shift of fine-tuning datasets from the initial pre-trained data distribution. We show that low-rank fine-tuning inadvertently preserves undesirable biases and toxic behaviors.
  arXiv  Detail & Related papers  (2024-05-28T20:43:53Z)
• DPZero: Private Fine-Tuning of Language Models without Backpropagation [49.365749361283704]
  We introduce DPZero, a novel private zeroth-order algorithm with nearly dimension-independent rates. The memory efficiency of DPZero is demonstrated in privately fine-tuning RoBERTa and OPT on several downstream tasks.
  arXiv  Detail & Related papers  (2023-10-14T18:42:56Z)
• Democratizing LLMs: An Exploration of Cost-Performance Trade-offs in Self-Refined Open-Source Models [53.859446823312126]
  SoTA open source models of varying sizes from 7B - 65B, on average, improve 8.2% from their baseline performance. Strikingly, even models with extremely small memory footprints, such as Vicuna-7B, show a 11.74% improvement overall and up to a 25.39% improvement in high-creativity, open ended tasks.
  arXiv  Detail & Related papers  (2023-10-11T15:56:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.