Unlearning at Scale: Implementing the Right to be Forgotten in Large Language Models

• URL: http://arxiv.org/abs/2508.12220v1
• Date: Sun, 17 Aug 2025 03:29:22 GMT
• Title: Unlearning at Scale: Implementing the Right to be Forgotten in Large Language Models
• Authors: Abdullah X,
• Abstract summary: Our approach treats as a minimal program and logs permicrobatch record.<n>Under pinned stack and deterministic kernels, replaying the training tail yields the same parameters as training retain set.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We study the right to be forgotten (GDPR Art. 17) for large language models and frame unlearning as a reproducible systems problem. Our approach treats training as a deterministic program and logs a minimal per-microbatch record (ordered ID hash, RNG seed, learning-rate value, optimizer-step counter, and accumulation boundary). Under a pinned stack and deterministic kernels, replaying the training tail while filtering only the forget closure yields the same parameters as training on the retain set (bit-identical in the training dtype) when preconditions hold. To meet latency and availability constraints, we add complementary paths: (i) exact reverts of recent steps via micro-checkpoints or dense per-step deltas, (ii) cohort-scoped adapter deletion when the base is frozen, and (iii) a curvature-guided anti-update followed by a short retain-tune, audit-gated with escalation to exact replay. We report storage/latency budgets and a toy artifact validating mechanics; in a controlled run that satisfies the preconditions we demonstrate byte-identical equality of model and optimizer states.

Related papers

• Reasoning Core: A Scalable Procedural Data Generation Suite for Symbolic Pre-training and Post-Training [2.62112541805429]
  Reasoning Core is a scalable suite that procedurally generates verifiable symbolic reasoning data across core formal domains.<n>Each task is paired with an external solver for rigorous verification and admits continuous difficulty control for curriculum design.<n>Experiments show that mixing Reasoning Core data into pre-training improves downstream reasoning while preserving, or slightly improving, language modeling quality.
  arXiv  Detail & Related papers  (2026-03-02T18:59:29Z)
• Learning to Detect Language Model Training Data via Active Reconstruction [65.4791582049743]
  We introduce textbfActive Data Reconstruction Attack (ADRA)<n>ADRA induces a model to reconstruct a given text through training.<n>Our algorithms consistently outperform existing MIAs in detecting pre-training, post-training, and distillation data.
  arXiv  Detail & Related papers  (2026-02-22T03:20:06Z)
• Process-Tensor Tomography of SGD: Measuring Non-Markovian Memory via Back-Flow of Distinguishability [1.078600700827543]
  We build a simple model-agnostic witness of training memory based on emphback-flow of distinguishability.<n>We observe consistent positive back-flow with tight bootstrap confidence intervals, amplification under higher momentum, and more micro-steps.<n>We position this as a principled diagnostic and empirical evidence that practical SGD deviates from the Markov idealization.
  arXiv  Detail & Related papers  (2026-01-23T09:03:25Z)
• Grokked Models are Better Unlearners [5.8757712547216485]
  Starting from grokked checkpoints consistently yields more efficient forgetting.<n>Post-grokking models learn more modular representations with reduced gradient alignment between forget and retain subsets.
  arXiv  Detail & Related papers  (2025-12-03T04:35:49Z)
• Geometric-Disentangelment Unlearning [106.99160454669902]
  gradient ascent on forget samples often harms retained knowledge.<n>We propose the Geometric-disment Unlearning (GU) that decomposes any candidate forget gradient update into tangential and normal components to retain space and executes only the normal component.<n>Our method is plug-and-play and can be attached to existing gradient-based unlearning procedures to mitigate side effects.
  arXiv  Detail & Related papers  (2025-11-21T09:58:25Z)
• PoGO: A Scalable Proof of Useful Work via Quantized Gradient Descent and Merkle Proofs [0.0]
  We present a design called Proof of Gradient Optimization (PoGO) for blockchain consensus.<n>PoGO miners produce verifiable evidence of training large-scale machine-learning models.<n>We provide an empirical cost analysis showing that verification is significantly cheaper than training.
  arXiv  Detail & Related papers  (2025-04-10T08:09:34Z)
• Instance-dependent Early Stopping [57.912273923450726]
  We propose an Instance-dependent Early Stopping (IES) method that adapts the early stopping mechanism from the entire training set to the instance level.<n>IES considers an instance as mastered if the second-order differences of its loss value remain within a small range around zero.<n>IES can reduce backpropagation instances by 10%-50% while maintaining or even slightly improving the test accuracy and transfer learning performance of a model.
  arXiv  Detail & Related papers  (2025-02-11T13:34:09Z)
• Rethinking Classifier Re-Training in Long-Tailed Recognition: A Simple Logits Retargeting Approach [102.0769560460338]
  We develop a simple logits approach (LORT) without the requirement of prior knowledge of the number of samples per class. Our method achieves state-of-the-art performance on various imbalanced datasets, including CIFAR100-LT, ImageNet-LT, and iNaturalist 2018.
  arXiv  Detail & Related papers  (2024-03-01T03:27:08Z)
• Adaptive scheduling for adaptive sampling in POS taggers construction [0.27624021966289597]
  We introduce an adaptive scheduling for adaptive sampling as a novel way of machine learning in the construction of part-of-speech taggers. We analyze the shape of the learning curve geometrically in conjunction with a functional model to increase or decrease it at any time. We also improve the robustness of sampling by paying greater attention to those regions of the training data base subject to a temporary inflation in performance.
  arXiv  Detail & Related papers  (2024-02-04T15:02:17Z)
• Noisy Correspondence Learning with Self-Reinforcing Errors Mitigation [63.180725016463974]
  Cross-modal retrieval relies on well-matched large-scale datasets that are laborious in practice. We introduce a novel noisy correspondence learning framework, namely textbfSelf-textbfReinforcing textbfErrors textbfMitigation (SREM)
  arXiv  Detail & Related papers  (2023-12-27T09:03:43Z)
• AdaNPC: Exploring Non-Parametric Classifier for Test-Time Adaptation [64.9230895853942]
  Domain generalization can be arbitrarily hard without exploiting target domain information. Test-time adaptive (TTA) methods are proposed to address this issue. In this work, we adopt Non-Parametric to perform the test-time Adaptation (AdaNPC)
  arXiv  Detail & Related papers  (2023-04-25T04:23:13Z)
• Adaptive Cross Batch Normalization for Metric Learning [75.91093210956116]
  Metric learning is a fundamental problem in computer vision. We show that it is equally important to ensure that the accumulated embeddings are up to date. In particular, it is necessary to circumvent the representational drift between the accumulated embeddings and the feature embeddings at the current training iteration.
  arXiv  Detail & Related papers  (2023-03-30T03:22:52Z)
• Leveraging Time Irreversibility with Order-Contrastive Pre-training [3.1848820580333737]
  We explore an "order-contrastive" method for self-supervised pre-training on longitudinal data. We prove a finite-sample guarantee for the downstream error of a representation learned with order-contrastive pre-training. Our results indicate that pre-training methods designed for particular classes of distributions and downstream tasks can improve the performance of self-supervised learning.
  arXiv  Detail & Related papers  (2021-11-04T02:56:52Z)
• Overcoming Catastrophic Forgetting via Direction-Constrained Optimization [43.53836230865248]
  We study a new design of the optimization algorithm for training deep learning models with a fixed architecture of the classification network in a continual learning framework. We present our direction-constrained optimization (DCO) method, where for each task we introduce a linear autoencoder to approximate its corresponding top forbidden principal directions. We demonstrate that our algorithm performs favorably compared to other state-of-art regularization-based continual learning methods.
  arXiv  Detail & Related papers  (2020-11-25T08:45:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.