Less is More: Geometric Unlearning for LLMs with Minimal Data Disclosure

• URL: http://arxiv.org/abs/2605.01735v1
• Date: Sun, 03 May 2026 06:20:03 GMT
• Title: Less is More: Geometric Unlearning for LLMs with Minimal Data Disclosure
• Abstract summary: Geometric Unlearning (GU) operates directly on the model's prompt-time planning states without access to the original training corpus.<n> GU distills a compact, low-rank geometry of desired safe behavior from a small set of safe reference prompts.<n>A teacher-distillation regularizer on synthetic non-target anchors further reduces collateral drift.
• Score: 10.0091317316944
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As large language models (LLMs) are increasingly deployed in real-world systems, they must support post-hoc removal of specific content to meet privacy and governance requirements. This motivates selective unlearning, which suppresses information about a particular entity or topic while preserving the LLM's general utility. However, most existing LLM unlearning methods require access to the original training corpus and rely on output-level refusal tuning or broad gradient updates, creating a tension among unlearning strength, non-target preservation, and data availability. We propose Geometric Unlearning (GU), an approach that operates directly on the model's prompt-time planning states without access to the original training corpus. GU distills a compact, low-rank geometry of desired safe behavior from a small set of safe reference prompts, and uses lightweight anchor-in-context synthetic prompts to trigger localized, projection-based alignment of hidden planning representations to this safe geometry. A teacher-distillation regularizer on synthetic non-target anchors further reduces collateral drift. Across privacy-oriented unlearning benchmarks (ToFU and UnlearnPII), GU achieves strong target suppression with minimal impact on non-target performance, demonstrating that effective unlearning can be achieved with minimal synthetic data.

Related papers

• Forgetting Only What Matters: Layer-Selective Unlearning toward Robust LLMs [1.988145627448243]
  Large Language Models (LLMs) can memorize and reproduce sensitive, copyrighted, or otherwise undesirable training content.<n>We propose Forgetting Only What Matters via Unlearning Layers (FOM-UL), a layer-level unlearning framework that selects transformer layers using a forget-to-retain significance score.
  arXiv  Detail & Related papers  (2026-09-09T16:50:41Z)
• Operationalising the Right to be Forgotten in LLMs: A Lightweight Sequential Unlearning Framework for Privacy-Aligned Deployment in Politically Sensitive Environments [0.21485350418225244]
  Large Language Models (LLMs) are increasingly deployed in politically sensitive environments, where memorisation of personal data or confidential content raises regulatory concerns.<n>We introduce a lightweight sequential unlearning framework that explicitly separates retention and suppression objectives.<n> Experiments on the SemEval-2025 LLM Unlearning benchmark demonstrate effective behavioural suppression with minimal impact on factual accuracy and robustness.
  arXiv  Detail & Related papers  (2026-04-14T08:45:12Z)
• MeGU: Machine-Guided Unlearning with Target Feature Disentanglement [73.49657372882082]
  We propose a novel framework that guides unlearning through concept-aware re-alignment.<n>MeGU enables controlled and selective forgetting, effectively mitigating both under-unlearning and over-unlearning.
  arXiv  Detail & Related papers  (2026-02-19T05:20:31Z)
• Rendering Data Unlearnable by Exploiting LLM Alignment Mechanisms [3.648393062009244]
  Large language models (LLMs) are increasingly trained on massive, heterogeneous text corpora.<n>This raises serious concerns about the unauthorised use of proprietary or personal data during model training.<n>We propose Disclaimer Injection, a novel data-level defence that renders text unlearnable to LLMs.
  arXiv  Detail & Related papers  (2026-01-06T20:34:15Z)
• DRAGON: Guard LLM Unlearning in Context via Negative Detection and Reasoning [15.58340591381191]
  Unlearning in Large Language Models (LLMs) is crucial for protecting private data and removing harmful knowledge.<n>Most existing approaches rely on fine-tuning to balance unlearning efficiency with general language capabilities.<n>We propose Detect-Reasoning Augmented GeneratiON (DRAGON) to overcome these limitations.
  arXiv  Detail & Related papers  (2025-11-08T01:13:28Z)
• Grounded in Reality: Learning and Deploying Proactive LLM from Offline Logs [72.08224879435762]
  textttLearn-to-Ask is a simulator-free framework for learning and deploying proactive dialogue agents.<n>Our approach culminates in the successful deployment of LLMs into a live, large-scale online AI service.
  arXiv  Detail & Related papers  (2025-10-29T12:08:07Z)
• Federated Learning-Enabled Hybrid Language Models for Communication-Efficient Token Transmission [87.68447072141402]
  Hybrid Language Models (HLMs) combine the low-latency efficiency of Small Language Models (SLMs) on edge devices with the high accuracy of Large Language Models (LLMs) on centralized servers.<n>We propose FedHLM, a communication-efficient HLM framework that integrates uncertainty-aware inference with Federated Learning (FL)
  arXiv  Detail & Related papers  (2025-06-30T02:56:11Z)
• GUARD: Guided Unlearning and Retention via Data Attribution for Large Language Models [17.83305806604326]
  GUARD is a framework for guided unlearning and retention via data attribution.<n>It assigns adaptive, nonuniform unlearning weights to samples, inversely proportional to their proxy attribution scores.<n>We provide rigorous theoretical guarantees that GUARD significantly improves retention while maintaining forgetting metrics comparable to prior methods.
  arXiv  Detail & Related papers  (2025-06-12T17:49:09Z)
• FedShield-LLM: A Secure and Scalable Federated Fine-Tuned Large Language Model [0.48342038441006796]
  Federated Learning (FL) offers a decentralized framework for training and fine-tuning Large Language Models (LLMs)<n>FL addresses privacy and security concerns while navigating challenges associated with the substantial computational demands of LLMs.<n>We propose a novel method, FedShield-LLM, that uses pruning with Fully Homomorphic Encryption (FHE) for Low-Rank Adaptation (LoRA) parameters.
  arXiv  Detail & Related papers  (2025-06-06T00:05:05Z)
• Robust Anti-Backdoor Instruction Tuning in LVLMs [53.766434746801366]
  We introduce a lightweight, certified-agnostic defense framework for large visual language models (LVLMs)<n>Our framework finetunes only adapter modules and text embedding layers under instruction tuning.<n>Experiments against seven attacks on Flickr30k and MSCOCO demonstrate that ours reduces their attack success rate to nearly zero.
  arXiv  Detail & Related papers  (2025-06-04T01:23:35Z)
• UniErase: Towards Balanced and Precise Unlearning in Language Models [69.04923022755547]
  Large language models (LLMs) require iterative updates to address the outdated information problem.<n>UniErase is a novel unlearning framework that demonstrates precision and balanced performances between knowledge unlearning and ability retaining.
  arXiv  Detail & Related papers  (2025-05-21T15:53:28Z)
• Towards Robust and Parameter-Efficient Knowledge Unlearning for LLMs [25.91643745340183]
  Large Language Models (LLMs) have demonstrated strong reasoning and memorization capabilities via pretraining on massive textual corpora.<n>This poses risk of privacy and copyright violations, highlighting the need for efficient machine unlearning methods.<n>We propose Low-rank Knowledge Unlearning (LoKU), a novel framework that enables robust and efficient unlearning for LLMs.
  arXiv  Detail & Related papers  (2024-08-13T04:18:32Z)
• Robust Utility-Preserving Text Anonymization Based on Large Language Models [80.5266278002083]
  Anonymizing text that contains sensitive information is crucial for a wide range of applications.<n>Existing techniques face the emerging challenges of the re-identification ability of large language models.<n>We propose a framework composed of three key components: a privacy evaluator, a utility evaluator, and an optimization component.
  arXiv  Detail & Related papers  (2024-07-16T14:28:56Z)
• Unveiling the Misuse Potential of Base Large Language Models via In-Context Learning [61.2224355547598]
  Open-sourcing of large language models (LLMs) accelerates application development, innovation, and scientific progress. Our investigation exposes a critical oversight in this belief. By deploying carefully designed demonstrations, our research demonstrates that base LLMs could effectively interpret and execute malicious instructions.
  arXiv  Detail & Related papers  (2024-04-16T13:22:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.