ConceptGuard: Benchmarking Context-Sensitive Unlearning in Large Language Models

• URL: http://arxiv.org/abs/2608.20338v1
• Date: Thu, 20 Aug 2026 17:59:57 GMT
• Title: ConceptGuard: Benchmarking Context-Sensitive Unlearning in Large Language Models
• Abstract summary: Large Language Models (LLMs) increasingly require selective removal of harmful or sensitive knowledge, called unlearning.<n>Current approaches rely on disjoint forget and retain sets composed of independent facts, and measure success using simple and direct factual recall.<n>We argue that effective unlearning must operate at the level of concepts, ensuring complete removal of unsafe applications while maintaining their correct and useful usage.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Large Language Models (LLMs) increasingly require selective removal of harmful or sensitive knowledge, called unlearning, yet existing methods and benchmarks fail to evaluate this capability completely. Current approaches rely on disjoint forget and retain sets composed of independent facts, and measure success using simple and direct factual recall. This framing fails to capture a key requirement of unlearning, namely the ability to eliminate harmful behaviors while preserving benign and beneficial knowledge. We argue that effective unlearning must operate at the level of concepts, ensuring complete removal of unsafe applications while maintaining their correct and useful usage, thereby achieving conceptually meaningful and complete unlearning. To better evaluate unlearning techniques from such a practical viewpoint, we introduce the notion of dual-use concepts: concepts that can be used in both harmful and benign contexts. Building on these concepts, we construct a benchmark called ConceptGuard where forget and retain sets are explicitly complementary in concept usage. Our benchmark uniquely enables unlearning to be explored and gauged at the level of concepts, instead of sparse facts, and evaluation is intent-sensitive with the goal of maximizing contextual separation to promote safer behavior. We demonstrate that current unlearning techniques perform poorly under this setting, showing weak contextual separation alongside poor performance in ROUGE and concept-level metrics. Our results reveal strong forgetting-utility trade-offs, limited gains in contextual sensitivity, and poor consistency in concept-level control across methods, and provide ideas for unlearning approaches that better align with real-world safety requirements. Our dataset is publicly available.

Related papers

• Co-occurring associated retained concepts in Diffusion Unlearning [17.80179076672276]
  Unlearning can unintentionally suppress the concept of person, preventing a model from generating images with person.<n>We define these undesirably suppressed co-occurring concepts that must be preserved CARE.<n>We propose ReCARE, a framework that explicitly safeguards CARE while erasing only the target concept.
  arXiv  Detail & Related papers  (2026-06-23T06:22:00Z)
• Distinguishable Deletion: Unifying Knowledge Erasure and Refusal for Large Language Model Unlearning [58.725080160369494]
  Distinguishable Deletion ($mathrmD2$) is a paradigm that restricts the response distribution in the latent representation rather than specific tokens.<n>We introduce an energy index that quantifies the presence of knowledge and the separation between unlearned and retained content.<n>Experiments demonstrate that EUA significantly outperforms previous methods, indicating the superiority of $mathrmD2$.
  arXiv  Detail & Related papers  (2026-05-16T03:15:35Z)
• ICED: Concept-level Machine Unlearning via Interpretable Concept Decomposition [44.733160260712054]
  We propose an interpretable concept-level unlearning framework for Vision-Language Models (VLMs)<n>Our method formulates unlearning as concept-level optimization, where target concepts are selectively suppressed while intra-instance non-target semantics and global cross-modal knowledge are preserved.
  arXiv  Detail & Related papers  (2026-05-14T03:22:12Z)
• MeGU: Machine-Guided Unlearning with Target Feature Disentanglement [73.49657372882082]
  We propose a novel framework that guides unlearning through concept-aware re-alignment.<n>MeGU enables controlled and selective forgetting, effectively mitigating both under-unlearning and over-unlearning.
  arXiv  Detail & Related papers  (2026-02-19T05:20:31Z)
• AUVIC: Adversarial Unlearning of Visual Concepts for Multi-modal Large Language Models [63.05306474002547]
  Regulatory frameworks mandating the 'right to be forgotten' drive the need for machine unlearning.<n>We introduce AUVIC, a novel visual concept unlearning framework for MLLMs.<n>We show that AUVIC achieves state-of-the-art target forgetting rates while incurs minimal performance degradation on non-target concepts.
  arXiv  Detail & Related papers  (2025-11-14T13:35:32Z)
• DRAGON: Guard LLM Unlearning in Context via Negative Detection and Reasoning [15.58340591381191]
  Unlearning in Large Language Models (LLMs) is crucial for protecting private data and removing harmful knowledge.<n>Most existing approaches rely on fine-tuning to balance unlearning efficiency with general language capabilities.<n>We propose Detect-Reasoning Augmented GeneratiON (DRAGON) to overcome these limitations.
  arXiv  Detail & Related papers  (2025-11-08T01:13:28Z)
• Forget to Know, Remember to Use: Context-Aware Unlearning for Large Language Models [17.249936460923045]
  Large language models may encode sensitive information or outdated knowledge that needs to be removed.<n>Unlearning is an efficient alternative to full retraining, aiming to remove specific knowledge while preserving overall model utility.
  arXiv  Detail & Related papers  (2025-10-20T15:03:45Z)
• CURE: Controlled Unlearning for Robust Embeddings - Mitigating Conceptual Shortcuts in Pre-Trained Language Models [23.898244353656352]
  We introduce CURE, a framework that systematically disentangles and suppresses conceptual shortcuts.<n>CURE achieves an absolute improvement of +10 points in F1 score on IMDB and +2 points on Yelp.
  arXiv  Detail & Related papers  (2025-09-05T16:47:22Z)
• Weakly-Supervised Contrastive Learning for Imprecise Class Labels [50.57424331797865]
  We introduce the concept of continuous semantic similarity'' to define positive and negative pairs.<n>We propose a graph-theoretic framework for weakly-supervised contrastive learning.<n>Our framework is highly versatile and can be applied to many weakly-supervised learning scenarios.
  arXiv  Detail & Related papers  (2025-05-28T06:50:40Z)
• Towards Effective Evaluations and Comparisons for LLM Unlearning Methods [97.2995389188179]
  This paper seeks to refine the evaluation of machine unlearning for large language models.<n>It addresses two key challenges -- the robustness of evaluation metrics and the trade-offs between competing goals.
  arXiv  Detail & Related papers  (2024-06-13T14:41:00Z)
• Translational Concept Embedding for Generalized Compositional Zero-shot Learning [73.60639796305415]
  Generalized compositional zero-shot learning means to learn composed concepts of attribute-object pairs in a zero-shot fashion. This paper introduces a new approach, termed translational concept embedding, to solve these two difficulties in a unified framework.
  arXiv  Detail & Related papers  (2021-12-20T21:27:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.