Forget What Matters, Keep the Rest: Selective Unlearning of Informative Tokens

• URL: http://arxiv.org/abs/2604.17785v1
• Date: Mon, 20 Apr 2026 04:20:29 GMT
• Title: Forget What Matters, Keep the Rest: Selective Unlearning of Informative Tokens
• Abstract summary: We propose Entropy-guided Token Weighting (ETW) as a token-level unlearning regularizer.<n>ETW uses entropy of the predictive distribution as a proxy for token informativeness.<n>We demonstrate that informative tokens tend to have higher entropy, whereas structural tokens tend to have lower entropy.
• Score: 9.36751822006717
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Unlearning in large language models (LLMs) has emerged as a promising safeguard against adversarial behaviors. When the forgetting loss is applied uniformly without considering token-level semantic importance, model utility can be unnecessarily degraded. Recent studies have explored token-wise loss regularizers that prioritize informative tokens, but largely rely on ground-truth confidence or external linguistic parsers, which limits their ability to capture contextual information or the model's overall predictive state. Intuitively, function words like "the" primarily serve syntactic roles and are highly predictable with little ambiguity, but informative words admit multiple plausible alternatives with greater uncertainty. Based on this intuition, we propose Entropy-guided Token Weighting (ETW), a token-level unlearning regularizer that uses entropy of the predictive distribution as a proxy for token informativeness. We demonstrate that informative tokens tend to have higher entropy, whereas structural tokens tend to have lower entropy. This behavior enables ETW to achieve more effective unlearning while better preserving model utility than existing token-level approaches.

Related papers

• Learning What to Forget: Improving LLM Unlearning via Learned Token-Level Importance [30.72675145829489]
  We introduce Alternating Token-Weighted Unlearning (ATWU), a lightweight framework that jointly learns token forget-specificity and model parameters during unlearning.<n>ATWU achieves state of the art forget-retain trade-offs, outperforming sample-level methods, probability-based token weightings, and auxiliary-model-based approaches.
  arXiv  Detail & Related papers  (2026-06-04T15:56:32Z)
• Unlearning What Matters: Token-Level Attribution for Precise Language Model Unlearning [5.454773103061359]
  TokenUnlearn is a token-level attribution framework that identifies and selectively targets critical tokens.<n>Our approach combines knowledge-aware signals via masking, and entropy-aware signals to yield importance scores for precise token selection.
  arXiv  Detail & Related papers  (2026-05-01T02:59:03Z)
• Probability-Entropy Calibration: An Elastic Indicator for Adaptive Fine-tuning [55.2818264614932]
  RankTuner introduces a probability--entropy calibration signal, the Relative Rank Indicator, which compares the rank of the ground-truth token with its expected rank under the prediction distribution.<n>The inverse indicator is used as a token-wise Relative Scale to reweight the fine-tuning objective, focusing updates on truly under-learned tokens.
  arXiv  Detail & Related papers  (2026-02-02T07:27:19Z)
• Reflection Pretraining Enables Token-Level Self-Correction in Biological Sequence Models [82.79223371188756]
  Chain-of-Thought (CoT) prompting has advanced task-solving capabilities in natural language processing with large language models.<n>Applying CoT to non-natural language domains, such as protein and RNA language models, is not yet possible.<n>We introduce pretraining, for the first time in a biological sequence model, which enables the model to engage in intermediate reasoning.
  arXiv  Detail & Related papers  (2025-12-24T05:25:17Z)
• Temporal Predictors of Outcome in Reasoning Language Models [0.0]
  Chain-of-thought (CoT) paradigm uses the elicitation of step-by-step rationales as a proxy for reasoning.<n>We show that, for harder questions, a drop in predictive accuracy highlights a selection artifact.<n>Overall, our results imply that for reasoning models, internal self-assessment of success tends to emerge after only a few tokens.
  arXiv  Detail & Related papers  (2025-11-03T08:57:18Z)
• ssToken: Self-modulated and Semantic-aware Token Selection for LLM Fine-tuning [51.133569963553576]
  ssToken is a Self-modulated and Semantic-aware Token Selection approach.<n>We show that both self-modulated selection and semantic-aware selection alone outperform full-data fine-tuning.
  arXiv  Detail & Related papers  (2025-10-21T03:21:04Z)
• Forgetting: A New Mechanism Towards Better Large Language Model Fine-tuning [51.92313556418432]
  Supervised fine-tuning (SFT) plays a critical role for pretrained large language models (LLMs)<n>We suggest categorizing tokens within each corpus into two parts -- positive and negative tokens -- based on whether they are useful to improve model performance.<n>We conduct experiments on well-established benchmarks, finding that this forgetting mechanism not only improves overall model performance and also facilitate more diverse model responses.
  arXiv  Detail & Related papers  (2025-08-06T11:22:23Z)
• IGD: Token Decisiveness Modeling via Information Gain in LLMs for Personalized Recommendation [79.22388408461458]
  We introduce an Information Gain-based Decisiveness-aware Token handling (IGD) strategy that integrates token decisiveness into both tuning and decoding.<n>IGD consistently improves recommendation accuracy, achieving significant gains on widely used ranking metrics compared to strong baselines.
  arXiv  Detail & Related papers  (2025-06-16T08:28:19Z)
• Not Every Token Needs Forgetting: Selective Unlearning to Limit Change in Utility in Large Language Model Unlearning [95.53571199301963]
  Conventional unlearning approaches indiscriminately update model parameters to forget all tokens in a target document.<n>We propose Selective Unlearning (SU), which identifies a critical subset of tokens within the forgetting set that is relevant to the unwanted information.<n>Experiments on two benchmarks and six baseline unlearning algorithms demonstrate that SU not only achieves effective unlearning on the targeted forget data, but also significantly preserves the model's utility in the retaining set.
  arXiv  Detail & Related papers  (2025-06-01T07:36:45Z)
• Tokens, the oft-overlooked appetizer: Large language models, the distributional hypothesis, and meaning [29.745218855471787]
  Tokenization is a necessary component within the current architecture of many language mod-els.<n>We argue that tokenization is necessary for reasonably human-like language performance.<n>We discuss implications for architectural choices, meaning construction, the primacy of language for thought.
  arXiv  Detail & Related papers  (2024-12-14T18:18:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.