Output Vector Editing for Memorization Mitigation in Large Language Models

• URL: http://arxiv.org/abs/2606.18767v1
• Date: Wed, 17 Jun 2026 07:29:18 GMT
• Title: Output Vector Editing for Memorization Mitigation in Large Language Models
• Abstract summary: Large language models memorize and reproduce sequences from their training data, creating privacy, copyright, and security risks.<n>Existing neuron-level mitigation methods equate editing with zeroing out neuron activations, but the activation only controls whether a neuron engages.<n>We propose output vector editing, a constrained-optimization edit that locates a small set of neurons responsible for a memorized continuation.
• Score: 68.30351930772788
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models memorize and reproduce sequences from their training data, creating privacy, copyright, and security risks. Existing neuron-level mitigation methods equate editing with zeroing out neuron activations, but the activation only controls whether a neuron engages; the output vector is what writes to the residual stream and, through superposition, encodes multiple features. We propose output vector editing, a constrained-optimization weight edit that locates a small set of MLP neurons responsible for a memorized continuation and minimally modifies their output vectors to introduce a distractor in vocabulary space, redirecting their residual-stream contributions while leaving activations unchanged. Evaluating on four models from 360M to 7B parameters (SmolLM-360M, OLMo-1B, OLMo-7B, Llama2-7B), we center on OLMo-7B (whose open weights and pretraining corpus enable systematic mining) and mine 6831 memorized sequences, achieving up to 87.9% suppression. The 2.7$\times$ gap over zero ablation on the same located neurons shows the suppression comes from the output-vector edit, not localization alone. Four edit modes span a spectrum from aggressive suppression to minimal redirection; in ensemble they cover 96.5% of memorized sequences, while our recommended single-mode configuration reaches 81.5% with no catastrophic locality failures. We further identify a mechanistic boundary at ${\sim}14%$ of sequences unreachable by MLP-only editing; while these failures are not attention-driven overall, ablating the top contributing attention heads recovers 60--64% of them, with stronger recovery on continuations that copy tokens from the prefix, positioning attention as a complementary fallback rather than a primary mechanism. Edit mode ordering and the success-locality trade-off transfer across all four models, with success rates scaling with model size rather than family.

Related papers

• A Few Neurons Reveal When LLMs Misuse Tools: Sparse Detection and Selective Steering for Reliable Tool Use [28.57781785937644]
  We introduce PRISMS (Probing Representations In Support of Monitoring and Steering), a closed-loop framework that shares a failure-specific neuron basis between sparse detection and activation.<n>Across six models, PRISMS reduces pooled over-calling rate by 80% while increasing tool-required accuracy by 14.2 percentage points.
  arXiv  Detail & Related papers  (2026-07-31T19:03:44Z)
• Now You (Still) See Me: Detecting Evasive Steganographic Payloads in LLMs [52.149036302760386]
  Large language models can be fine-tuned to encode prompt-borne secrets into fluent, seemingly benign outputs.<n>Recent work proposes detection using linear probes that recover the secret from internal activations.<n>We show that this defense can be systematically evaded, but that detectability can be recovered through a targeted data-level intervention.
  arXiv  Detail & Related papers  (2026-06-08T12:27:11Z)
• Moir: Let the Model Direct Its Own Story for Robust Cross-Domain Knowledge Editing [19.592670426853115]
  Moir estimates the preservation covariance $C$ directly from the model itself by sampling from its own decoding distribution.<n>Across OLMo-2, Llama-3.1, and Qwen-3 (7-8B), under both MEMIT and AlphaEdit, Moir consistently extends preservation in the most vulnerable domains.<n>Moir retains 79.9% GSM8K accuracy compared to 10.9% with the Wikipedia baseline.
  arXiv  Detail & Related papers  (2026-05-10T16:05:49Z)
• Perturbation Probing: A Two-Pass-per-Prompt Diagnostic for FFN Behavioral Circuits in Aligned LLMs [9.127363793428119]
  Perturbation probing generates task-specific causal hypotheses for FFN neurons in large language models.<n>Across eight behavioral circuits, 13 models, and four architecture families, we identify two circuit structures that organize LLM behavior.
  arXiv  Detail & Related papers  (2026-04-30T04:13:33Z)
• ProjLens: Unveiling the Role of Projectors in Multimodal Model Safety [54.4092272526747]
  Multimodal Large Language Models (MLLMs) have achieved remarkable success in cross-modal understanding and generation, yet their deployment is threatened by critical safety vulnerabilities.<n>We propose ProjLens, an interpretability framework designed to demystify MLLMs backdoors.
  arXiv  Detail & Related papers  (2026-04-21T04:52:38Z)
• Breaking the Illusion of Identity in LLM Tooling [0.0]
  No existing mitigation provides a systematic deployable constraint set output register.<n>This paper proposes seven outputside rules, each targeting a documented linguistic mechanism.
  arXiv  Detail & Related papers  (2026-04-08T09:15:14Z)
• NeuReasoner: Towards Explainable, Controllable, and Unified Reasoning via Mixture-of-Neurons [25.628859853488862]
  We present NeuReasoner, a unified reasoning framework driven by Neurons (MoN)<n>NeuReasoner achieves performance gains of up to 27.0% while reducing token consumption by 19.6% 63.3%.
  arXiv  Detail & Related papers  (2026-04-03T11:20:16Z)
• Surgical Repair of Collapsed Attention Heads in ALiBi Transformers [0.0]
  We identify a systematic attention collapse pathology in the BLOOM family of transformer language models.<n>ALiBi positional encoding causes 31-44% of attention heads to attend almost entirely to the beginning-of-sequence token.<n>We introduce surgical reinitialization: targeted Q/K/V reinitialization with zeroed output projections.
  arXiv  Detail & Related papers  (2026-03-10T12:57:49Z)
• Self-Purification Mitigates Backdoors in Multimodal Diffusion Language Models [74.1970982768771]
  We show that well-established data-poisoning pipelines can successfully implant backdoors into MDLMs.<n>We introduce a backdoor defense framework for MDLMs named DiSP (Diffusion Self-Purification)
  arXiv  Detail & Related papers  (2026-02-24T15:47:52Z)
• A Few Bad Neurons: Isolating and Surgically Correcting Sycophancy [7.405817106579332]
  Behavioral alignment in large language models is often achieved through broad fine-tuning.<n>We propose a method for alignment that identifies and updates only the neurons most responsible for a given behavior.<n>Our results show that sparse, neuron-level updates offer a scalable and precise alternative to full-model fine-tuning.
  arXiv  Detail & Related papers  (2026-01-26T20:20:13Z)
• ShortcutBreaker: Low-Rank Noisy Bottleneck with Global Perturbation Attention for Multi-Class Unsupervised Anomaly Detection [59.89803740308262]
  ShortcutBreaker is a novel unified feature-reconstruction framework for MUAD tasks.<n>It features two key innovations to address the issue of shortcuts.<n>The proposed method achieves a remarkable image-level AUROC of 99.8%, 98.9%, 90.6%, and 87.8% on four datasets.
  arXiv  Detail & Related papers  (2025-10-21T06:51:30Z)
• Lazy Layers to Make Fine-Tuned Diffusion Models More Traceable [70.77600345240867]
  A novel arbitrary-in-arbitrary-out (AIAO) strategy makes watermarks resilient to fine-tuning-based removal. Unlike the existing methods of designing a backdoor for the input/output space of diffusion models, in our method, we propose to embed the backdoor into the feature space of sampled subpaths. Our empirical studies on the MS-COCO, AFHQ, LSUN, CUB-200, and DreamBooth datasets confirm the robustness of AIAO.
  arXiv  Detail & Related papers  (2024-05-01T12:03:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.