Moir: Let the Model Direct Its Own Story for Robust Cross-Domain Knowledge Editing

• URL: http://arxiv.org/abs/2607.20433v1
• Date: Sun, 10 May 2026 16:05:49 GMT
• Title: Moir: Let the Model Direct Its Own Story for Robust Cross-Domain Knowledge Editing
• Abstract summary: Moir estimates the preservation covariance $C$ directly from the model itself by sampling from its own decoding distribution.<n>Across OLMo-2, Llama-3.1, and Qwen-3 (7-8B), under both MEMIT and AlphaEdit, Moir consistently extends preservation in the most vulnerable domains.<n>Moir retains 79.9% GSM8K accuracy compared to 10.9% with the Wikipedia baseline.
• Score: 19.592670426853115
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While language models remain frozen at their training state, the world evolves continuously. Knowledge editing has emerged as a key alternative to full retraining, but its deployment is bottlenecked by the erosion of core capabilities: mathematical and programmatic reasoning collapse while encyclopedic recall remains intact. We trace this asymmetric degradation to a distributional mismatch. Covariance-based editors preserve only the subspaces spanned by their reference corpus, but fail to capture the operative distribution shaped by post-training such as SFT and DPO. Static external corpora, including Wikipedia and even the original pretraining mixture, cannot recover this shifted manifold. We propose Moir, which estimates the preservation covariance $C$ directly from the model itself by sampling from its own decoding distribution. Seeding generation with a single random vocabulary token bypasses the instruction-following templates that otherwise dominate sampled outputs, exposing the broader subspaces the model has internalized. Moir requires no external data and serves as a drop-in component for any covariance-based editor, a practical advantage given that the pre- and post-training corpora of most modern LLMs are not publicly accessible. Across OLMo-2, Llama-3.1, and Qwen-3 (7-8B), under both MEMIT and AlphaEdit and in batch and sequential regimes, Moir consistently extends preservation in the most vulnerable domains, most strikingly on Qwen3-8B after 20,000 AlphaEdit batch edits, it retains 79.9% GSM8K accuracy compared to 10.9% with the Wikipedia baseline. These results suggest that aligning the preservation distribution with the model's operative distribution is a key factor in non-destructive editing, and that the model itself may be the most accessible source of that distribution for deployed systems.

Related papers

• Sample-Adaptive Latent Rewards for Uncertainty-Guided Diffusion Post-Training [51.18425726646089]
  We propose textscSURE, a unified latent-space framework for image and video diffusion models.<n>It learns reward distributions and directly uses their reliability to guide dense post-training.<n>textscSURE-REFL achieves the highest VBench quality, semantic, and total scores among the evaluated methods.
  arXiv  Detail & Related papers  (2026-08-06T14:55:42Z)
• MixSD: Mixed Contextual Self-Distillation for Knowledge Injection [29.7616760417696]
  Supervised fine-tuning (SFT) is widely used to inject new knowledge into language models.<n>We argue this forgetting arises because fine-tuning targets from humans or external systems diverge from the model's autoregressive distribution.<n>We propose MixSD, a simple external-teacher-free method for distribution-aligned knowledge injection.
  arXiv  Detail & Related papers  (2026-05-16T07:57:09Z)
• Self-Distilled Trajectory-Aware Boltzmann Modeling: Bridging the Training-Inference Discrepancy in Diffusion Language Models [65.89572755202245]
  Diffusion Language Models (DLMs) offer stronger global awareness and highly parallel generation.<n>Post-training DLMs with standard Negative Evidence Lower Bound (NELBO)-based supervised fine-tuning remains inefficient.<n>We propose a self-distilled trajectory-based post-training framework that aligns training with the easy-to-hard structure of inference.
  arXiv  Detail & Related papers  (2026-05-12T09:39:06Z)
• HoReN: Normalized Hopfield Retrieval for Large-Scale Sequential Model Editing [12.303016850029683]
  Large language models encode vast factual knowledge that inevitably becomes outdated or incorrect after deployment.<n>One line of work installs new facts by directly modifying base weights through locate-then-edit procedures.<n>A complementary line leaves base weights intact and routes edits through external memory, but it faces routing challenges.<n>We propose HoReN, a codebook-based editor with enhanced routing built on three ideas.
  arXiv  Detail & Related papers  (2026-05-02T15:51:31Z)
• STABLE: Gated Continual Learning for Large Language Models [0.0]
  STABLE is a gated continual self editing framework that constrains forgetting during sequential updates.<n>Each candidate edit is evaluated against a stability budget using one of three metrics.<n>Experiments on the Qwen-2.5-7B model show that gating effectively mitigates forgetting while preserving adaptability.
  arXiv  Detail & Related papers  (2025-10-17T16:14:05Z)
• SelfAug: Mitigating Catastrophic Forgetting in Retrieval-Augmented Generation via Distribution Self-Alignment [49.86376148975563]
  Large language models (LLMs) have revolutionized natural language processing through their capabilities in understanding and executing diverse tasks.<n> supervised fine-tuning, particularly in Retrieval-Augmented Generation (RAG) scenarios, often leads to catastrophic forgetting.<n>We propose SelfAug, a self-distribution alignment method that aligns input sequence logits to preserve the model's semantic distribution.
  arXiv  Detail & Related papers  (2025-09-04T06:50:47Z)
• Consistent Diffusion Models: Mitigating Sampling Drift by Learning to be Consistent [97.64313409741614]
  We propose to enforce a emphconsistency property which states that predictions of the model on its own generated data are consistent across time. We show that our novel training objective yields state-of-the-art results for conditional and unconditional generation in CIFAR-10 and baseline improvements in AFHQ and FFHQ.
  arXiv  Detail & Related papers  (2023-02-17T18:45:04Z)
• Divide and Contrast: Source-free Domain Adaptation via Adaptive Contrastive Learning [122.62311703151215]
  Divide and Contrast (DaC) aims to connect the good ends of both worlds while bypassing their limitations. DaC divides the target data into source-like and target-specific samples, where either group of samples is treated with tailored goals. We further align the source-like domain with the target-specific samples using a memory bank-based Maximum Mean Discrepancy (MMD) loss to reduce the distribution mismatch.
  arXiv  Detail & Related papers  (2022-11-12T09:21:49Z)
• Self-Conditioned Generative Adversarial Networks for Image Editing [61.50205580051405]
  Generative Adversarial Networks (GANs) are susceptible to bias, learned from either the unbalanced data, or through mode collapse. We argue that this bias is responsible not only for fairness concerns, but that it plays a key role in the collapse of latent-traversal editing methods when deviating away from the distribution's core.
  arXiv  Detail & Related papers  (2022-02-08T18:08:24Z)
• Do We Really Need to Access the Source Data? Source Hypothesis Transfer for Unsupervised Domain Adaptation [102.67010690592011]
  Unsupervised adaptationUDA (UDA) aims to leverage the knowledge learned from a labeled source dataset to solve similar tasks in a new unlabeled domain. Prior UDA methods typically require to access the source data when learning to adapt the model. This work tackles a practical setting where only a trained source model is available and how we can effectively utilize such a model without source data to solve UDA problems.
  arXiv  Detail & Related papers  (2020-02-20T03:13:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.