Decoupled Mixture-of-Experts for Parametric Knowledge Injection

• URL: http://arxiv.org/abs/2606.14243v1
• Date: Fri, 12 Jun 2026 08:21:28 GMT
• Title: Decoupled Mixture-of-Experts for Parametric Knowledge Injection
• Abstract summary: Decoupled Mixture-of-Experts (DMoE) is a modular architecture for parametric knowledge injection.<n>DMoE converts external knowledge corpora into independently updatable expert modules and uses a lightweight uncertainty-aware router to activate relevant experts.<n>Experiments on knowledge-intensive benchmarks show that DMoE consistently improves answer quality over retrieval and adapter-based baselines.
• Score: 29.992826207672568
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Knowledge injection aims to equip large language models (LLMs) with external, domain-specific, or time-sensitive knowledge. Existing approaches typically face a trade-off between flexibility and integration: retrieval-augmented generation keeps knowledge outside the model but only provides prompt-level augmentation, whereas post-training based methods encode new knowledge into shared parameters but may introduce catastrophic forgetting, knowledge conflict, and costly updates. In this paper, we propose Decoupled Mixture-of-Experts (DMoE), a modular architecture for parametric knowledge injection that decouples both experts and the router from the base model. DMoE converts external knowledge corpora into independently updatable expert modules and uses a lightweight uncertainty-aware router to activate relevant experts only when the base model lacks sufficient knowledge during generation. To support efficient auto-regressive inference, DMoE attaches experts only to the final-layer feed-forward network, preserving KV-cache reuse while enabling parameter-level knowledge augmentation. Experiments on knowledge-intensive benchmarks show that DMoE consistently improves answer quality over retrieval and adapter-based baselines.

Related papers

• Sparse Subspace-to-Expert Sharing for Task-Agnostic Continual Learning [10.01449025634975]
  Existing methods treat parameters uniformly, failing to distinguish between specific task knowledge and shared capabilities.<n>We introduce Mixture of Sparse Experts for Task Agnostic Continual Learning (SETA), a framework that resolves the plasticity-stability conflict through adaptive sparse subspace decomposition.<n>We show SETA achieves competitive or superior overall performance relative to state-of-the-art continual learning baselines.
  arXiv  Detail & Related papers  (2026-06-05T17:53:52Z)
• Dynamic Mixture of Latent Memories for Self-Evolving Agents [57.20419419302731]
  MoLEM is a generative mixture of latent memory framework based on a dynamic mixture-of-experts (MoE)<n>We train the framework on continual-learning sequences spanning math, science, and code domains.<n>After the full continual-learning sequence, our method improves the average accuracy by 10.40% over the Vanilla pretrained baseline.
  arXiv  Detail & Related papers  (2026-05-21T03:35:10Z)
• DeepRefine: Agent-Compiled Knowledge Refinement via Reinforcement Learning [60.35032268606525]
  Agent-compiled knowledge bases provide persistent external knowledge for large language model (LLM) agents in downstream tasks.<n>Yet their quality is systematically limited by emphincompleteness, emphincorrectness, and emphredundancy
  arXiv  Detail & Related papers  (2026-05-11T12:48:31Z)
• Knowledge Capsules: Structured Nonparametric Memory Units for LLMs [0.6524460254566904]
  Large language models (LLMs) encode knowledge in parametric weights, making it costly to update or extend without retraining.<n>Retrieval-augmented generation (RAG) mitigates this limitation by appending retrieved text to the input, but operates purely through context expansion.<n>We propose Knowledge Capsules, structured nonparametric memory units that represent normalized relational knowledge.
  arXiv  Detail & Related papers  (2026-04-22T12:21:00Z)
• Split-on-Share: Mixture of Sparse Experts for Task-Agnostic Continual Learning [10.01449025634975]
  Continual learning in Large Language Models (LLMs) is hindered by the plasticity-stability dilemma.<n>We introduce SETA, a framework that resolves the plasticity-stability conflict by decomposing the model into modular subspaces.<n>We show that SETA consistently outperforms state-of-the-art parameter-efficient fine-tuning-based continual learning methods.
  arXiv  Detail & Related papers  (2026-01-24T22:39:22Z)
• PropMEND: Hypernetworks for Knowledge Propagation in LLMs [82.99849359892112]
  We present a hypernetwork-based approach for knowledge propagation, named PropMEND.<n>We show almost 2x accuracy on challenging multi-hop questions whose answers are not explicitly stated in the injected fact.<n>We also introduce a new dataset, Controlled RippleEdit, to evaluate the generalization of our hypernetwork.
  arXiv  Detail & Related papers  (2025-06-10T15:44:19Z)
• Decoupling Reasoning and Knowledge Injection for In-Context Knowledge Editing [12.5122702720856]
  In-context editing (ICE) offers a lightweight solution by injecting new knowledge directly into the input context.<n>Existing ICE approaches do not explicitly separate the newly injected knowledge from the model's original reasoning process.<n>We propose DecKER, a novel ICE framework that decouples reasoning from knowledge editing by generating a masked reasoning path.
  arXiv  Detail & Related papers  (2025-05-31T12:51:12Z)
• GIVE: Structured Reasoning of Large Language Models with Knowledge Graph Inspired Veracity Extrapolation [108.2008975785364]
  Graph Inspired Veracity Extrapolation (GIVE) is a novel reasoning method that merges parametric and non-parametric memories to improve accurate reasoning with minimal external input.<n>GIVE guides the LLM agent to select the most pertinent expert data (observe), engage in query-specific divergent thinking (reflect), and then synthesize this information to produce the final output (speak)
  arXiv  Detail & Related papers  (2024-10-11T03:05:06Z)
• MATTER: Memory-Augmented Transformer Using Heterogeneous Knowledge Sources [12.783393023641505]
  We introduce an efficient memory-augmented transformer called MATTER. MATTER retrieves and reads from both unstructured sources (paragraphs) and semi-structured sources (QA pairs) in the form of fixed-length neural memories. We demonstrate that our model outperforms existing efficient retrieval-augmented models on popular QA benchmarks in terms of both accuracy and speed.
  arXiv  Detail & Related papers  (2024-06-07T06:35:37Z)
• Auto-selected Knowledge Adapters for Lifelong Person Re-identification [54.42307214981537]
  Lifelong Person Re-Identification requires systems to continually learn from non-overlapping datasets across different times and locations. Existing approaches, either rehearsal-free or rehearsal-based, still suffer from the problem of catastrophic forgetting. We introduce a novel framework AdalReID, that adopts knowledge adapters and a parameter-free auto-selection mechanism for lifelong learning.
  arXiv  Detail & Related papers  (2024-05-29T11:42:02Z)
• UNTER: A Unified Knowledge Interface for Enhancing Pre-trained Language Models [100.4659557650775]
  We propose a UNified knowledge inTERface, UNTER, to provide a unified perspective to exploit both structured knowledge and unstructured knowledge. With both forms of knowledge injected, UNTER gains continuous improvements on a series of knowledge-driven NLP tasks.
  arXiv  Detail & Related papers  (2023-05-02T17:33:28Z)
• Kformer: Knowledge Injection in Transformer Feed-Forward Layers [107.71576133833148]
  We propose a novel knowledge fusion model, namely Kformer, which incorporates external knowledge through the feed-forward layer in Transformer. We empirically find that simply injecting knowledge into FFN can facilitate the pre-trained language model's ability and facilitate current knowledge fusion methods.
  arXiv  Detail & Related papers  (2022-01-15T03:00:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.