Knowledge Capsules: Structured Nonparametric Memory Units for LLMs

• URL: http://arxiv.org/abs/2604.20487v2
• Date: Thu, 23 Apr 2026 09:58:53 GMT
• Title: Knowledge Capsules: Structured Nonparametric Memory Units for LLMs
• Abstract summary: Large language models (LLMs) encode knowledge in parametric weights, making it costly to update or extend without retraining.<n>Retrieval-augmented generation (RAG) mitigates this limitation by appending retrieved text to the input, but operates purely through context expansion.<n>We propose Knowledge Capsules, structured nonparametric memory units that represent normalized relational knowledge.
• Score: 0.6524460254566904
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) encode knowledge in parametric weights, making it costly to update or extend without retraining. Retrieval-augmented generation (RAG) mitigates this limitation by appending retrieved text to the input, but operates purely through context expansion, where external knowledge competes as tokens within the attention mechanism. As a result, its influence is indirect and often unstable, particularly in long context and multi hop reasoning scenarios. We propose Knowledge Capsules, structured nonparametric memory units that represent normalized relational knowledge and can be constructed directly from document corpora using a frozen base model. Instead of injecting knowledge as text, we introduce an External Key Value Injection (KVI) framework that compiles capsules into attention-compatible key value representations, enabling external knowledge to directly participate in the model's attention computation. By shifting knowledge integration from context-level augmentation to memory level interaction, the proposed framework consistently outperforms RAG and GraphRAG across multiple QA benchmarks, with improved stability and accuracy in long context and multi hop reasoning, while requiring no parameter updates.

Related papers

• Decoupled Mixture-of-Experts for Parametric Knowledge Injection [29.992826207672568]
  Decoupled Mixture-of-Experts (DMoE) is a modular architecture for parametric knowledge injection.<n>DMoE converts external knowledge corpora into independently updatable expert modules and uses a lightweight uncertainty-aware router to activate relevant experts.<n>Experiments on knowledge-intensive benchmarks show that DMoE consistently improves answer quality over retrieval and adapter-based baselines.
  arXiv  Detail & Related papers  (2026-06-12T08:21:28Z)
• MERIT: Memory-Enhanced Retrieval for Interpretable Knowledge Tracing [23.48465917752293]
  Large Language Models (LLMs) offer strong reasoning capabilities but struggle with limited context windows and hallucinations.<n>We propose MERIT (Memory-Enhanced Retrieval for Interpretable Knowledge Tracing), a training-free framework combining frozen LLM reasoning with structured pedagogical memory.<n>By grounding the LLM in interpretable memory, MERIT achieves state-of-the-art performance on real-world datasets without gradient updates.
  arXiv  Detail & Related papers  (2026-03-03T10:07:38Z)
• How Much Reasoning Do Retrieval-Augmented Models Add beyond LLMs? A Benchmarking Framework for Multi-Hop Inference over Hybrid Knowledge [26.8497597881469]
  HybridRAG-Bench is a framework for constructing benchmarks to evaluate retrieval-intensive, multi-hop reasoning over hybrid knowledge.<n>We show that HybridRAG-Bench rewards genuine retrieval and reasoning rather than parametric recall, offering a principled testbed for evaluating hybrid knowledge-augmented reasoning systems.
  arXiv  Detail & Related papers  (2026-02-10T19:04:01Z)
• Multi-hop Reasoning via Early Knowledge Alignment [68.28168992785896]
  Early Knowledge Alignment (EKA) aims to align Large Language Models with contextually relevant retrieved knowledge.<n>EKA significantly improves retrieval precision, reduces cascading errors, and enhances both performance and efficiency.<n>EKA proves effective as a versatile, training-free inference strategy that scales seamlessly to large models.
  arXiv  Detail & Related papers  (2025-12-23T08:14:44Z)
• RePo: Language Models with Context Re-Positioning [10.269249887819988]
  In-context learning is fundamental to modern Large Language Models (LLMs)<n> prevailing architectures impose a rigid and fixed contextual structure by assigning linear or constant positional indices.<n>We propose RePo, a novel mechanism that reduces extraneous load via context re-positioning.
  arXiv  Detail & Related papers  (2025-12-16T13:30:30Z)
• Representation Interventions Enable Lifelong Unstructured Knowledge Control [54.86207134539453]
  Large language models (LLMs) often produce incorrect or outdated content. Updating their knowledge efficiently and accurately without costly retraining is a major challenge.<n>We introduce RILKE, a robust and scalable method that treats knowledge control as interventions within the model's representation space.<n>During training, RILKE learns paraphrase-robust and edit-localized modules that limit each update to a low-dimensional subspace to minimize cross-edit interference.<n>In inference, a query-adaptive router selects the appropriate module to guide the model's generation.
  arXiv  Detail & Related papers  (2025-11-25T22:15:00Z)
• MemOS: A Memory OS for AI System [116.87568350346537]
  Large Language Models (LLMs) have become an essential infrastructure for Artificial General Intelligence (AGI)<n>Existing models mainly rely on static parameters and short-lived contextual states, limiting their ability to track user preferences or update knowledge over extended periods.<n>MemOS is a memory operating system that treats memory as a manageable system resource.
  arXiv  Detail & Related papers  (2025-07-04T17:21:46Z)
• Prompting is not Enough: Exploring Knowledge Integration and Controllable Generation [89.65955788873532]
  Open-domain question answering (OpenQA) represents a cornerstone in natural language processing (NLP)<n>We propose a novel framework named GenKI, which aims to improve the OpenQA performance by exploring Knowledge Integration and controllable Generation.
  arXiv  Detail & Related papers  (2025-05-26T08:18:33Z)
• Quantifying Memory Utilization with Effective State-Size [73.52115209375343]
  We develop a measure of textitmemory utilization'<n>This metric is tailored to the fundamental class of systems with textitinput-invariant and textitinput-varying linear operators
  arXiv  Detail & Related papers  (2025-04-28T08:12:30Z)
• Oreo: A Plug-in Context Reconstructor to Enhance Retrieval-Augmented Generation [28.568010424711563]
  Retrieval-Augmented Generation (RAG) aims to augment the capabilities of Large Language Models (LLMs)<n>We introduce a compact, efficient, and pluggable module designed to refine retrieved chunks before using them for generation.
  arXiv  Detail & Related papers  (2025-02-18T16:38:39Z)
• Parametric Retrieval Augmented Generation [32.29608109539912]
  Parametric RAG is a new RAG paradigm that integrates external knowledge directly into the parameters of feed-forward networks.<n>It substantially enhances both the effectiveness and efficiency of knowledge augmentation in large language models.
  arXiv  Detail & Related papers  (2025-01-27T10:04:49Z)
• GIVE: Structured Reasoning of Large Language Models with Knowledge Graph Inspired Veracity Extrapolation [108.2008975785364]
  Graph Inspired Veracity Extrapolation (GIVE) is a novel reasoning method that merges parametric and non-parametric memories to improve accurate reasoning with minimal external input.<n>GIVE guides the LLM agent to select the most pertinent expert data (observe), engage in query-specific divergent thinking (reflect), and then synthesize this information to produce the final output (speak)
  arXiv  Detail & Related papers  (2024-10-11T03:05:06Z)
• Robust and Scalable Model Editing for Large Language Models [75.95623066605259]
  We propose EREN (Edit models by REading Notes) to improve the scalability and robustness of LLM editing. Unlike existing techniques, it can integrate knowledge from multiple edits, and correctly respond to syntactically similar but semantically unrelated inputs.
  arXiv  Detail & Related papers  (2024-03-26T06:57:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.