KARLA: Knowledge-base Augmented Retrieval for Language Models

• URL: http://arxiv.org/abs/2606.26807v1
• Date: Thu, 25 Jun 2026 09:44:40 GMT
• Title: KARLA: Knowledge-base Augmented Retrieval for Language Models
• Abstract summary: We propose a new method that allows an LLM to automatically pull in factual knowledge from a knowledge base during token generation.<n>This means that (1)factual knowledge in the LLM output can be updated without retraining the LLM, (2)facts in the LLM output can be traced to the knowledge base for transparency and explainability, and (3)smaller models can achieve the same factual accuracy as larger models.
• Score: 12.509786917611125
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We propose a new method that allows an LLM to automatically pull in factual knowledge from a knowledge base during token generation. This means that (1)~factual knowledge in the LLM output can be updated without retraining the LLM, (2)~facts in the LLM output can be traced to the knowledge base for transparency and explainability, and (3)~smaller models can achieve the same factual accuracy as larger models. Our core idea is to train the model to produce special tokens that trigger a query to the knowledge base. Our experiments show that our method improves factual grounding in both short and long-form generation, and allows factual revisions to take effect through KB edits rather than parameter updates.

Related papers

• PASTA: A Paraphrasing And Self-Training Approach for Knowledge Updating in LLMs [11.157564521901634]
  PASTA is a framework for integrating detailed factual information from news articles as new knowledge into Large Language Models.<n>Our framework combines data augmentation, question-answering generation, and a novel self-learning DPO process.<n>It achieves remarkable improvement from 0.02 to 0.82 accuracy while maintaining general language capabilities.
  arXiv  Detail & Related papers  (2026-06-27T13:02:13Z)
• How new data permeates LLM knowledge and how to dilute it [19.96863816288517]
  Large language models learn and continually learn through the accumulation of gradient-based updates.<n>We demonstrate that when learning new information, LLMs exhibit a "priming" effect: learning a new fact can cause the model to inappropriately apply that knowledge in unrelated contexts.<n>We show that the degree of priming after learning new information can be predicted by measuring the token probability of key words before learning.
  arXiv  Detail & Related papers  (2025-04-13T11:25:04Z)
• Efficient Tuning of Large Language Models for Knowledge-Grounded Dialogue Generation [21.52726424882653]
  We introduce KEDiT, an efficient method for fine-tuning large language models for knowledge-grounded dialogue generation.<n> KEDiT operates in two main phases: first, it employs an information bottleneck to compress retrieved knowledge into learnable parameters, retaining essential information while minimizing computational overhead.<n> experimental results on the Wizard of Wikipedia and a newly constructed PubMed-Dialog dataset demonstrate that KEDiT excels in generating contextually relevant and informative responses.
  arXiv  Detail & Related papers  (2025-04-10T13:54:36Z)
• Enhancing LLM Knowledge Learning through Generalization [73.16975077770765]
  We show that an LLM's ability to continually predict the same factual knowledge tokens given diverse paraphrased contexts is positively correlated with its capacity to extract that knowledge via question-answering.<n>We propose two strategies to enhance LLMs' ability to predict the same knowledge tokens given varied contexts, thereby enhancing knowledge acquisition.
  arXiv  Detail & Related papers  (2025-03-05T17:56:20Z)
• Mitigating Heterogeneous Token Overfitting in LLM Knowledge Editing [31.871873092103684]
  Large language models (LLMs) have achieved remarkable performance on various natural language tasks.<n>They are trained on static corpora and their knowledge can become outdated quickly in the fast-changing world.<n>This motivates the development of knowledge editing (KE) to update specific knowledge in LLMs without changing unrelated others or compromising their pre-trained capabilities.
  arXiv  Detail & Related papers  (2025-02-02T00:10:51Z)
• Robust and Scalable Model Editing for Large Language Models [75.95623066605259]
  We propose EREN (Edit models by REading Notes) to improve the scalability and robustness of LLM editing. Unlike existing techniques, it can integrate knowledge from multiple edits, and correctly respond to syntactically similar but semantically unrelated inputs.
  arXiv  Detail & Related papers  (2024-03-26T06:57:23Z)
• Editing Conceptual Knowledge for Large Language Models [65.38231526537476]
  This paper pioneers the investigation of editing conceptual knowledge for Large Language Models (LLMs) We construct a novel benchmark dataset ConceptEdit and establish a suite of new metrics for evaluation. experimental results reveal that, although existing editing methods can efficiently modify concept-level definition to some extent, they also have the potential to distort the related instantial knowledge.
  arXiv  Detail & Related papers  (2024-03-10T16:57:10Z)
• Learning to Edit: Aligning LLMs with Knowledge Editing [101.96620267293731]
  We propose a Learning to Edit (LTE) framework, focusing on teaching large language models to apply updated knowledge into input questions. LTE features a two-phase process: (i) the Alignment Phase, which fine-tunes LLMs on a meticulously curated parallel dataset to make reliable, in-scope edits. We demonstrate LTE's superiority in knowledge editing performance, robustness in both batch and sequential editing, minimal interference on general tasks, and rapid editing speeds.
  arXiv  Detail & Related papers  (2024-02-19T07:45:17Z)
• Supervised Knowledge Makes Large Language Models Better In-context Learners [94.89301696512776]
  Large Language Models (LLMs) exhibit emerging in-context learning abilities through prompt engineering. The challenge of improving the generalizability and factuality of LLMs in natural language understanding and question answering remains under-explored. We propose a framework that enhances the reliability of LLMs as it: 1) generalizes out-of-distribution data, 2) elucidates how LLMs benefit from discriminative models, and 3) minimizes hallucinations in generative tasks.
  arXiv  Detail & Related papers  (2023-12-26T07:24:46Z)
• Forgetting before Learning: Utilizing Parametric Arithmetic for Knowledge Updating in Large Language Models [53.52344131257681]
  We propose a new paradigm for fine-tuning called F-Learning, which employs parametric arithmetic to facilitate the forgetting of old knowledge and learning of new knowledge. Experimental results on two publicly available datasets demonstrate that our proposed F-Learning can obviously improve the knowledge updating performance of both full fine-tuning and LoRA fine-tuning.
  arXiv  Detail & Related papers  (2023-11-14T09:12:40Z)
• DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models [79.01926242857613]
  Large language models (LLMs) are prone to hallucinations, generating content that deviates from facts seen during pretraining. We propose a simple decoding strategy for reducing hallucinations with pretrained LLMs. We find that this Decoding by Contrasting Layers (DoLa) approach is able to better surface factual knowledge and reduce the generation of incorrect facts.
  arXiv  Detail & Related papers  (2023-09-07T17:45:31Z)
• DocTER: Evaluating Document-based Knowledge Editing [53.14000724633775]
  We explore knowledge editing using easily accessible documents instead of manually labeled factual triples.<n>A comprehensive four-perspective evaluation is introduced: Edit Success, Locality, Reasoning, and Cross-lingual Transfer.<n>Experiments on popular knowledge editing methods demonstrate that editing with documents presents significantly greater challenges than using triples.
  arXiv  Detail & Related papers  (2023-08-19T09:17:19Z)
• Give Us the Facts: Enhancing Large Language Models with Knowledge Graphs for Fact-aware Language Modeling [34.59678835272862]
  ChatGPT, a representative large language model (LLM), has gained considerable attention due to its powerful emergent abilities. This paper proposes to enhance LLMs with knowledge graph-enhanced large language models (KGLLMs) KGLLM provides a solution to enhance LLMs' factual reasoning ability, opening up new avenues for LLM research.
  arXiv  Detail & Related papers  (2023-06-20T12:21:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.