Co-LMLM: Continuous-Query Limited Memory Language Models

• URL: http://arxiv.org/abs/2607.07707v1
• Date: Wed, 08 Jul 2026 17:59:45 GMT
• Title: Co-LMLM: Continuous-Query Limited Memory Language Models
• Abstract summary: Limited memory language models externalize factual knowledge during pretraining to a knowledge base (KB)<n>We propose continuous-query LMLM (CO-LMLM), where the KB pairs continuous keys with textual knowledge values.<n>CO-LMLM generates flexible vector queries at minimal cost, while still integrating human-readable and attributable retrieved knowledge into its generation.
• Score: 29.846433433159387
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Limited memory language models (LMLMs) externalize factual knowledge during pretraining to a knowledge base (KB), rather than memorizing it in their weights. During generation, the model then fetches knowledge from the KB as needed. This recently introduced paradigm provides multiple advantages, including knowledge control capabilities that remain beyond conventional LLMs. We propose continuous-query LMLM (CO-LMLM), where the KB pairs continuous keys with textual knowledge values, a significant departure from prior reliance on relational KB and queries. CO-LMLM generates flexible vector queries at minimal cost, while still integrating human-readable and attributable retrieved knowledge into its generation. We pair this design with an annotation pipeline that tags free-form factual spans in arbitrary text, removing prior work's restriction to Wikipedia. Across pretraining on Wikipedia and FineWeb-Edu and at multiple model scales, CO-LMLM outperforms prior LMLMs and vanilla LLMs in both perplexity and factual precision. At 360M scale, this includes lower perplexity than models pretrained on 40x more data, and SimpleQA-verified performance that is in line with gpt-4o-mini and higher than Claude Sonnet 4.5.

Related papers

• KARLA: Knowledge-base Augmented Retrieval for Language Models [12.509786917611125]
  We propose a new method that allows an LLM to automatically pull in factual knowledge from a knowledge base during token generation.<n>This means that (1)factual knowledge in the LLM output can be updated without retraining the LLM, (2)facts in the LLM output can be traced to the knowledge base for transparency and explainability, and (3)smaller models can achieve the same factual accuracy as larger models.
  arXiv  Detail & Related papers  (2026-06-25T09:44:40Z)
• KBSpec: LLM-driven Formal Specification Generation with Evolving Domain Knowledge Base [9.796586560498026]
  We propose KBSpec, which augments large language models (LLMs) with dual-source knowledge of formal specification language corpora.<n>Results show that KBSpec improves verification pass rates by 10-25% over state-of-the-art LLM-based approaches.
  arXiv  Detail & Related papers  (2026-06-19T11:37:00Z)
• Efficient Knowledge Probing of Large Language Models by Adapting Pre-trained Embeddings [27.08405655200845]
  Large language models (LLMs) acquire knowledge across diverse domains such as science, history, and geography.<n>These methods require making forward passes through the underlying model to probe the LLM's knowledge about a specific fact.<n>We propose embedding models that effectively encode factual knowledge as text or graphs as proxies for LLMs.
  arXiv  Detail & Related papers  (2025-08-08T05:32:31Z)
• Prompting is not Enough: Exploring Knowledge Integration and Controllable Generation [89.65955788873532]
  Open-domain question answering (OpenQA) represents a cornerstone in natural language processing (NLP)<n>We propose a novel framework named GenKI, which aims to improve the OpenQA performance by exploring Knowledge Integration and controllable Generation.
  arXiv  Detail & Related papers  (2025-05-26T08:18:33Z)
• An Empirical Study of Many-to-Many Summarization with Large Language Models [82.10000188179168]
  Large language models (LLMs) have shown strong multi-lingual abilities, giving them the potential to perform Many-to-many summarization (M2MS) in real applications.<n>This work presents a systematic empirical study on LLMs' M2MS ability.
  arXiv  Detail & Related papers  (2025-05-19T11:18:54Z)
• LightPROF: A Lightweight Reasoning Framework for Large Language Model on Knowledge Graph [57.382255728234064]
  Large Language Models (LLMs) have impressive capabilities in text understanding and zero-shot reasoning.<n> Knowledge Graphs (KGs) provide rich and reliable contextual information for the reasoning process of LLMs.<n>We propose a novel Lightweight and efficient Prompt learning-ReasOning Framework for KGQA (LightPROF)
  arXiv  Detail & Related papers  (2025-04-04T03:03:47Z)
• KBLaM: Knowledge Base augmented Language Model [8.247901935078357]
  We propose Knowledge Base augmented Language Model (KBLaM) for augmenting Large Language Models (LLMs) with external knowledge.<n>KBLaM works with a knowledge base (KB) constructed from a corpus of documents, transforming each piece of knowledge in the KB into continuous key-value vector pairs.<n>Our approach enables integrating a large KB of more than 10K triples into an 8B pre-trained LLM of only 8K context window on one single A100 80GB GPU.
  arXiv  Detail & Related papers  (2024-10-14T12:45:10Z)
• How Reliable are LLMs as Knowledge Bases? Re-thinking Facutality and Consistency [60.25969380388974]
  Large Language Models (LLMs) are increasingly explored as knowledge bases (KBs)<n>Current evaluation methods focus too narrowly on knowledge retention, overlooking other crucial criteria for reliable performance.<n>We propose new criteria and metrics to quantify factuality and consistency, leading to a final reliability score.
  arXiv  Detail & Related papers  (2024-07-18T15:20:18Z)
• Robust and Scalable Model Editing for Large Language Models [75.95623066605259]
  We propose EREN (Edit models by REading Notes) to improve the scalability and robustness of LLM editing. Unlike existing techniques, it can integrate knowledge from multiple edits, and correctly respond to syntactically similar but semantically unrelated inputs.
  arXiv  Detail & Related papers  (2024-03-26T06:57:23Z)
• Knowledge-Augmented Language Model Prompting for Zero-Shot Knowledge Graph Question Answering [7.888547093390469]
  Large Language Models (LLMs) are capable of performing zero-shot closed-book question answering tasks. We propose to augment the knowledge directly in the input of LLMs. Our framework, Knowledge-Augmented language model PromptING (KAPING), requires no model training, thus completely zero-shot.
  arXiv  Detail & Related papers  (2023-06-07T04:15:21Z)
• LLM-Pruner: On the Structural Pruning of Large Language Models [65.02607075556742]
  Large language models (LLMs) have shown remarkable capabilities in language understanding and generation. We tackle the compression of LLMs within the bound of two constraints: being task-agnostic and minimizing the reliance on the original training dataset. Our method, named LLM-Pruner, adopts structural pruning that selectively removes non-critical coupled structures.
  arXiv  Detail & Related papers  (2023-05-19T12:10:53Z)
• Augmented Large Language Models with Parametric Knowledge Guiding [72.71468058502228]
  Large Language Models (LLMs) have significantly advanced natural language processing (NLP) with their impressive language understanding and generation capabilities. Their performance may be suboptimal for domain-specific tasks that require specialized knowledge due to limited exposure to the related data. We propose the novel Parametric Knowledge Guiding (PKG) framework, which equips LLMs with a knowledge-guiding module to access relevant knowledge.
  arXiv  Detail & Related papers  (2023-05-08T15:05:16Z)
• Self-Prompting Large Language Models for Zero-Shot Open-Domain QA [67.08732962244301]
  Open-Domain Question Answering (ODQA) aims to answer questions without explicitly providing background documents. This task becomes notably challenging in a zero-shot setting where no data is available to train tailored retrieval-reader models. We propose a Self-Prompting framework to explicitly utilize the massive knowledge encoded in the parameters of Large Language Models.
  arXiv  Detail & Related papers  (2022-12-16T18:23:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.