Knowledgeless Language Models: Suppressing Parametric Recall for Evidence-Grounded Language Modeling

• URL: http://arxiv.org/abs/2607.12831v1
• Date: Tue, 14 Jul 2026 14:48:53 GMT
• Title: Knowledgeless Language Models: Suppressing Parametric Recall for Evidence-Grounded Language Modeling
• Abstract summary: We introduce Knowledge--''Less'' Language Models (KLLMs)<n> KLLMs are pretrained on corpora in which named entities are anonymized.<n>We show that KLLMs rely less on parametric knowledge and more on external evidence, leading to improved reliability under realistic conditions.
• Score: 21.14539989233461
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Language models encode substantial factual knowledge in their parameters, which can lead to unreliable behavior when this knowledge is outdated, incomplete, or misaligned with the provided context. In this work, we study whether modifying the pretraining signal can systematically shift models away from parametric recall and toward evidence-grounded reasoning. We introduce Knowledge--''Less'' Language Models (KLLMs), a fundamentally different epistemic training paradigm for LLMs, which are pretrained on corpora in which named entities are anonymized, thereby removing a primary channel for entity-linked factual supervision. This intervention substantially reduces closed-book factual recall, while often improving performance on tasks where relevant information is provided as context. Across multiple model scales, KLLMs consistently outperform matched baselines on contextual question answering, fact verification, and hallucination detection benchmarks. Crucially, in retrieval-grounded settings with imperfect evidence, KLLMs show improved robustness and achieve up to 20--25\% relative gains over standard language models. They further exhibit better calibration, with improved ECE, Brier score, and AUROC, as well as more reliable abstention behavior. Our results demonstrate that suppressing entity-linked supervision during pretraining induces a shift in epistemic behavior: KLLMs rely less on parametric knowledge and more on external evidence, leading to improved reliability under realistic conditions. This suggests that pretraining-time control over knowledge acquisition can complement retrieval-augmented and tool-based systems by providing a more evidence-sensitive base model.

Related papers

• Evaluating and Improving Evidence-Grounded Fact-Checking in LLMs via Multi-Round Evidence Ablation [28.999694748110404]
  We show that off-the-shelf fact-checking systems rely more on their parametric knowledge than on the evidence provided.<n>We propose REAL (Rigorous Evidence Ablation Learning), a training framework that promotes evidence-dependent verification.<n>Our findings highlight that strong fact-checking performance can still coexist with weak evidence dependency.
  arXiv  Detail & Related papers  (2026-09-08T16:04:20Z)
• On Predicting the Post-training Potential of Pre-trained LLMs [60.07459271263409]
  We introduce a new task of predicting post-training potential - forecasting a base model's performance before post-training.<n>We propose RuDE, a unified framework that bypasses the generation gap of base models by leveraging response discrimination.<n>Experiments demonstrate a correlation greater than 90% with post-training performance.
  arXiv  Detail & Related papers  (2026-05-12T11:33:49Z)
• REFLEX: Self-Refining Explainable Fact-Checking via Disentangling Truth into Style and Substance [14.932352020762991]
  We propose REason-guided Fact-checking with Latent EXplanations REFLEX paradigm.<n>It is a plug-and-play, self-refining paradigm that leverages the internal knowledge in backbone model to improve both verdict accuracy and explanation quality.<n>With only 465 self-refined training samples, RELFEX achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2025-11-25T12:06:23Z)
• Knowledge-Level Consistency Reinforcement Learning: Dual-Fact Alignment for Long-Form Factuality [27.687276551678583]
  Hallucination and factuality deficits remain key obstacles to the reliability of large language models.<n>We propose a novel framework that focuses on the knowledge consistency between the policy model's expressed knowledge and the base model's parametric knowledge.
  arXiv  Detail & Related papers  (2025-09-28T09:23:06Z)
• TRAIL: Joint Inference and Refinement of Knowledge Graphs with Large Language Models [5.678291291711662]
  TRAIL is a novel, unified framework for Thinking, Reasoning, And Incremental Learning.<n>It couples joint inference and dynamic KG refinement with large language models.<n>Extensive experiments on multiple benchmarks demonstrate that TRAIL outperforms existing KG-augmented and retrieval-augmented LLM baselines by 3% to 13%.
  arXiv  Detail & Related papers  (2025-08-06T14:25:05Z)
• UniErase: Towards Balanced and Precise Unlearning in Language Models [69.04923022755547]
  Large language models (LLMs) require iterative updates to address the outdated information problem.<n>UniErase is a novel unlearning framework that demonstrates precision and balanced performances between knowledge unlearning and ability retaining.
  arXiv  Detail & Related papers  (2025-05-21T15:53:28Z)
• The First Few Tokens Are All You Need: An Efficient and Effective Unsupervised Prefix Fine-Tuning Method for Reasoning Models [69.798277882245]
  We introduce Unsupervised Prefix Fine-Tuning (UPFT) to enhance large language models' reasoning efficiency.<n>UPFT removes the need for labeled data or exhaustive sampling.<n> Experiments show that UPFT matches the performance of supervised methods.
  arXiv  Detail & Related papers  (2025-03-04T18:56:03Z)
• Enhancing LLM Reliability via Explicit Knowledge Boundary Modeling [41.19330514054401]
  Large language models (LLMs) are prone to hallucination stemming from misaligned self-awareness.<n>We propose the Explicit Knowledge Boundary Modeling framework to integrate fast and slow reasoning systems to harmonize reliability and usability.
  arXiv  Detail & Related papers  (2025-03-04T03:16:02Z)
• The Reliability Paradox: Exploring How Shortcut Learning Undermines Language Model Calibration [5.616884466478886]
  Pre-trained language models (PLMs) have enabled significant performance gains in the field of natural language processing.<n>Recent studies have found PLMs to suffer from miscalibration, indicating a lack of accuracy in the confidence estimates provided by these models.<n>This paper investigates whether lower calibration error implies reliable decision rules for a language model.
  arXiv  Detail & Related papers  (2024-12-17T08:04:28Z)
• Improving Language Models Meaning Understanding and Consistency by Learning Conceptual Roles from Dictionary [65.268245109828]
  Non-human-like behaviour of contemporary pre-trained language models (PLMs) is a leading cause undermining their trustworthiness. A striking phenomenon is the generation of inconsistent predictions, which produces contradictory results. We propose a practical approach that alleviates the inconsistent behaviour issue by improving PLM awareness.
  arXiv  Detail & Related papers  (2023-10-24T06:15:15Z)
• ReEval: Automatic Hallucination Evaluation for Retrieval-Augmented Large Language Models via Transferable Adversarial Attacks [91.55895047448249]
  This paper presents ReEval, an LLM-based framework using prompt chaining to perturb the original evidence for generating new test cases. We implement ReEval using ChatGPT and evaluate the resulting variants of two popular open-domain QA datasets. Our generated data is human-readable and useful to trigger hallucination in large language models.
  arXiv  Detail & Related papers  (2023-10-19T06:37:32Z)
• Large Language Models with Controllable Working Memory [64.71038763708161]
  Large language models (LLMs) have led to a series of breakthroughs in natural language processing (NLP) What further sets these models apart is the massive amounts of world knowledge they internalize during pretraining. How the model's world knowledge interacts with the factual information presented in the context remains under explored.
  arXiv  Detail & Related papers  (2022-11-09T18:58:29Z)
• NoiER: An Approach for Training more Reliable Fine-TunedDownstream Task Models [54.184609286094044]
  We propose noise entropy regularisation (NoiER) as an efficient learning paradigm that solves the problem without auxiliary models and additional data. The proposed approach improved traditional OOD detection evaluation metrics by 55% on average compared to the original fine-tuned models.
  arXiv  Detail & Related papers  (2021-08-29T06:58:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.