Probing for Knowledge Attribution in Large Language Models

• URL: http://arxiv.org/abs/2602.22787v1
• Date: Thu, 26 Feb 2026 09:21:12 GMT
• Title: Probing for Knowledge Attribution in Large Language Models
• Authors: Ivo Brink, Alexander Boer, Dennis Ulmer,
• Abstract summary: Large language models (LLMs) often generate fluent but unfounded claims, or hallucinations.<n> Proper mitigation depends on knowing whether a model's answer is based on the prompt or its internal weights.<n>We show that a probe, a simple linear classifier trained on model hidden representations, can reliably predict contributive attribution.
• Score: 45.47366023067617
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) often generate fluent but unfounded claims, or hallucinations, which fall into two types: (i) faithfulness violations - misusing user context - and (ii) factuality violations - errors from internal knowledge. Proper mitigation depends on knowing whether a model's answer is based on the prompt or its internal weights. This work focuses on the problem of contributive attribution: identifying the dominant knowledge source behind each output. We show that a probe, a simple linear classifier trained on model hidden representations, can reliably predict contributive attribution. For its training, we introduce AttriWiki, a self-supervised data pipeline that prompts models to recall withheld entities from memory or read them from context, generating labelled examples automatically. Probes trained on AttriWiki data reveal a strong attribution signal, achieving up to 0.96 Macro-F1 on Llama-3.1-8B, Mistral-7B, and Qwen-7B, transferring to out-of-domain benchmarks (SQuAD, WebQuestions) with 0.94-0.99 Macro-F1 without retraining. Attribution mismatches raise error rates by up to 70%, demonstrating a direct link between knowledge source confusion and unfaithful answers. Yet, models may still respond incorrectly even when attribution is correct, highlighting the need for broader detection frameworks.

Related papers

• LLM Microscope: What Model Internals Reveal About Answer Correctness and Context Utilization [9.410181019585822]
  We operationalize interpretability methods to ascertain whether we can predict the correctness of model outputs.<n>We consider correct, incorrect, and irrelevant context and introduce metrics to distinguish amongst them.<n>Our model-internals-based metric significantly outperforms prompting baselines at distinguishing between correct and incorrect context.
  arXiv  Detail & Related papers  (2025-10-05T03:14:05Z)
• Erasing Without Remembering: Implicit Knowledge Forgetting in Large Language Models [81.62767292169225]
  We investigate knowledge forgetting in large language models with a focus on its generalisation.<n>We propose PerMU, a novel probability perturbation-based unlearning paradigm.<n>Experiments are conducted on a diverse range of datasets, including TOFU, Harry Potter, ZsRE, WMDP, and MUSE.
  arXiv  Detail & Related papers  (2025-02-27T11:03:33Z)
• Eliciting Latent Knowledge from Quirky Language Models [1.8035046415192353]
  Eliciting Latent Knowledge aims to find patterns in a capable neural network's activations that robustly track the true state of the world. We introduce 12 datasets and a suite of "quirky" language models (LMs) that are finetuned to make systematic errors when answering questions. We find that, especially in middle layers, linear probes usually report an LM's knowledge independently of what the LM outputs.
  arXiv  Detail & Related papers  (2023-12-02T05:47:22Z)
• R-Tuning: Instructing Large Language Models to Say `I Don't Know' [66.11375475253007]
  Large language models (LLMs) have revolutionized numerous domains with their impressive performance but still face their challenges. Previous instruction tuning methods force the model to complete a sentence no matter whether the model knows the knowledge or not. We present a new approach called Refusal-Aware Instruction Tuning (R-Tuning) Experimental results demonstrate R-Tuning effectively improves a model's ability to answer known questions and refrain from answering unknown questions.
  arXiv  Detail & Related papers  (2023-11-16T08:45:44Z)
• Physics of Language Models: Part 3.1, Knowledge Storage and Extraction [51.68385617116854]
  Large language models (LLMs) can store a vast amount of world knowledge, often extractable via question-answering. We find a strong correlation between the model's ability to extract knowledge and various diversity measures of the training data.
  arXiv  Detail & Related papers  (2023-09-25T17:37:20Z)
• Can Large Language Models Infer Causation from Correlation? [104.96351414570239]
  We test the pure causal inference skills of large language models (LLMs) We formulate a novel task Corr2Cause, which takes a set of correlational statements and determines the causal relationship between the variables. We show that these models achieve almost close to random performance on the task.
  arXiv  Detail & Related papers  (2023-06-09T12:09:15Z)
• Mitigating False-Negative Contexts in Multi-document QuestionAnswering with Retrieval Marginalization [29.797379277423143]
  We develop a new parameterization of set-valued retrieval that properly handles unanswerable queries. We show that marginalizing over this set during training allows a model to mitigate false negatives in annotated supporting evidences. On IIRC, we show that joint modeling with marginalization on alternative contexts improves model performance by 5.5 F1 points and achieves a new state-of-the-art performance of 50.6 F1.
  arXiv  Detail & Related papers  (2021-03-22T23:44:35Z)
• TACRED Revisited: A Thorough Evaluation of the TACRED Relation Extraction Task [80.38130122127882]
  TACRED is one of the largest, most widely used crowdsourced datasets in Relation Extraction (RE) In this paper, we investigate the questions: Have we reached a performance ceiling or is there still room for improvement? We find that label errors account for 8% absolute F1 test error, and that more than 50% of the examples need to be relabeled.
  arXiv  Detail & Related papers  (2020-04-30T15:07:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.