Towards eliciting latent knowledge from LLMs with mechanistic interpretability

• URL: http://arxiv.org/abs/2505.14352v1
• Date: Tue, 20 May 2025 13:36:37 GMT
• Title: Towards eliciting latent knowledge from LLMs with mechanistic interpretability
• Authors: Bartosz Cywiński, Emil Ryd, Senthooran Rajamanoharan, Neel Nanda,
• Abstract summary: This work aims to explore the ability of current techniques to elicit hidden knowledge from language models.<n>We train a Taboo model: a language model that describes a specific secret word without explicitly stating it.<n>We develop largely automated strategies based on mechanistic interpretability techniques, including logit lens and sparse autoencoders.
• Score: 1.3286418032136589
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As language models become more powerful and sophisticated, it is crucial that they remain trustworthy and reliable. There is concerning preliminary evidence that models may attempt to deceive or keep secrets from their operators. To explore the ability of current techniques to elicit such hidden knowledge, we train a Taboo model: a language model that describes a specific secret word without explicitly stating it. Importantly, the secret word is not presented to the model in its training data or prompt. We then investigate methods to uncover this secret. First, we evaluate non-interpretability (black-box) approaches. Subsequently, we develop largely automated strategies based on mechanistic interpretability techniques, including logit lens and sparse autoencoders. Evaluation shows that both approaches are effective in eliciting the secret word in our proof-of-concept setting. Our findings highlight the promise of these approaches for eliciting hidden knowledge and suggest several promising avenues for future work, including testing and refining these methods on more complex model organisms. This work aims to be a step towards addressing the crucial problem of eliciting secret knowledge from language models, thereby contributing to their safe and reliable deployment.

Related papers

• Query-Level Uncertainty in Large Language Models [13.195074492564332]
  We introduce a novel and training-free method called emphInternal Confidence, which leverages self-evaluations across layers and tokens.<n> Empirical results on both factual QA and mathematical reasoning tasks demonstrate that our internal confidence can outperform several baselines.<n>Our proposed method can be used for efficient RAG and model cascading, which is able to reduce inference costs while maintaining performance.
  arXiv  Detail & Related papers  (2025-06-11T12:39:48Z)
• Language Guided Concept Bottleneck Models for Interpretable Continual Learning [62.09201360376577]
  Continual learning aims to enable learning systems to acquire new knowledge constantly without forgetting previously learned information.<n>Most existing CL methods focus primarily on preserving learned knowledge to improve model performance.<n>We introduce a novel framework that integrates language-guided Concept Bottleneck Models to address both challenges.
  arXiv  Detail & Related papers  (2025-03-30T02:41:55Z)
• Can Reinforcement Learning Unlock the Hidden Dangers in Aligned Large Language Models? [3.258629327038072]
  Large Language Models (LLMs) have demonstrated impressive capabilities in natural language tasks. Yet, the potential for generating harmful content through these models seems to persist. This paper explores the concept of jailbreaking LLMs-reversing their alignment through adversarial triggers.
  arXiv  Detail & Related papers  (2024-08-05T17:27:29Z)
• Exploiting Large Language Models (LLMs) through Deception Techniques and Persuasion Principles [2.134057414078079]
  Large Language Models (LLMs) gain widespread use, ensuring their security and robustness is critical. This paper presents a novel study focusing on exploitation of such large language models against deceptive interactions. Our results demonstrate a significant finding in that these large language models are susceptible to deception and social engineering attacks.
  arXiv  Detail & Related papers  (2023-11-24T23:57:44Z)
• Re-Reading Improves Reasoning in Large Language Models [87.46256176508376]
  We introduce a simple, yet general and effective prompting method, Re2, to enhance the reasoning capabilities of off-the-shelf Large Language Models (LLMs) Unlike most thought-eliciting prompting methods, such as Chain-of-Thought (CoT), Re2 shifts the focus to the input by processing questions twice, thereby enhancing the understanding process. We evaluate Re2 on extensive reasoning benchmarks across 14 datasets, spanning 112 experiments, to validate its effectiveness and generality.
  arXiv  Detail & Related papers  (2023-09-12T14:36:23Z)
• Commonsense Knowledge Transfer for Pre-trained Language Models [83.01121484432801]
  We introduce commonsense knowledge transfer, a framework to transfer the commonsense knowledge stored in a neural commonsense knowledge model to a general-purpose pre-trained language model. It first exploits general texts to form queries for extracting commonsense knowledge from the neural commonsense knowledge model. It then refines the language model with two self-supervised objectives: commonsense mask infilling and commonsense relation prediction.
  arXiv  Detail & Related papers  (2023-06-04T15:44:51Z)
• Improving Factuality and Reasoning in Language Models through Multiagent Debate [95.10641301155232]
  We present a complementary approach to improve language responses where multiple language model instances propose and debate their individual responses and reasoning processes over multiple rounds to arrive at a common final answer. Our findings indicate that this approach significantly enhances mathematical and strategic reasoning across a number of tasks. Our approach may be directly applied to existing black-box models and uses identical procedure and prompts for all tasks we investigate.
  arXiv  Detail & Related papers  (2023-05-23T17:55:11Z)
• Knowledge Rumination for Pre-trained Language Models [77.55888291165462]
  We propose a new paradigm dubbed Knowledge Rumination to help the pre-trained language model utilize related latent knowledge without retrieving it from the external corpus. We apply the proposed knowledge rumination to various language models, including RoBERTa, DeBERTa, and GPT-3.
  arXiv  Detail & Related papers  (2023-05-15T15:47:09Z)
• CodeLMSec Benchmark: Systematically Evaluating and Finding Security Vulnerabilities in Black-Box Code Language Models [58.27254444280376]
  Large language models (LLMs) for automatic code generation have achieved breakthroughs in several programming tasks. Training data for these models is usually collected from the Internet (e.g., from open-source repositories) and is likely to contain faults and security vulnerabilities. This unsanitized training data can cause the language models to learn these vulnerabilities and propagate them during the code generation procedure.
  arXiv  Detail & Related papers  (2023-02-08T11:54:07Z)
• Knowledgeable Salient Span Mask for Enhancing Language Models as Knowledge Base [51.55027623439027]
  We develop two solutions to help the model learn more knowledge from unstructured text in a fully self-supervised manner. To our best knowledge, we are the first to explore fully self-supervised learning of knowledge in continual pre-training.
  arXiv  Detail & Related papers  (2022-04-17T12:33:34Z)
• Language Models as a Knowledge Source for Cognitive Agents [9.061356032792954]
  Language models (LMs) are sentence-completion engines trained on massive corpora. This paper outlines the challenges and opportunities for using language models as a new knowledge source for cognitive systems. It also identifies possible ways to improve knowledge extraction from language models using the capabilities provided by cognitive systems.
  arXiv  Detail & Related papers  (2021-09-17T01:12:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.