Retrieval is Enough: Training-Free Interpretability with a Tool-Using Agent

• URL: http://arxiv.org/abs/2607.16448v1
• Date: Fri, 17 Jul 2026 18:47:31 GMT
• Title: Retrieval is Enough: Training-Free Interpretability with a Tool-Using Agent
• Abstract summary: Training-based methods are typically more powerful, in part because they leverage large activation datasets during training.<n>This raises a natural question - do they actually surface insights that go beyond what is recoverable from the training dataset itself?<n>We equip an LLM agent with a vector database of activations paired with their textual contexts, along with tools for manipulating activations.<n>The agent iteratively queries the database, forms hypotheses from the retrieved samples, and validates them by constructing linear probes.
• Score: 56.93673666934672
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Interpretability methods for neural network activations span a wide cost spectrum, from cheap, training-free techniques (such as linear probes, PCA, SVD) to more expensive training-based ones (such as SAEs and activation oracles). Training-based methods are typically more powerful, in part because they leverage large activation datasets during training. This raises a natural question - do they actually surface insights that go beyond what is recoverable from the training dataset itself? To address this, we equip an LLM agent with a vector database of activations paired with their textual contexts, along with tools for manipulating activations - projecting out directions in latent space, computing activation differences and averages. The agent iteratively queries the database, forms hypotheses from the retrieved samples, and validates them by constructing linear probes. We call this method HARP, for Hypothesis-driven Agentic Retrieval and Probing. Despite not involving any training, HARP outperforms both activation oracles and SAE-based agents on concept discovery, concept detection, model steering, and secret elicitation. The training-free design also makes HARP substantially cheaper and more flexible: new datasets can be indexed on demand whenever existing ones prove insufficient. More broadly, our results suggest that current training-based methods do not yet extract insights beyond their training data, and motivate benchmarks that explicitly require interpretability methods to demonstrate such insights. We release our code at https://github.com/SriramB-98/HARP

Related papers

• Efficient Agentic Reinforcement Learning with On-Policy Intrinsic Knowledge Boundary Enhancement [16.192937389387982]
  We propose AKBE (Agentic Knowledge Boundary Enhancement), an on-policy method that dynamically probes the model's intrinsic knowledge boundary.<n> Experiments on seven QA benchmarks demonstrate that AKBE improves task accuracy by +1.85 on average and reduces tool calls by 18% over standard agentic RL.
  arXiv  Detail & Related papers  (2026-05-26T12:42:23Z)
• Data-Centric Interpretability for LLM-based Multi-Agent Reinforcement Learning [39.84288631342219]
  We analyze large-scale reinforcement learning training runs from the sophisticated environment of Full-Press Diplomacy.<n>We introduce Meta-Autointerp, a method for grouping SAE features into interpretable hypotheses about training dynamics.<n>We find that even subjectively interesting and seemingly helpful SAE features may be worse than useless to humans.
  arXiv  Detail & Related papers  (2026-02-05T01:21:22Z)
• Investigating task-specific prompts and sparse autoencoders for activation monitoring [0.0]
  Internal activations of language models encode additional information that could be useful for this.<n>Recent work has proposed several approaches which may improve on naive linear probing.<n>We develop and test novel refinements of these methods and compare them against each other.
  arXiv  Detail & Related papers  (2025-04-28T21:28:17Z)
• Leveraging Skills from Unlabeled Prior Data for Efficient Online Exploration [54.8229698058649]
  Unlabeled offline trajectory data can be leveraged to learn efficient exploration strategies.<n>Our method SUPE consistently outperforms prior strategies across a suite of 42 long-horizon, sparse-reward tasks.
  arXiv  Detail & Related papers  (2024-10-23T17:58:45Z)
• BoostAdapter: Improving Vision-Language Test-Time Adaptation via Regional Bootstrapping [64.8477128397529]
  We propose a training-required and training-free test-time adaptation framework. We maintain a light-weight key-value memory for feature retrieval from instance-agnostic historical samples and instance-aware boosting samples. We theoretically justify the rationality behind our method and empirically verify its effectiveness on both the out-of-distribution and the cross-domain datasets.
  arXiv  Detail & Related papers  (2024-10-20T15:58:43Z)
• Reinforcement Learning from Passive Data via Latent Intentions [86.4969514480008]
  We show that passive data can still be used to learn features that accelerate downstream RL. Our approach learns from passive data by modeling intentions. Our experiments demonstrate the ability to learn from many forms of passive data, including cross-embodiment video data and YouTube videos.
  arXiv  Detail & Related papers  (2023-04-10T17:59:05Z)
• Offline Robot Reinforcement Learning with Uncertainty-Guided Human Expert Sampling [11.751910133386254]
  Recent advances in batch (offline) reinforcement learning have shown promising results in learning from available offline data. We propose a novel approach that uses uncertainty estimation to trigger the injection of human demonstration data. Our experiments show that this approach is more sample efficient when compared to a naive way of combining expert data with data collected from a sub-optimal agent.
  arXiv  Detail & Related papers  (2022-12-16T01:41:59Z)
• Extremely Simple Activation Shaping for Out-of-Distribution Detection [10.539058676970267]
  Out-of-distribution (OOD) detection is an important area that stress-tests a model's ability to handle unseen situations. Existing OOD detection methods either incur extra training steps, additional data or make nontrivial modifications to the trained network. We propose an extremely simple, post-hoc, on-the-fly activation shaping method, ASH, where a large portion of a sample's activation at a late layer is removed. Experiments show that such a simple treatment enhances in-distribution and out-of-distribution distinction so as to allow state-of-the-art OOD
  arXiv  Detail & Related papers  (2022-09-20T17:09:49Z)
• Retrieval-Augmented Reinforcement Learning [63.32076191982944]
  We train a network to map a dataset of past experiences to optimal behavior. The retrieval process is trained to retrieve information from the dataset that may be useful in the current context. We show that retrieval-augmented R2D2 learns significantly faster than the baseline R2D2 agent and achieves higher scores.
  arXiv  Detail & Related papers  (2022-02-17T02:44:05Z)
• Learning Dexterous Manipulation from Suboptimal Experts [69.8017067648129]
  Relative Entropy Q-Learning (REQ) is a simple policy algorithm that combines ideas from successful offline and conventional RL algorithms. We show how REQ is also effective for general off-policy RL, offline RL, and RL from demonstrations.
  arXiv  Detail & Related papers  (2020-10-16T18:48:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.