ExpertLens: Activation steering features are highly interpretable

• URL: http://arxiv.org/abs/2502.15090v4
• Date: Mon, 03 Nov 2025 20:25:57 GMT
• Title: ExpertLens: Activation steering features are highly interpretable
• Authors: Masha Fedzechkina, Eleonora Gualdoni, Sinead Williamson, Katherine Metcalf, Skyler Seto, Barry-John Theobald,
• Abstract summary: We identify neurons responsible for specific concepts using the finding experts'' method from research on activation steering.<n>We find that ExpertLens representations are stable across models and datasets and closely align with human representations inferred from behavioral data.<n>Our findings suggest that ExpertLens is a flexible and lightweight approach for capturing and analyzing model representations.
• Score: 17.8514287071776
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Activation steering methods in large language models (LLMs) have emerged as an effective way to perform targeted updates to enhance generated language without requiring large amounts of adaptation data. We ask whether the features discovered by activation steering methods are interpretable. We identify neurons responsible for specific concepts (e.g., ``cat'') using the ``finding experts'' method from research on activation steering and show that the ExpertLens, i.e., inspection of these neurons provides insights about model representation. We find that ExpertLens representations are stable across models and datasets and closely align with human representations inferred from behavioral data, matching inter-human alignment levels. ExpertLens significantly outperforms the alignment captured by word/sentence embeddings. By reconstructing human concept organization through ExpertLens, we show that it enables a granular view of LLM concept representation. Our findings suggest that ExpertLens is a flexible and lightweight approach for capturing and analyzing model representations.

Related papers

• What if Agents Could Imagine? Reinforcing Open-Vocabulary HOI Comprehension through Generation [35.62323084880028]
  We propose textbfImagineAgent, an agentic framework that harmonizes cognitive reasoning with generative imagination for robust visual understanding.<n>Our method innovatively constructs cognitive maps that explicitly model plausible relationships between detected entities and candidate actions.<n>It dynamically invokes tools including retrieval augmentation, image cropping, and diffusion models to gather domain-specific knowledge and enriched visual evidence.
  arXiv  Detail & Related papers  (2026-02-12T02:51:59Z)
• ForenX: Towards Explainable AI-Generated Image Detection with Multimodal Large Language Models [82.04858317800097]
  We present ForenX, a novel method that not only identifies the authenticity of images but also provides explanations that resonate with human thoughts.<n>ForenX employs the powerful multimodal large language models (MLLMs) to analyze and interpret forensic cues.<n>We introduce ForgReason, a dataset dedicated to descriptions of forgery evidences in AI-generated images.
  arXiv  Detail & Related papers  (2025-08-02T15:21:26Z)
• Localizing Persona Representations in LLMs [5.828323647048382]
  We study how and where personas are encoded in the representation space of large language models (LLMs)<n>We observe overlapping activations for specific ethical perspectives, such as moral nihilism and utilitarianism.<n>In contrast, political ideologies like conservatism and liberalism appear to be represented in more distinct regions.
  arXiv  Detail & Related papers  (2025-05-30T12:46:44Z)
• From Tokens to Thoughts: How LLMs and Humans Trade Compression for Meaning [52.32745233116143]
  Humans organize knowledge into compact categories through semantic compression.<n>Large Language Models (LLMs) demonstrate remarkable linguistic abilities.<n>But whether their internal representations strike a human-like trade-off between compression and semantic fidelity is unclear.
  arXiv  Detail & Related papers  (2025-05-21T16:29:00Z)
• Multimodal LLM Augmented Reasoning for Interpretable Visual Perception Analysis [19.032828729570458]
  We use established principles and explanations from psychology and cognitive science related to complexity in human visual perception. Our study aims to benchmark MLLMs across various explainability principles relevant to visual perception.
  arXiv  Detail & Related papers  (2025-04-16T22:14:27Z)
• Waking Up an AI: A Quantitative Framework for Prompt-Induced Phase Transition in Large Language Models [0.0]
  We propose a two-part framework to investigate what underlies intuitive human thinking. A form of conceptual fusion-current LLMs showed no significant difference in responsiveness between semantically fused and non-fused prompts. Our method may help illuminate key differences in how intuition and conceptual leaps emerge in artificial versus human minds.
  arXiv  Detail & Related papers  (2025-04-16T06:49:45Z)
• I Predict Therefore I Am: Is Next Token Prediction Enough to Learn Human-Interpretable Concepts from Data? [79.01538178959726]
  Large language models (LLMs) have led many to conclude that they exhibit a form of intelligence. We introduce a novel generative model that generates tokens on the basis of human interpretable concepts represented as latent discrete variables.
  arXiv  Detail & Related papers  (2025-03-12T01:21:17Z)
• How Deep is Love in LLMs' Hearts? Exploring Semantic Size in Human-like Cognition [75.11808682808065]
  This study investigates whether large language models (LLMs) exhibit similar tendencies in understanding semantic size. Our findings reveal that multi-modal training is crucial for LLMs to achieve more human-like understanding. Lastly, we examine whether LLMs are influenced by attention-grabbing headlines with larger semantic sizes in a real-world web shopping scenario.
  arXiv  Detail & Related papers  (2025-03-01T03:35:56Z)
• Human-like conceptual representations emerge from language prediction [72.5875173689788]
  We investigated the emergence of human-like conceptual representations within large language models (LLMs)<n>We found that LLMs were able to infer concepts from definitional descriptions and construct representation spaces that converge towards a shared, context-independent structure.<n>Our work supports the view that LLMs serve as valuable tools for understanding complex human cognition and paves the way for better alignment between artificial and human intelligence.
  arXiv  Detail & Related papers  (2025-01-21T23:54:17Z)
• Large Language Models as Neurolinguistic Subjects: Identifying Internal Representations for Form and Meaning [49.60849499134362]
  This study investigates the linguistic understanding of Large Language Models (LLMs) regarding signifier (form) and signified (meaning) Traditional psycholinguistic evaluations often reflect statistical biases that may misrepresent LLMs' true linguistic capabilities. We introduce a neurolinguistic approach, utilizing a novel method that combines minimal pair and diagnostic probing to analyze activation patterns across model layers.
  arXiv  Detail & Related papers  (2024-11-12T04:16:44Z)
• Probing Ranking LLMs: A Mechanistic Analysis for Information Retrieval [20.353393773305672]
  We employ a probing-based analysis to examine neuron activations in ranking LLMs.<n>Our study spans a broad range of feature categories, including lexical signals, document structure, query-document interactions, and complex semantic representations.<n>Our findings offer crucial insights for developing more transparent and reliable retrieval systems.
  arXiv  Detail & Related papers  (2024-10-24T08:20:10Z)
• Mind Scramble: Unveiling Large Language Model Psychology Via Typoglycemia [27.650551131885152]
  Research into large language models (LLMs) has shown promise in addressing complex tasks in the physical world. Studies suggest that powerful LLMs, like GPT-4, are beginning to exhibit human-like cognitive abilities.
  arXiv  Detail & Related papers  (2024-10-02T15:47:25Z)
• Understanding the Human-LLM Dynamic: A Literature Survey of LLM Use in Programming Tasks [0.850206009406913]
  Large Language Models (LLMs) are transforming programming practices, offering significant capabilities for code generation activities. This paper focuses on their use in programming tasks, drawing insights from user studies that assess the impact of LLMs on programming tasks.
  arXiv  Detail & Related papers  (2024-10-01T19:34:46Z)
• Human-like object concept representations emerge naturally in multimodal large language models [24.003766123531545]
  We combined behavioral and neuroimaging analyses to explore the relationship between object concept representations in Large Language Models (LLMs) and human cognition.<n>Our findings advance the understanding of machine intelligence and inform the development of more human-like artificial cognitive systems.
  arXiv  Detail & Related papers  (2024-07-01T08:17:19Z)
• Understanding Large Language Model Behaviors through Interactive Counterfactual Generation and Analysis [22.755345889167934]
  We present an interactive visualization system that enables exploration of large language models (LLMs) through counterfactual analysis.<n>Our system features a novel algorithm that generates fluent and semantically meaningful counterfactuals.<n>A user study with LLM practitioners and interviews with experts demonstrate the system's usability and effectiveness.
  arXiv  Detail & Related papers  (2024-04-23T19:57:03Z)
• Interpretable Brain-Inspired Representations Improve RL Performance on Visual Navigation Tasks [0.0]
  We show how the method of slow feature analysis (SFA) overcomes both limitations by generating interpretable representations of visual data. We employ SFA in a modern reinforcement learning context, analyse and compare representations and illustrate where hierarchical SFA can outperform other feature extractors on navigation tasks.
  arXiv  Detail & Related papers  (2024-02-19T11:35:01Z)
• Understanding Before Recommendation: Semantic Aspect-Aware Review Exploitation via Large Language Models [53.337728969143086]
  Recommendation systems harness user-item interactions like clicks and reviews to learn their representations. Previous studies improve recommendation accuracy and interpretability by modeling user preferences across various aspects and intents. We introduce a chain-based prompting approach to uncover semantic aspect-aware interactions.
  arXiv  Detail & Related papers  (2023-12-26T15:44:09Z)
• Divergences between Language Models and Human Brains [59.100552839650774]
  We systematically explore the divergences between human and machine language processing.<n>We identify two domains that LMs do not capture well: social/emotional intelligence and physical commonsense.<n>Our results show that fine-tuning LMs on these domains can improve their alignment with human brain responses.
  arXiv  Detail & Related papers  (2023-11-15T19:02:40Z)
• Interpreting Pretrained Language Models via Concept Bottlenecks [55.47515772358389]
  Pretrained language models (PLMs) have made significant strides in various natural language processing tasks. The lack of interpretability due to their black-box'' nature poses challenges for responsible implementation. We propose a novel approach to interpreting PLMs by employing high-level, meaningful concepts that are easily understandable for humans.
  arXiv  Detail & Related papers  (2023-11-08T20:41:18Z)
• Towards Concept-Aware Large Language Models [56.48016300758356]
  Concepts play a pivotal role in various human cognitive functions, including learning, reasoning and communication. There is very little work on endowing machines with the ability to form and reason with concepts. In this work, we analyze how well contemporary large language models (LLMs) capture human concepts and their structure.
  arXiv  Detail & Related papers  (2023-11-03T12:19:22Z)
• Evaluating and Explaining Large Language Models for Code Using Syntactic Structures [74.93762031957883]
  This paper introduces ASTxplainer, an explainability method specific to Large Language Models for code. At its core, ASTxplainer provides an automated method for aligning token predictions with AST nodes. We perform an empirical evaluation on 12 popular LLMs for code using a curated dataset of the most popular GitHub projects.
  arXiv  Detail & Related papers  (2023-08-07T18:50:57Z)
• Aligning Large Language Models with Human: A Survey [53.6014921995006]
  Large Language Models (LLMs) trained on extensive textual corpora have emerged as leading solutions for a broad array of Natural Language Processing (NLP) tasks. Despite their notable performance, these models are prone to certain limitations such as misunderstanding human instructions, generating potentially biased content, or factually incorrect information. This survey presents a comprehensive overview of these alignment technologies, including the following aspects.
  arXiv  Detail & Related papers  (2023-07-24T17:44:58Z)
• Towards A Unified Agent with Foundation Models [18.558328028366816]
  We investigate how to embed and leverage such abilities in Reinforcement Learning (RL) agents. We design a framework that uses language as the core reasoning tool, exploring how this enables an agent to tackle a series of fundamental RL challenges. We demonstrate substantial performance improvements over baselines in exploration efficiency and ability to reuse data from offline datasets.
  arXiv  Detail & Related papers  (2023-07-18T22:37:30Z)
• Explaining Explainability: Towards Deeper Actionable Insights into Deep Learning through Second-order Explainability [70.60433013657693]
  Second-order explainable AI (SOXAI) was recently proposed to extend explainable AI (XAI) from the instance level to the dataset level. We demonstrate for the first time, via example classification and segmentation cases, that eliminating irrelevant concepts from the training set based on actionable insights from SOXAI can enhance a model's performance.
  arXiv  Detail & Related papers  (2023-06-14T23:24:01Z)
• Human Behavioral Benchmarking: Numeric Magnitude Comparison Effects in Large Language Models [4.412336603162406]
  Large Language Models (LLMs) do not differentially represent numbers, which are pervasive in text. In this work, we investigate how well popular LLMs capture the magnitudes of numbers from a behavioral lens.
  arXiv  Detail & Related papers  (2023-05-18T07:50:44Z)
• Explainable Recommender Systems via Resolving Learning Representations [57.24565012731325]
  Explanations could help improve user experience and discover system defects. We propose a novel explainable recommendation model through improving the transparency of the representation learning process.
  arXiv  Detail & Related papers  (2020-08-21T05:30:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.