What are They Thinking? Delineation, Probing and Tracking of Concepts in LLMs

• URL: http://arxiv.org/abs/2605.28823v1
• Date: Tue, 07 Apr 2026 03:50:09 GMT
• Title: What are They Thinking? Delineation, Probing and Tracking of Concepts in LLMs
• Abstract summary: We show how to develop probes that detect the presence or absence of a broad set of concepts.<n>This is done with four separate concepts and three different LLMs.<n>When this process is scaled to many more concepts, it will create the ability to easily monitor new models.
• Score: 5.802955093242262
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As the influence of LLMs expands, it is imperative to gain insight into their decisions. One way to do that is to develop probes that detect the presence or absence of a broad set of concepts within the embeddings computed in an LLM - which is what we might say a model is "thinking" about. Such probes should be low-cost and easily applicable to any LLM, so that monitoring for many concepts is possible during normal operation. In this paper, we take the first steps towards developing the capability of creating many such probes by defining and executing examples of the key tasks needed: first, the careful delineation of a concept through the creation of a dataset with the concept both present and then absent. Then, the training and testing of a set of linear probes to detect the concept on any layer of an LLM, including an exploration of the complexity of the probe needed. Finally, we show that such probes can track concepts across larger contexts. This is done with four separate concepts and three different LLMs. When this process is scaled to many more concepts, it will create the ability to easily monitor new models.

Related papers

• Geo-Spatial Concept Probing of Large Language Models: Abstraction, Compositionality, and Grounding [1.6658912537684456]
  Large Language Models (LLMs) still struggle with genuine concept understanding.<n>We set up a concept-centric benchmark, targeting spatial concepts such as direction, distance, topology, and their compositions.<n>We conduct extensive experiments across multiple LLM architectures and training regimes to analyze how model scale and design impact conceptual understanding.
  arXiv  Detail & Related papers  (2026-08-07T15:46:38Z)
• Symbolic and Abstractive Reasoning with Complex Visual Queries [57.59111649292774]
  We explore a novel abstract data type termed complex visual query (CVQ)<n>CVQ probes symbolic and abstractive reasoning, which is a critical yet underexplored dimension of human-like neuro-symbolic reasoning for MLLMs.<n>We introduce a two-stage training framework that progressively equips MLLMs with robust visual reasoning capabilities.
  arXiv  Detail & Related papers  (2026-06-08T08:30:51Z)
• Concept Component Analysis: A Principled Approach for Concept Extraction in LLMs [51.378834857406325]
  Mechanistic interpretability seeks to mitigate the issues through extracts from large language models.<n>Sparse autoencoders (SAEs) have emerged as a popular approach for extracting interpretable and monosemantic concepts.<n>We show that SAEs suffer from a fundamental theoretical ambiguity: the well-defined correspondence between LLM representations and human-interpretable concepts remains unclear.
  arXiv  Detail & Related papers  (2026-01-28T09:27:05Z)
• Rethinking Reinforcement fine-tuning of LLMs: A Multi-armed Bandit Learning Perspective [54.209612511049734]
  Two fundamental questions still lack a clear understanding: 1) what is the role of each optimizing choice? 2) which ones are the bottlenecks?<n>This paper aims to shed light on them, and it faces the challenge of several entangled confounding factors in the fine-tuning process.
  arXiv  Detail & Related papers  (2026-01-21T02:37:44Z)
• Satori: Reinforcement Learning with Chain-of-Action-Thought Enhances LLM Reasoning via Autoregressive Search [57.28671084993782]
  Large language models (LLMs) have demonstrated remarkable reasoning capabilities across diverse domains.<n>Recent studies have shown that increasing test-time computation enhances LLMs' reasoning capabilities.<n>We propose a two-stage training paradigm: 1) a small-scale format tuning stage to internalize the COAT reasoning format and 2) a large-scale self-improvement stage leveraging reinforcement learning.
  arXiv  Detail & Related papers  (2025-02-04T17:26:58Z)
• CBVLM: Training-free Explainable Concept-based Large Vision Language Models for Medical Image Classification [7.641602003136712]
  Concept Bottleneck Models (CBMs) tackle the latter by constraining the model output on a set of predefined and human-interpretable concepts.<n>We propose a simple, yet effective, methodology, CBVLM, which tackles both of the aforementioned challenges.<n>We validate our approach with extensive experiments across four medical datasets and twelve LVLMs.
  arXiv  Detail & Related papers  (2025-01-21T16:38:04Z)
• From Words to Actions: Unveiling the Theoretical Underpinnings of LLM-Driven Autonomous Systems [59.40480894948944]
  Large language model (LLM) empowered agents are able to solve decision-making problems in the physical world. Under this model, the LLM Planner navigates a partially observable Markov decision process (POMDP) by iteratively generating language-based subgoals via prompting. We prove that the pretrained LLM Planner effectively performs Bayesian aggregated imitation learning (BAIL) through in-context learning.
  arXiv  Detail & Related papers  (2024-05-30T09:42:54Z)
• Exploring Concept Depth: How Large Language Models Acquire Knowledge and Concept at Different Layers? [57.04803703952721]
  Large language models (LLMs) have shown remarkable performances across a wide range of tasks.<n>However, the mechanisms by which these models encode tasks of varying complexities remain poorly understood.<n>We introduce the idea of "Concept Depth" to suggest that more complex concepts are typically acquired in deeper layers.
  arXiv  Detail & Related papers  (2024-04-10T14:56:40Z)
• Can LLMs Learn New Concepts Incrementally without Forgetting? [21.95081572612883]
  Large Language Models (LLMs) have achieved remarkable success across various tasks, yet their ability to learn incrementally without forgetting remains underexplored. We introduce Concept-1K, a novel dataset comprising 1,023 recently emerged concepts across diverse domains. Using Concept-1K as a testbed, we aim to answer the question: Can LLMs learn new concepts incrementally without forgetting like humans?'
  arXiv  Detail & Related papers  (2024-02-13T15:29:50Z)
• The Tyranny of Possibilities in the Design of Task-Oriented LLM Systems: A Scoping Survey [1.0489539392650928]
  The paper begins by defining a minimal task-oriented LLM system and exploring the design space of such systems. We discuss a pattern in our results and formulate them into three conjectures. In all, the scoping survey presents seven conjectures that can help guide future research efforts.
  arXiv  Detail & Related papers  (2023-12-29T13:35:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.