H-Probes: Extracting Hierarchical Structures From Latent Representations of Language Models

• URL: http://arxiv.org/abs/2605.00847v2
• Date: Wed, 06 May 2026 20:35:10 GMT
• Title: H-Probes: Extracting Hierarchical Structures From Latent Representations of Language Models
• Abstract summary: We develop H-probes that extract hierarchical structure from latent representations.<n>In synthetic tree tasks, the H-probes robustly find the subspaces containing hierarchical structure necessary to complete the tasks.<n>We find analogous, though weaker, hierarchical structure in real-world hierarchical contexts such as mathematical reasoning traces.
• Score: 1.7357690546529352
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Representing and navigating hierarchy is a fundamental primitive of reasoning. Large language models have demonstrated proficiency in a wide variety of tasks requiring hierarchical reasoning, but there exists limited analysis on how the models geometrically represent the necessary latent constructions for such thinking. To this end, we develop H-probes, a collection of linear probes that extract hierarchical structure, specifically depth and pairwise distance, from latent representations. In synthetic tree traversal tasks, the H-probes robustly find the subspaces containing hierarchical structure necessary to complete the tasks; furthermore, in comprehensive ablation experiments, we show that these hierarchy-containing subspaces are low-dimensional, causally important for high task performance, and generalize within- and out-of-domain. Furthermore, we find analogous, though weaker, hierarchical structure in real-world hierarchical contexts such as mathematical reasoning traces. These results demonstrate that models represent hierarchy not only at the level of syntax and concepts, but at deeper levels of abstraction -- including the reasoning process itself.

Related papers

• An expressivity analysis of hierarchical modelling in deep transformers via bounded-depth grammars [54.11540943172608]
  Deep neural networks are widely believed to derive their expressive power from their ability to form textbfhierarchical representations<n>In language modeling, textbftransformers have emerged as the dominant architecture, with early layers capturing local syntactic patterns and later layers encoding more complex clause-level dependencies.<n>We argue that these architectures possess the structural capacity to encode abstract grammatical states into low-dimensional, linearly separable subspaces within the residual stream.
  arXiv  Detail & Related papers  (2026-06-16T05:02:13Z)
• SC-Taxo: Hierarchical Taxonomy Generation under Semantic Consistency Constraints using Large Language Models [24.295783768613344]
  A high-quality scientific taxonomy offers a structured and hierarchical representation of a research field.<n>Existing taxonomy generation approaches often suffer from structural inconsistencies and semantic misalignment across hierarchical levels.
  arXiv  Detail & Related papers  (2026-05-01T12:42:44Z)
• TopoOR: A Unified Topological Scene Representation for the Operating Room [54.50897207203292]
  Surgical Scene Graphs abstract the complexity of surgical operating rooms (OR) into a structure of entities and their relations.<n>We introduce TopoOR, a new paradigm that models multimodal operating rooms as a higher-order structure.<n>We also propose a higher-order attention mechanism that explicitly preserves manifold structure and modality-specific features.
  arXiv  Detail & Related papers  (2026-03-10T10:19:42Z)
• Emergent Structured Representations Support Flexible In-Context Inference in Large Language Models [77.98801218316505]
  Large language models (LLMs) exhibit emergent behaviors suggestive of human-like reasoning.<n>We investigate the internal processing of LLMs during in-context concept inference.
  arXiv  Detail & Related papers  (2026-02-08T03:14:39Z)
• Provable Learning of Random Hierarchy Models and Hierarchical Shallow-to-Deep Chaining [58.69016084278948]
  We consider a hierarchical context-free grammar introduced by arXiv:2307.02129 and conjectured to separate deep and shallow networks.<n>We prove that, under mild conditions, a deep convolutional network can be efficiently trained to learn this function class.
  arXiv  Detail & Related papers  (2026-01-27T16:19:54Z)
• Parrot Mind: Towards Explaining the Complex Task Reasoning of Pretrained Large Language Models with Template-Content Structure [66.33623392497599]
  We show that a structure called template-content structure (T-C structure) can reduce the possible space from exponential level to linear level. We demonstrate that models can achieve task composition, further reducing the space needed to learn from linear to logarithmic.
  arXiv  Detail & Related papers  (2023-10-09T06:57:45Z)
• How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Model [47.617093812158366]
  We introduce the Random Hierarchy Model: a family of synthetic tasks inspired by the hierarchical structure of language and images. We find that deep networks learn the task by developing internal representations invariant to exchanging equivalent groups. Our results indicate how deep networks overcome the curse of dimensionality by building invariant representations.
  arXiv  Detail & Related papers  (2023-07-05T09:11:09Z)
• Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representations [62.65877150123775]
  Causal abstraction is a promising theoretical framework for explainable artificial intelligence. Existing causal abstraction methods require a brute-force search over alignments between the high-level model and the low-level one. We present distributed alignment search (DAS), which overcomes these limitations.
  arXiv  Detail & Related papers  (2023-03-05T00:57:49Z)
• Hierarchical Pyramid Representations for Semantic Segmentation [0.0]
  We learn the structures of objects and the hierarchy among objects because context is based on these intrinsic properties. In this study, we design novel hierarchical, contextual, and multiscale pyramidal representations to capture the properties from an input image. Our proposed method achieves state-of-the-art performance in PASCAL Context.
  arXiv  Detail & Related papers  (2021-04-05T06:39:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.