Abstract representational geometry supports inference in large language models

• URL: http://arxiv.org/abs/2606.23345v1
• Date: Mon, 22 Jun 2026 13:50:34 GMT
• Title: Abstract representational geometry supports inference in large language models
• Abstract summary: Internal mechanisms of large language models remain largely opaque, making it unclear whether they form comparable abstract representations or rely on task-specific statistical regularities when performing comparable reasoning tasks.<n>We report that although LLMs exhibit generalizable reasoning less frequently than humans, when such inference occurs, their internal states exhibit abstract geometric structures that resemble those reported in the hippocampus.
• Score: 9.40587716832914
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: A defining feature of human intelligence is the ability to adapt to changing environments by inferring latent task structure from sparse observations. Neuroscientific research indicates that this capability relies on the hippocampus constructing abstract representations, expressed as low-dimensional, approximately orthogonal manifolds in neural state space. However, the internal mechanisms of large language models (LLMs) remain largely opaque, making it unclear whether they form comparable abstract representations or instead rely on task-specific statistical regularities when performing comparable reasoning tasks. Here we adapt a contextual reversal-learning paradigm to a text-based setting and compare humans and LLMs at both the Behavioural and representational levels. We report that although LLMs exhibit generalizable reasoning less frequently than humans, when such inference occurs, their internal states exhibit abstract geometric structures that resemble those reported in the hippocampus. Notably, this representational geometry is not uniformly distributed but is organized hierarchically across model depth: whereas lower layers show early, stable encoding of stimulus identity, higher layers form a hippocampal-like functional band enriched for abstract context geometry associated with inference. Furthermore, complementary intervention experiments mechanistically implicate geometry in reasoning: task-sequence language modelling induces geometric disentanglement, whereas geometric regularization of higher layers increases the emergence of generalizable inference. Together, these findings establish abstract representational geometry as a mechanistic principle supporting inference in large language models.

Related papers

• An expressivity analysis of hierarchical modelling in deep transformers via bounded-depth grammars [54.11540943172608]
  Deep neural networks are widely believed to derive their expressive power from their ability to form textbfhierarchical representations<n>In language modeling, textbftransformers have emerged as the dominant architecture, with early layers capturing local syntactic patterns and later layers encoding more complex clause-level dependencies.<n>We argue that these architectures possess the structural capacity to encode abstract grammatical states into low-dimensional, linearly separable subspaces within the residual stream.
  arXiv  Detail & Related papers  (2026-06-16T05:02:13Z)
• A Navigable Manifold of Hypothesized Consciousness-Spectrum States in Language Model Representations [0.0]
  We study the geometric structure and dynamics of patterns along this spectrum in transformer embedding spaces.<n>We show that embeddings exhibit a globally organized geometry aligned with this spectrum.<n>Results suggest that embedding spaces encode structured and navigable geometry aligned with a hypothesized consciousness-spectrum taxonomy.
  arXiv  Detail & Related papers  (2026-06-04T04:33:17Z)
• Reasoning emerges from constrained inference manifolds in large language models [98.6966322378456]
  We study reasoning as an intrinsic dynamical process by examining the evolution of internal representations during inference.<n>We find that such geometric compression, although pervasive, is not sufficient for stable or reliable reasoning.
  arXiv  Detail & Related papers  (2026-05-02T10:41:20Z)
• SpatialImaginer: Towards Adaptive Visual Imagination for Spatial Reasoning [67.67774742200626]
  spatial intelligence, which refers to the ability to reason about geometric and physical structure from visual observations, is a core challenge for large language models.<n>We propose a unified multimodal generation framework that integrates textual reasoning with visual imagination.<n>Our framework adopts a divide-and-conquer strategy, using text chain-of-thought for high-level semantic planning and the visual imagination for geometry-sensitive state transformation and consistency preservation.
  arXiv  Detail & Related papers  (2026-04-19T11:21:59Z)
• Latent Structure of Affective Representations in Large Language Models [6.345340156849189]
  We investigate the latent structure of affective representations in large language models (LLMs) using geometric data analysis tools.<n>First, we show that LLMs learn coherent latent representations of affective emotions that align with widely used valence--arousal models from psychology.<n>Second, we find that these representations exhibit nonlinear geometric structure that can nonetheless be well-approximated linearly.<n>Third, we demonstrate that the learned latent representation space can be leveraged to quantify uncertainty in emotion processing tasks.
  arXiv  Detail & Related papers  (2026-04-08T02:13:48Z)
• Emergent Structured Representations Support Flexible In-Context Inference in Large Language Models [77.98801218316505]
  Large language models (LLMs) exhibit emergent behaviors suggestive of human-like reasoning.<n>We investigate the internal processing of LLMs during in-context concept inference.
  arXiv  Detail & Related papers  (2026-02-08T03:14:39Z)
• Visual Language Hypothesis [14.062822951292402]
  We study visual representation learning from a structural and topological perspective.<n>We show that approximating the quotient also places structural demands on the model architecture.
  arXiv  Detail & Related papers  (2025-12-29T09:43:10Z)
• REMA: A Unified Reasoning Manifold Framework for Interpreting Large Language Model [29.40036398095681]
  We define the Reasoning Manifold, a latent low-dimensional geometric structure formed by the internal representations corresponding to all correctly reasoned generations.<n>We build REMA, a framework that explains the origins of failures by quantitatively comparing the spatial relationships of internal model representations corresponding to both erroneous and correct reasoning samples.<n>Our experiments on diverse language and multimodal models and tasks demonstrate the low-dimensional nature of the reasoning manifold and the high separability between erroneous and correct reasoning representations.
  arXiv  Detail & Related papers  (2025-09-26T16:02:27Z)
• Unraveling the geometry of visual relational reasoning [11.82509693248749]
  Humans readily generalize abstract relations, such as recognizing "constant" in shape or color, whereas neural networks struggle, limiting their flexible reasoning.<n>We introduce SimplifiedRPM, a novel benchmark for systematically evaluating abstract relational reasoning.<n>We also conduct human experiments to quantify relational difficulty, enabling direct model-human comparisons.<n>Our results establish a geometric foundation for relational reasoning, paving the way for more human-like visual reasoning in AI.
  arXiv  Detail & Related papers  (2025-02-24T18:07:54Z)
• Skews in the Phenomenon Space Hinder Generalization in Text-to-Image Generation [59.138470433237615]
  We introduce statistical metrics that quantify both the linguistic and visual skew of a dataset for relational learning. We show that systematically controlled metrics are strongly predictive of generalization performance. This work informs an important direction towards quality-enhancing the data diversity or balance to scaling up the absolute size.
  arXiv  Detail & Related papers  (2024-03-25T03:18:39Z)
• APo-VAE: Text Generation in Hyperbolic Space [116.11974607497986]
  In this paper, we investigate text generation in a hyperbolic latent space to learn continuous hierarchical representations. An Adrial Poincare Variversaational Autoencoder (APo-VAE) is presented, where both the prior and variational posterior of latent variables are defined over a Poincare ball via wrapped normal distributions. Experiments in language modeling and dialog-response generation tasks demonstrate the winning effectiveness of the proposed APo-VAE model.
  arXiv  Detail & Related papers  (2020-04-30T19:05:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.