Geometry of Human Perceptual Domains Emerges Transiently in LLM Representations

• URL: http://arxiv.org/abs/2605.27970v1
• Date: Wed, 27 May 2026 05:04:29 GMT
• Title: Geometry of Human Perceptual Domains Emerges Transiently in LLM Representations
• Abstract summary: We study the emergence of intrinsic geometrical structure corresponding to perceptual modalities within the residual streams of open-weight transformer architectures.<n>We observe the emergence of layer-wise geometric structure across multiple perceptual domains, despite the absence of any direct perceptual supervision during training.<n>This provides new insight into how and where human-like perceptual geometry arises in large language models.
• Score: 1.503974529275767
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While large language models (LLMs) are trained purely on textual data, prior work has shown that their internal representations can exhibit rich geometric structure in embedding space. Building on this line of work, we investigate whether such structure is similar to human perceptual organisation across different domains (e.g., color, pitch, emotion, and taste). Specifically, we study the layer-wise emergence of intrinsic geometrical structure corresponding to perceptual modalities within the residual streams of multiple open-weight transformer architectures. Our results reveal three key findings. First, we observe the emergence of layer-wise geometric structure across multiple perceptual domains, despite the absence of any direct perceptual supervision during training. Second, these perceptual domains exhibit distinct emergence profiles, with both geometric structure and its alignment with human baselines following domain- and model-specific trajectories across depth. Third, this emergence follows a consistent representational trajectory: geometry is weak or diffuse in early layers, becomes progressively organised in intermediate layers, and is attenuated in later layers, suggesting that perceptual geometry arises transiently as part of the model's internal transformation pipeline. This provides new insight into how and where human-like perceptual geometry arises in LLMs, offering a principled pathway for mechanistic analysis of internal representations.

Related papers

• Transformer Geometry Observatory TGO-III: Semantic Geometry Observatory [2.4900509453584156]
  This work investigates semantic geometry and class separability as representations evolve across layers of ViT-Small/16 through TGO-III: Semantic Geometry Observatory.<n>It is a framework designed to analyze the emergence of semantic organization, feature evolution, and class-wise representation geometry throughout training.
  arXiv  Detail & Related papers  (2026-08-03T08:21:16Z)
• SeeSE3: Emergence of 3D Space in Vision Features [54.140615953727355]
  We investigate the relation between the structure of the space of visual features and the group of Euclidean transformations $SE(3)$.<n>We show that self-supervised vision models, which, in principle, have not been trained with direct 3D supervision or active agency, possess latent subspaces that are remarkably strongly correlated with 3D Euclidean space, when probed correctly.<n>We propose a new class of "Latent-Space Navigation" techniques that perform visual odometry and localization purely in the latent space, bypassing the need for explicit 3D reconstruction.
  arXiv  Detail & Related papers  (2026-07-15T18:00:31Z)
• A Navigable Manifold of Hypothesized Consciousness-Spectrum States in Language Model Representations [0.0]
  We study the geometric structure and dynamics of patterns along this spectrum in transformer embedding spaces.<n>We show that embeddings exhibit a globally organized geometry aligned with this spectrum.<n>Results suggest that embedding spaces encode structured and navigable geometry aligned with a hypothesized consciousness-spectrum taxonomy.
  arXiv  Detail & Related papers  (2026-06-04T04:33:17Z)
• Are Common Substructures Transferable? Riemannian Graph Foundation Model with Neural Vector Bundles [53.65187206795911]
  We develop a graph intrinsic geometry learning framework called Neural Vector Bundle.<n>Building on this, we design GAUGE, a pretrainable neural architecture that constructs the vector bundle and flattens geometrically compatible local coordinates.<n>We empirically validate its superior expressiveness in challenging tasks including zero-shot link prediction and graph isomorphism.
  arXiv  Detail & Related papers  (2026-06-02T07:35:42Z)
• From Extrinsic to Intrinsic: Geodesic-Guided Representation Learning for 3D Geometric Data [56.41479248637621]
  We introduce a novel 3D representation learning paradigm, namely textbfPRISM, for textbfPre-training.<n>PRISM learns embeddings by textbfRecovering the textbfIntrinsic textbfSurface geodesic textbfMetric.<n> Experiments demonstrate that our approach shows satisfactory accuracy, robustness, and high efficiency in geodesic distance prediction.
  arXiv  Detail & Related papers  (2026-06-01T13:54:26Z)
• Uncovering the Latent Potential of Deep Intermediate Representations [16.988545433223933]
  We show that task-relevant information is distributed non-monotonically across layers and cannot be recovered by nave aggregation.<n>We introduce Layer-wise Optimal Embedding Selection (LOES), a constructive spectral method that identifies task-discriminative subspaces.<n>Our method reveals how semantic factors are distributed across layers, thereby enabling cross-lingual and cross-modal interpretability analyses.
  arXiv  Detail & Related papers  (2026-05-21T20:58:42Z)
• Geometry-aware Prototype Learning for Cross-domain Few-shot Medical Image Segmentation [11.12183614587322]
  Cross-domain few-shot medical image segmentation (CD-FSMIS) requires a model to generalise simultaneously to novel anatomical categories and unseen imaging domains.<n>Existing prototypical approaches inevitably entangle anatomical structure with domain-specific appearance variations.<n>We propose GeoProto, a geometry-aware CD-FSMIS framework that enriches matching with explicit structural priors.
  arXiv  Detail & Related papers  (2026-05-11T17:32:26Z)
• Emergent Structured Representations Support Flexible In-Context Inference in Large Language Models [77.98801218316505]
  Large language models (LLMs) exhibit emergent behaviors suggestive of human-like reasoning.<n>We investigate the internal processing of LLMs during in-context concept inference.
  arXiv  Detail & Related papers  (2026-02-08T03:14:39Z)
• GeoGNN: Quantifying and Mitigating Semantic Drift in Text-Attributed Graphs [59.61242815508687]
  Graph neural networks (GNNs) on text--attributed graphs (TAGs) encode node texts using pretrained language models (PLMs) and propagate these embeddings through linear neighborhood aggregation.<n>This work introduces a local PCA-based metric that measures the degree of semantic drift and provides the first quantitative framework to analyze how different aggregation mechanisms affect manifold structure.
  arXiv  Detail & Related papers  (2025-11-12T06:48:43Z)
• The Neural Differential Manifold: An Architecture with Explicit Geometric Structure [8.201374511929538]
  This paper introduces the Neural Differential Manifold (NDM), a novel neural network architecture that explicitly incorporates geometric structure into its fundamental design.<n>We analyze the theoretical advantages of this approach, including its potential for more efficient optimization, enhanced continual learning, and applications in scientific discovery and controllable generative modeling.
  arXiv  Detail & Related papers  (2025-10-29T02:24:27Z)
• Geometric Origins of Bias in Deep Neural Networks: A Human Visual System Perspective [1.7315645623674356]
  Bias formation in deep neural networks (DNNs) remains a critical yet poorly understood challenge.<n>Inspired by the human visual system, we propose a geometric analysis framework linking the geometric complexity of class-specific perceptual Manifolds to model bias.<n>To support this analysis, we present the Perceptual-Manifold-Geometry library, designed for calculating the geometric properties of perceptual Manifolds.
  arXiv  Detail & Related papers  (2025-02-17T13:54:02Z)
• A Hitchhiker's Guide to Geometric GNNs for 3D Atomic Systems [87.30652640973317]
  Recent advances in computational modelling of atomic systems represent them as geometric graphs with atoms embedded as nodes in 3D Euclidean space. Geometric Graph Neural Networks have emerged as the preferred machine learning architecture powering applications ranging from protein structure prediction to molecular simulations and material generation. This paper provides a comprehensive and self-contained overview of the field of Geometric GNNs for 3D atomic systems.
  arXiv  Detail & Related papers  (2023-12-12T18:44:19Z)
• The geometry of hidden representations of large transformer models [43.16765170255552]
  Large transformers are powerful architectures used for self-supervised data analysis across various data types. We show that the semantic structure of the dataset emerges from a sequence of transformations between one representation and the next. We show that the semantic information of the dataset is better expressed at the end of the first peak, and this phenomenon can be observed across many models trained on diverse datasets.
  arXiv  Detail & Related papers  (2023-02-01T07:50:26Z)
• UCLID-Net: Single View Reconstruction in Object Space [60.046383053211215]
  We show that building a geometry preserving 3-dimensional latent space helps the network concurrently learn global shape regularities and local reasoning in the object coordinate space. We demonstrate both on ShapeNet synthetic images, which are often used for benchmarking purposes, and on real-world images that our approach outperforms state-of-the-art ones.
  arXiv  Detail & Related papers  (2020-06-06T09:15:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.