Transformer Geometry Observatory TGO-III: Semantic Geometry Observatory

• URL: http://arxiv.org/abs/2608.01876v1
• Date: Mon, 03 Aug 2026 08:21:16 GMT
• Title: Transformer Geometry Observatory TGO-III: Semantic Geometry Observatory
• Abstract summary: This work investigates semantic geometry and class separability as representations evolve across layers of ViT-Small/16 through TGO-III: Semantic Geometry Observatory.<n>It is a framework designed to analyze the emergence of semantic organization, feature evolution, and class-wise representation geometry throughout training.
• Score: 2.4900509453584156
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: With the widespread adoption of Vision Transformers in modern AI, the need to analyze their inherent representational behavior has become increasingly important. While most existing studies emphasize token geometries and training dynamics, the evolution of representational covariance structures and class-level geometric organization remains comparatively underexplored. In this work, we investigate semantic geometry and class separability as representations evolve across the layers of ViT-Small/16 through TGO-III: Semantic Geometry Observatory. It is a framework designed to analyze the emergence of semantic organization, feature evolution, and class-wise representation geometry throughout training. The framework employs multiple complementary observatories, including Linear Probe Accuracy, Fisher Ratio, Class Centroid Distances, Local Intrinsic Dimension, and Local PCA Rank, to quantify the progressive evolution of discriminative representations. Our analysis reveals that class representations become progressively more linearly separable, Fisher discriminability increases, class centroids move farther apart, and local representation manifolds exhibit structured class-dependent geometric complexity. These observations provide empirical evidence supporting the Semantic Expansion Hypothesis, suggesting that the manifold expansion observed in previous observatories is accompanied by the progressive organization of representations into increasingly discriminative semantic structures. Collectively, TGO-III extends the Transformer Geometry Observatory framework by establishing a direct connection between manifold geometry, covariance evolution, and semantic organization during Transformer training.

Related papers

• Transformer Geometry Observatory TGO-IV: Developmental Topology Observatory [2.4900509453584156]
  Transformer Geometry Observatory-TGO-IV introduces a topological framework for analysing the evolution of Transformer representations.<n>Rather than studying local geometric properties alone, TGO-IV constructs Vietoris--Rips simplicial complexes from token-level representation point clouds.<n>The proposed framework comprises complementary topological observatories including Persistence Diagrams, Barcode Diagrams, Betti Curves, Persistence Landscapes, Bottleneck Distance, and Wasserstein Distance.
  arXiv  Detail & Related papers  (2026-08-07T13:11:22Z)
• Continual Learning in Transition [77.0733079568852]
  Classical continual learning has primarily focused on enabling models to update and retain knowledge through parameter-centric mechanisms.<n>Emerging paradigms are reshaping the scope of CL beyond this traditional model adaptation view.<n>We examine the evolution of continual learning through three dimensions: When, How, and Where learning occurs.
  arXiv  Detail & Related papers  (2026-08-06T16:07:26Z)
• Transformer Geometry Observatory TGO-II: Representational Similarity Observatory [2.4900509453584156]
  Vision Transformers have achieved remarkable success across computer vision and language applications.<n>We present Transformer Geometry Observatory-II (TGO-II), a representation geometry analysis framework designed to investigate how Transformer representations evolve during supervised training.
  arXiv  Detail & Related papers  (2026-07-02T16:22:53Z)
• A Navigable Manifold of Hypothesized Consciousness-Spectrum States in Language Model Representations [0.0]
  We study the geometric structure and dynamics of patterns along this spectrum in transformer embedding spaces.<n>We show that embeddings exhibit a globally organized geometry aligned with this spectrum.<n>Results suggest that embedding spaces encode structured and navigable geometry aligned with a hypothesized consciousness-spectrum taxonomy.
  arXiv  Detail & Related papers  (2026-06-04T04:33:17Z)
• Geometry of Human Perceptual Domains Emerges Transiently in LLM Representations [1.503974529275767]
  We study the emergence of intrinsic geometrical structure corresponding to perceptual modalities within the residual streams of open-weight transformer architectures.<n>We observe the emergence of layer-wise geometric structure across multiple perceptual domains, despite the absence of any direct perceptual supervision during training.<n>This provides new insight into how and where human-like perceptual geometry arises in large language models.
  arXiv  Detail & Related papers  (2026-05-27T05:04:29Z)
• LAST: Bridging Vision-Language and Action Manifolds via Gromov-Wasserstein Alignment [71.3712075841632]
  LAST reconstructs the action space to establish local metric compatibility with the Vision-Language-Action modality.<n> LAST enables VLA models with superior convergence and generalizability.
  arXiv  Detail & Related papers  (2026-05-27T03:48:10Z)
• Beyond Point-Wise Matching: Structural Representation Alignment for Accelerating Diffusion Transformers [93.3976834364707]
  We propose sREPA, a structural REPresentation Alignment framework to enforce consistency in the relational geometry of feature maps.<n>By encouraging the model to internalize holistic spatial layouts and structural correlations from pre-trained features, sREPA achieves faster and more stable convergence.
  arXiv  Detail & Related papers  (2026-05-16T12:01:04Z)
• SEMASIA: A Large-Scale Dataset of Semantically Structured Latent Representations [12.185380843937196]
  We introduce SEMASIA, a large-scale collection of latent representations extracted from approximately 1,700 pretrained vision models.<n>We analyze the conceptual organization of individual latent spaces, showing consistent prototype-like clustering.<n>We perform a large-scale regression analysis of how pretraining-data complexity, specialization, transfer learning, augmentation, and model scale relate to geometric and probing properties of embeddings.
  arXiv  Detail & Related papers  (2026-05-10T11:42:36Z)
• Hyperbolic Enhanced Representation Learning for Incomplete Multi-view Clustering [57.38215918201251]
  We propose HERL, a Hyperbolic Enhanced Representation Learning framework for Incomplete Multi-View Clustering.<n> operating within the Poincaré ball, HERL constructs a structure-aware latent space to enhance representation learning.<n>We show that HERL consistently outperforms state-of-the-art approaches.
  arXiv  Detail & Related papers  (2026-04-18T10:50:46Z)
• TopoOR: A Unified Topological Scene Representation for the Operating Room [54.50897207203292]
  Surgical Scene Graphs abstract the complexity of surgical operating rooms (OR) into a structure of entities and their relations.<n>We introduce TopoOR, a new paradigm that models multimodal operating rooms as a higher-order structure.<n>We also propose a higher-order attention mechanism that explicitly preserves manifold structure and modality-specific features.
  arXiv  Detail & Related papers  (2026-03-10T10:19:42Z)
• GeoGNN: Quantifying and Mitigating Semantic Drift in Text-Attributed Graphs [59.61242815508687]
  Graph neural networks (GNNs) on text--attributed graphs (TAGs) encode node texts using pretrained language models (PLMs) and propagate these embeddings through linear neighborhood aggregation.<n>This work introduces a local PCA-based metric that measures the degree of semantic drift and provides the first quantitative framework to analyze how different aggregation mechanisms affect manifold structure.
  arXiv  Detail & Related papers  (2025-11-12T06:48:43Z)
• Current Symmetry Group Equivariant Convolution Frameworks for Representation Learning [5.802794302956837]
  Euclidean deep learning is often inadequate for addressing real-world signals where the representation space is irregular and curved with complex topologies. We focus on the importance of symmetry group equivariant deep learning models and their realization of convolution-like operations on graphs, 3D shapes, and non-Euclidean spaces.
  arXiv  Detail & Related papers  (2024-09-11T15:07:18Z)
• The geometry of hidden representations of large transformer models [43.16765170255552]
  Large transformers are powerful architectures used for self-supervised data analysis across various data types. We show that the semantic structure of the dataset emerges from a sequence of transformations between one representation and the next. We show that the semantic information of the dataset is better expressed at the end of the first peak, and this phenomenon can be observed across many models trained on diverse datasets.
  arXiv  Detail & Related papers  (2023-02-01T07:50:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.