Transformer Geometry Observatory TGO-IV: Developmental Topology Observatory

• URL: http://arxiv.org/abs/2608.09997v1
• Date: Fri, 07 Aug 2026 13:11:22 GMT
• Title: Transformer Geometry Observatory TGO-IV: Developmental Topology Observatory
• Abstract summary: Transformer Geometry Observatory-TGO-IV introduces a topological framework for analysing the evolution of Transformer representations.<n>Rather than studying local geometric properties alone, TGO-IV constructs Vietoris--Rips simplicial complexes from token-level representation point clouds.<n>The proposed framework comprises complementary topological observatories including Persistence Diagrams, Barcode Diagrams, Betti Curves, Persistence Landscapes, Bottleneck Distance, and Wasserstein Distance.
• Score: 2.4900509453584156
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Transformers have had a profound impact on the world of language processing and computer vision. As efforts to answer the million-dollar question of ``How does a Transformer learn?" have been increasing, existing interpretability studies primarily analyze representations at isolated layers or the network as a whole, while the developmental evolution of individual representations and its manifolds across transformer layers remains underexplored. With this work, we aim at providing a comprehensive analysis of the evolution of representations as the representation point cloud transforms across the layers; thereby attempting to isolate layers or establish a trend which comes closer to justifying how and when raw input representations evolve into task-relevant feature representations. Thus, Transformer Geometry Observatory-TGO-IV introduces a topological framework for analysing the evolution of Transformer representations through the lens of Persistent Homology. Rather than studying local geometric properties alone, TGO-IV constructs Vietoris--Rips simplicial complexes from token-level representation point clouds and investigates the evolution of their persistent topological signatures across Transformer layers. The proposed framework comprises complementary topological observatories including Persistence Diagrams, Barcode Diagrams, Betti Curves, Persistence Landscapes, Bottleneck Distance, and Wasserstein Distance, enabling a comprehensive analysis of how the global topology of representation point clouds develops throughout the forward pass.

Related papers

• Global Divergence, Local Convergence: Representation Geometry in SSMs and Transformers [4.71326501896672]
  Recent state-space models (SSMs) such as Mamba achieve language modeling performance comparable to transformers.<n>This raises an important question: how do these structural differences influence the geometry and functional nature of their internal representations?<n>We study this question through a multi-scale analysis of representations in transformers, SSMs, and hybrid architecture.
  arXiv  Detail & Related papers  (2026-09-08T12:57:30Z)
• Transformer Geometry Observatory TGO-III: Semantic Geometry Observatory [2.4900509453584156]
  This work investigates semantic geometry and class separability as representations evolve across layers of ViT-Small/16 through TGO-III: Semantic Geometry Observatory.<n>It is a framework designed to analyze the emergence of semantic organization, feature evolution, and class-wise representation geometry throughout training.
  arXiv  Detail & Related papers  (2026-08-03T08:21:16Z)
• ARDepth: Auto-regressive Monocular Depth Estimation with Progressive Visual Conditioning [66.42476564611607]
  Diffusion models assume that depth can be recovered as a globally smooth field through iterative denoising.<n>We introduce ARDepth, which formulates depth estimation as structured auto-regressive generation.<n>We show that our approach achieves strong performance and produces structurally consistent depth predictions across scales.
  arXiv  Detail & Related papers  (2026-07-14T07:06:46Z)
• Transformer Geometry Observatory TGO-II: Representational Similarity Observatory [2.4900509453584156]
  Vision Transformers have achieved remarkable success across computer vision and language applications.<n>We present Transformer Geometry Observatory-II (TGO-II), a representation geometry analysis framework designed to investigate how Transformer representations evolve during supervised training.
  arXiv  Detail & Related papers  (2026-07-02T16:22:53Z)
• Towards Fast GNN Surrogates for CO2 Migration in Complex Geological Formations [58.079925240809644]
  We propose an end-to-end graph neural surrogate tailored to CO$$ibility plume migration forecasting in geological storage.<n>The method is evaluated on the SPE11A benchmark, a well-known industry test case designed to assess CO$$ storage scenarios.<n>The proposed model produces competitive forecasts of gas saturation and liquid-phase density, with cumulative errors that remain moderate over extended forecasting horizons.
  arXiv  Detail & Related papers  (2026-06-15T18:19:17Z)
• Spatial Representation Learning Beyond Pixels: Unifying Raster Data and Vector Semantics for Human-Centric Geospatial Foundation Models [12.200359585933766]
  Recent advances in self-supervised learning have given rise to Earth Observation Foundation Models (EOFMs)<n>This paper calls for a paradigm shift toward joint Spatial Representation Learning (SRL) in an unified embedding space that integrate perception with vector-based reasoning.<n>We contend that such integration is essential for developing next-generation geospatial AI systems capable of more accurate, interpretable, and grounded understanding of the Earth.
  arXiv  Detail & Related papers  (2026-06-01T15:21:58Z)
• GeoGNN: Quantifying and Mitigating Semantic Drift in Text-Attributed Graphs [59.61242815508687]
  Graph neural networks (GNNs) on text--attributed graphs (TAGs) encode node texts using pretrained language models (PLMs) and propagate these embeddings through linear neighborhood aggregation.<n>This work introduces a local PCA-based metric that measures the degree of semantic drift and provides the first quantitative framework to analyze how different aggregation mechanisms affect manifold structure.
  arXiv  Detail & Related papers  (2025-11-12T06:48:43Z)
• What Improves the Generalization of Graph Transformers? A Theoretical Dive into the Self-attention and Positional Encoding [67.59552859593985]
  Graph Transformers, which incorporate self-attention and positional encoding, have emerged as a powerful architecture for various graph learning tasks. This paper introduces first theoretical investigation of a shallow Graph Transformer for semi-supervised classification.
  arXiv  Detail & Related papers  (2024-06-04T05:30:16Z)
• The geometry of hidden representations of large transformer models [43.16765170255552]
  Large transformers are powerful architectures used for self-supervised data analysis across various data types. We show that the semantic structure of the dataset emerges from a sequence of transformations between one representation and the next. We show that the semantic information of the dataset is better expressed at the end of the first peak, and this phenomenon can be observed across many models trained on diverse datasets.
  arXiv  Detail & Related papers  (2023-02-01T07:50:26Z)
• Vision Transformers: From Semantic Segmentation to Dense Prediction [139.15562023284187]
  We explore the global context learning potentials of vision transformers (ViTs) for dense visual prediction. Our motivation is that through learning global context at full receptive field layer by layer, ViTs may capture stronger long-range dependency information. We formulate a family of Hierarchical Local-Global (HLG) Transformers, characterized by local attention within windows and global-attention across windows in a pyramidal architecture.
  arXiv  Detail & Related papers  (2022-07-19T15:49:35Z)
• CSformer: Bridging Convolution and Transformer for Compressive Sensing [65.22377493627687]
  This paper proposes a hybrid framework that integrates the advantages of leveraging detailed spatial information from CNN and the global context provided by transformer for enhanced representation learning. The proposed approach is an end-to-end compressive image sensing method, composed of adaptive sampling and recovery. The experimental results demonstrate the effectiveness of the dedicated transformer-based architecture for compressive sensing.
  arXiv  Detail & Related papers  (2021-12-31T04:37:11Z)
• Self-Supervised Graph Representation Learning via Topology Transformations [61.870882736758624]
  We present the Topology Transformation Equivariant Representation learning, a general paradigm of self-supervised learning for node representations of graph data. In experiments, we apply the proposed model to the downstream node and graph classification tasks, and results show that the proposed method outperforms the state-of-the-art unsupervised approaches.
  arXiv  Detail & Related papers  (2021-05-25T06:11:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.