Transformer Geometry Observatory TGO-I: Spectral Geometry Observatory

• URL: http://arxiv.org/abs/2606.19249v1
• Date: Wed, 17 Jun 2026 16:26:59 GMT
• Title: Transformer Geometry Observatory TGO-I: Spectral Geometry Observatory
• Abstract summary: We introduce Transformer Geometry Observatory (TGO), a framework to investigate the representational geometry and dynamics of Vision Transformers.<n>TGO-I, the first installment of the framework, focuses on the spectral geometry of ViT representations.<n>Our results reveal a consistent increase in dimensional utilization, accompanied by decreasing anisotropy, increasing spectral entropy, increasing participation ratio, and progressively flatter eigenspectra.
• Score: 2.4900509453584156
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Despite the widespread adoption of Vision Transformers (ViTs) and their success across numerous computer vision applications, the fundamental understanding of their dimensional and representational geometry remains relatively underexplored. To address this gap, we introduce Transformer Geometry Observatory (TGO), a systematic framework of experiments and analysis pipelines designed to investigate the representational geometry and dynamics of Vision Transformers. TGO-I, the first installment of the framework, focuses on the spectral geometry of ViT representations. Using a ViT-Small/16 model trained on ImageNet-100, we analyze Effective Rank, Stable Rank, Participation Ratio, Spectral Entropy, Spectral Flatness, Spectral Anisotropy, covariance structure, eigenspectra, and singular value spectra throughout training. Our results reveal a consistent increase in dimensional utilization, accompanied by decreasing anisotropy, increasing spectral entropy, increasing participation ratio, and progressively flatter eigenspectra. Contrary to the common intuition that training should concentrate information into a small number of dominant directions, we observe a progressive redistribution of variance across representational dimensions. This phenomenon is particularly pronounced in the final CLS token representation, which exhibits the highest effective dimensionality and lowest anisotropy within the network.

Related papers

• Diversity-aware View Partitioning for Scalable VGGT [48.654660686945526]
  VGGT achieve strong performance by jointly reasoning over multiple views with global attention.<n>We propose a training-free and plug-and-play inference framework that organizes views into diversity-aware balanced chunks.<n>Our framework also complements existing VGGT variants, enabling scalable multi-view reconstruction without sacrificing geometric fidelity.
  arXiv  Detail & Related papers  (2026-07-02T08:37:56Z)
• Beyond Compression: Quantifying Spectral Accessibility in Vision Representations [4.681075707831016]
  Vision-language models map visual features into a shared embedding space through learned projection layers.<n>This study examines changes in representation through spatial-frequency accessibility, measured by the linear recoverability of band-limited energy from model representations.
  arXiv  Detail & Related papers  (2026-06-02T15:42:54Z)
• LESSViT: Robust Hyperspectral Representation Learning under Spectral Configuration Shift [29.791943499456426]
  Low-rank Efficient Spatial-Spectral ViT (LESSViT) is a sensor-flexible architecture for cross-spectral generalization.<n>We introduce a hyperspectral masked autoencoder (HyperMAE) with decoupled spatial-spectral masking and hierarchical channel sampling.
  arXiv  Detail & Related papers  (2026-05-18T15:22:26Z)
• Applying Vision Transformers on Spectral Analysis of Astronomical Objects [1.4061979259370274]
  We fine-tune a ViT pretrained on ImageNet using millions of spectra from the SDSS and LAMOST surveys, represented as spectral plots.<n>We achieve classification accuracy higher than Support Vector Machines and Random Forests, and attain $R2$ values comparable to AstroCLIP's spectrum encoder.
  arXiv  Detail & Related papers  (2025-05-30T22:53:45Z)
• CARL: Camera-Agnostic Representation Learning for Spectral Image Analysis [69.02751635551724]
  Spectral imaging offers promising applications across diverse domains, including medicine and urban scene understanding.<n> variability in channel dimensionality and captured wavelengths among spectral cameras impede the development of AI-driven methodologies.<n>We introduce CARL, a model for Camera-Agnostic Representation Learning across RGB, multispectral, and hyperspectral imaging modalities.
  arXiv  Detail & Related papers  (2025-04-27T13:06:40Z)
• Approaching Deep Learning through the Spectral Dynamics of Weights [41.948042468042374]
  spectral dynamics of weights -- the behavior of singular values and vectors during optimization -- to clarify and unify several phenomena in deep learning. We identify a consistent bias in optimization across various experiments, from small-scale grokking'' to large-scale tasks like image classification with ConvNets, image generation with UNets, speech recognition with LSTMs, and language modeling with Transformers.
  arXiv  Detail & Related papers  (2024-08-21T17:48:01Z)
• HoloNets: Spectral Convolutions do extend to Directed Graphs [59.851175771106625]
  Conventional wisdom dictates that spectral convolutional networks may only be deployed on undirected graphs. Here we show this traditional reliance on the graph Fourier transform to be superfluous. We provide a frequency-response interpretation of newly developed filters, investigate the influence of the basis used to express filters and discuss the interplay with characteristic operators on which networks are based.
  arXiv  Detail & Related papers  (2023-10-03T17:42:09Z)
• Curve Your Attention: Mixed-Curvature Transformers for Graph Representation Learning [77.1421343649344]
  We propose a generalization of Transformers towards operating entirely on the product of constant curvature spaces. We also provide a kernelized approach to non-Euclidean attention, which enables our model to run in time and memory cost linear to the number of nodes and edges.
  arXiv  Detail & Related papers  (2023-09-08T02:44:37Z)
• 2-D SSM: A General Spatial Layer for Visual Transformers [79.4957965474334]
  A central objective in computer vision is to design models with appropriate 2-D inductive bias. We leverage an expressive variation of the multidimensional State Space Model. Our approach introduces efficient parameterization, accelerated computation, and a suitable normalization scheme.
  arXiv  Detail & Related papers  (2023-06-11T09:41:37Z)
• Investigating Expressiveness of Transformer in Spectral Domain for Graphs [6.092217185687028]
  We study and prove the link between the spatial and spectral domain in the realm of the transformer. We propose FeTA, a framework that aims to perform attention over the entire graph spectrum analogous to the attention in spatial space.
  arXiv  Detail & Related papers  (2022-01-23T18:03:22Z)
• Unsupervised Machine Learning for Exploratory Data Analysis of Exoplanet Transmission Spectra [68.8204255655161]
  We focus on unsupervised techniques for analyzing spectral data from transiting exoplanets. We show that there is a high degree of correlation in the spectral data, which calls for appropriate low-dimensional representations. We uncover interesting structures in the principal component basis, namely, well-defined branches corresponding to different chemical regimes.
  arXiv  Detail & Related papers  (2022-01-07T22:26:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.