Spectral Superposition: A Theory of Feature Geometry

• URL: http://arxiv.org/abs/2602.02224v1
• Date: Mon, 02 Feb 2026 15:28:38 GMT
• Title: Spectral Superposition: A Theory of Feature Geometry
• Authors: Georgi Ivanov, Narmeen Oozeer, Shivam Raval, Tasana Pejovic, Shriyash Upadhyay, Amir Abdullah,
• Abstract summary: We develop a theory for studying the geometric structre of features by analyzing the spectra of weight derived matrices.<n>In toy models of superposition, we use this theory to prove that capacity saturation forces spectral localization.
• Score: 1.3837984867394175
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Neural networks represent more features than they have dimensions via superposition, forcing features to share representational space. Current methods decompose activations into sparse linear features but discard geometric structure. We develop a theory for studying the geometric structre of features by analyzing the spectra (eigenvalues, eigenspaces, etc.) of weight derived matrices. In particular, we introduce the frame operator $F = WW^\top$, which gives us a spectral measure that describes how each feature allocates norm across eigenspaces. While previous tools could describe the pairwise interactions between features, spectral methods capture the global geometry (``how do all features interact?''). In toy models of superposition, we use this theory to prove that capacity saturation forces spectral localization: features collapse onto single eigenspaces, organize into tight frames, and admit discrete classification via association schemes, classifying all geometries from prior work (simplices, polygons, antiprisms). The spectral measure formalism applies to arbitrary weight matrices, enabling diagnosis of feature localization beyond toy settings. These results point toward a broader program: applying operator theory to interpretability.

Related papers

• Stationarity and Spectral Characterization of Random Signals on Simplicial Complexes [45.01439616647312]
  We propose a probabilistic framework for random signals defined on simplicial complexes.<n>Specifically, we generalize the classical notion of stationarity.<n>We empirically demonstrate the practicality of these benefits through multiple synthetic and real-world simulations.
  arXiv  Detail & Related papers  (2026-02-03T03:31:10Z)
• Critical quantum states and hierarchical spectral statistics in a Cantor potential [0.0]
  We study the spectral statistics and wave-function properties of a one-dimensional quantum system subject to a Cantor-type fractal potential.<n>We demonstrate how the self-similar geometry of the potential is imprinted on the quantum spectrum.
  arXiv  Detail & Related papers  (2026-01-13T08:25:41Z)
• Grand Unification of All Discrete Wigner Functions on $d \times d$ Phase Space [0.5242869847419834]
  Wigner functions help visualise quantum states and dynamics while supporting quantitative analysis in quantum information.<n>We introduce a stencil-based framework that exhausts all possible $dtimes d$ discrete Wigner functions for a single $d$-dimensional quantum system.
  arXiv  Detail & Related papers  (2025-03-12T12:55:37Z)
• Geometry of Lightning Self-Attention: Identifiability and Dimension [2.9816332334719773]
  We study the identifiability of deep attention by providing a description of the generic fibers of the parametrization for an arbitrary number of layers.<n>For a single-layer model, we characterize the singular and boundary points.<n>Finally, we formulate a conjectural extension of our results to normalized self-attention networks, prove it for a single layer, and numerically verify it in the deep case.
  arXiv  Detail & Related papers  (2024-08-30T12:00:36Z)
• Understanding Matrix Function Normalizations in Covariance Pooling through the Lens of Riemannian Geometry [63.694184882697435]
  Global Covariance Pooling (GCP) has been demonstrated to improve the performance of Deep Neural Networks (DNNs) by exploiting second-order statistics of high-level representations.<n>This paper provides a comprehensive and unified understanding of the matrix logarithm and power from a Riemannian geometry perspective.
  arXiv  Detail & Related papers  (2024-07-15T07:11:44Z)
• Gaussian Entanglement Measure: Applications to Multipartite Entanglement of Graph States and Bosonic Field Theory [50.24983453990065]
  An entanglement measure based on the Fubini-Study metric has been recently introduced by Cocchiarella and co-workers. We present the Gaussian Entanglement Measure (GEM), a generalization of geometric entanglement measure for multimode Gaussian states. By providing a computable multipartite entanglement measure for systems with a large number of degrees of freedom, we show that our definition can be used to obtain insights into a free bosonic field theory.
  arXiv  Detail & Related papers  (2024-01-31T15:50:50Z)
• Shape And Structure Preserving Differential Privacy [70.08490462870144]
  We show how the gradient of the squared distance function offers better control over sensitivity than the Laplace mechanism. We also show how using the gradient of the squared distance function offers better control over sensitivity than the Laplace mechanism.
  arXiv  Detail & Related papers  (2022-09-21T18:14:38Z)
• Geometry Interaction Knowledge Graph Embeddings [153.69745042757066]
  We propose Geometry Interaction knowledge graph Embeddings (GIE), which learns spatial structures interactively between the Euclidean, hyperbolic and hyperspherical spaces. Our proposed GIE can capture a richer set of relational information, model key inference patterns, and enable expressive semantic matching across entities.
  arXiv  Detail & Related papers  (2022-06-24T08:33:43Z)
• Sign and Basis Invariant Networks for Spectral Graph Representation Learning [75.18802152811539]
  We introduce SignNet and BasisNet -- new neural architectures that are invariant to all requisite symmetries and hence process collections of eigenspaces in a principled manner. Our networks are theoretically strong for graph representation learning -- they can approximate any spectral graph convolution. Experiments show the strength of our networks for learning spectral graph filters and learning graph positional encodings.
  arXiv  Detail & Related papers  (2022-02-25T23:11:59Z)
• Statistical Mechanics of Neural Processing of Object Manifolds [3.4809730725241605]
  This thesis lays the groundwork for a computational theory of neuronal processing of objects. We identify that the capacity of a manifold is determined that effective radius, R_M, and effective dimension, D_M.
  arXiv  Detail & Related papers  (2021-06-01T20:49:14Z)
• The role of feature space in atomistic learning [62.997667081978825]
  Physically-inspired descriptors play a key role in the application of machine-learning techniques to atomistic simulations. We introduce a framework to compare different sets of descriptors, and different ways of transforming them by means of metrics and kernels. We compare representations built in terms of n-body correlations of the atom density, quantitatively assessing the information loss associated with the use of low-order features.
  arXiv  Detail & Related papers  (2020-09-06T14:12:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.