Geometric Signatures of Reasoning: A Spectral Perspective on Task Hardness

• URL: http://arxiv.org/abs/2607.01571v1
• Date: Thu, 02 Jul 2026 01:03:49 GMT
• Title: Geometric Signatures of Reasoning: A Spectral Perspective on Task Hardness
• Abstract summary: Chain-of-thought (CoT) reasoning enables large language models to solve complex problems by generating intermediate reasoning steps.<n>We study the emphgeometry of CoT trajectories in the hidden state space of transformer models.<n>We show theoretically that trajectories with flatter eigenvalue spectra correspond to harder tasks, as they explore more of the hidden dimensions.<n> Lastly, we explore how kinematic features of the trajectory, mean position, positional dispersion, initial and current hidden states, mean velocity, mean speed, and speed dispersion, can be used to predict solution correctness before generation is complete.
• Score: 32.929617755703696
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Chain-of-thought (CoT) reasoning enables large language models (LLMs) to solve complex problems by generating intermediate reasoning steps. While much attention has been paid to the length and content of these reasoning chains, far less is known about their internal geometry. We study the \emph{geometry} of CoT trajectories in the hidden state space of transformer models, formalizing each reasoning chain as a discrete curve in $\mathbb{R}^d$ and characterizing it through spectral, positional, and kinematic geometric functionals. We introduce the effective dimension $d_ρ$ as a measure of trajectory complexity and show theoretically that trajectories with flatter eigenvalue spectra correspond to harder tasks, as they explore more of the hidden dimensions. Lastly, we explore how kinematic features of the trajectory, mean position, positional dispersion, initial and current hidden states, mean velocity, mean speed, and speed dispersion, can be used to predict solution correctness before generation is complete, and may inform future early-stopping strategies. Experimentally, on mathematical reasoning problems from the MATH500 dataset, $d_ρ$ achieves $0.93$ AUC in distinguishing easy from hard problems, while kinematic features potentially can predict correctness from only the first $20\%$ of generated tokens. These correctness signatures transfer across questions of varying difficulty, establishing that the shape of a model's internal reasoning trajectory is a principled window into both task hardness and solution quality.

Related papers

• Intrinsic Structure: Spectral Identifiability for Mechanistic Interpretability [51.56484100374058]
  We prove the first identifiability theorem for a mechanistic-interpretability primitive.<n>The spectrum is an identifiable, model-intrinsic fingerprint with a stated error bar, not a legible decomposition.
  arXiv  Detail & Related papers  (2026-08-10T19:42:01Z)
• PIEFS: Physics-Informed Eigenfunction Features with Learnable Scaling [42.058442499492536]
  We propose Physics-Informed Eigenfunction Features with Learnable Scaling (PIEFS)<n>PIEFS is a supervised neural representation-learning framework with a spectral inductive bias, based on a modified Dirichlet energy.
  arXiv  Detail & Related papers  (2026-07-04T03:55:40Z)
• Fitting Unknown Number of Hyperplanes with Manifold Optimization [57.48093263119306]
  Fitting unknown number of linearplanes to data is a fundamental yet challenging problem in machine learning.<n>Existing approaches often struggle with optimum optima or lack geometric consistency.
  arXiv  Detail & Related papers  (2026-05-27T14:02:20Z)
• Reasoning Models Don't Just Think Longer, They Move Differently [25.136448013973904]
  We study the distinction through hidden-state trajectories during chain-of-thought generation across competitive programming, mathematics, and Boolean satisfiability.<n>The clearest reasoning-specific separation appears in the code domain, where harder problems show more direct corrected trajectories.<n>These findings establish length correction as a prerequisite for generation-time trajectory analysis and show that reasoning training can be associated with distinct corrected trajectory geometry.
  arXiv  Detail & Related papers  (2026-05-14T22:37:33Z)
• Random Dot Product Graphs as Dynamical Systems: Limitations and Opportunities [0.0]
  We investigate the differential equations governing the evolution of a temporal network within Random Dot Product Graphs.<n>We identify three fundamental obstructions: gauge freedom from rotational ambiguity in latent positions, realizability constraints from the manifold structure of the probability matrix, and trajectory recovery artifacts from spectral embedding.<n>We prove an identifiability principle: symmetric dynamics cannot absorb skew-symmetric gauge contamination, so dynamics structure can resolve obstructions.
  arXiv  Detail & Related papers  (2026-03-05T21:56:07Z)
• The Spacetime of Diffusion Models: An Information Geometry Perspective [40.23096112113255]
  We show that the standard pullback approach, utilizing the deterministic probability flow ComplementODE decoder, is fundamentally flawed.<n>We introduce a latent spacetime $z=(x_t,t)$ that indexes the family of denoising distributions $p(x_t,t)$ across all noise scales.<n>The resulting structure induces a principled Diffusion Distance Edit, where geodesics trace minimal sequences of noise and denoise edits between data.
  arXiv  Detail & Related papers  (2025-05-23T06:16:58Z)
• Reasoning by Superposition: A Theoretical Perspective on Chain of Continuous Thought [64.43689151961054]
  We prove that a two-layer transformer with $D$ steps of continuous CoTs can solve the directed graph reachability problem.<n>In our construction, each continuous thought vector is a superposition state that encodes multiple search frontiers simultaneously.
  arXiv  Detail & Related papers  (2025-05-18T18:36:53Z)
• Learning the Universe: Learning to Optimize Cosmic Initial Conditions with Non-Differentiable Structure Formation Models [0.22940141855172033]
  We introduce Learning the Universe by Learning to Optimize (LULO) for reconstructing the 3D cosmic initial conditions.<n>LULO is a gradient-free framework for reconstructing the 3D cosmic initial conditions.<n>We demonstrate the method by accurately reconstructing initial conditions from $M_200mathrmc$ halos identified in a dark matter-only $N$-body simulation.
  arXiv  Detail & Related papers  (2025-02-18T19:17:01Z)
• Disentangled Representation Learning with the Gromov-Monge Gap [65.73194652234848]
  Learning disentangled representations from unlabelled data is a fundamental challenge in machine learning.<n>We introduce a novel approach to disentangled representation learning based on quadratic optimal transport.<n>We demonstrate the effectiveness of our approach for quantifying disentanglement across four standard benchmarks.
  arXiv  Detail & Related papers  (2024-07-10T16:51:32Z)
• CWF: Consolidating Weak Features in High-quality Mesh Simplification [50.634070540791555]
  We propose a smooth functional that simultaneously considers all of these requirements. The functional comprises a normal anisotropy term and a Centroidal Voronoi Tessellation (CVT) energy term.
  arXiv  Detail & Related papers  (2024-04-24T05:37:17Z)
• On interpretability and proper latent decomposition of autoencoders [6.85316573653194]
  This paper theoretically interpret the transformations of an autoencoder. We mathematically derive the metric tensor, which provides a mathematical description of the manifold. We propose a method -- proper latent decomposition (PLD) -- that generalizes proper decomposition of turbulent flows on the autoencoder latent space.
  arXiv  Detail & Related papers  (2022-11-15T17:55:35Z)
• Unveiling the Sampling Density in Non-Uniform Geometric Graphs [69.93864101024639]
  We consider graphs as geometric graphs: nodes are randomly sampled from an underlying metric space, and any pair of nodes is connected if their distance is less than a specified neighborhood radius. In a social network communities can be modeled as densely sampled areas, and hubs as nodes with larger neighborhood radius. We develop methods to estimate the unknown sampling density in a self-supervised fashion.
  arXiv  Detail & Related papers  (2022-10-15T08:01:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.