The information geometry of large language models is shared, learned, and controllable

• URL: http://arxiv.org/abs/2609.11063v1
• Date: Thu, 10 Sep 2026 04:09:49 GMT
• Title: The information geometry of large language models is shared, learned, and controllable
• Abstract summary: We show that behaviour determines geometry up to output-preserving symmetries.<n>Across transformer, state-space and recurrent models, output geometries agree more strongly than activation geometries.<n> Controlled language assignments show that geometry follows the language law across architectures.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models learn similar behaviours, yet it remains unclear what structure they share or how to change one behaviour without disturbing others. The Fisher-Rao geometry of next-token probabilities connects these questions: behaviour determines this geometry up to output-preserving symmetries, whereas activation geometry depends on coordinates. Across transformer, state-space and recurrent models, output geometries agree more strongly than activation geometries, and shared geometry supports semantic-category transfer. Agreement with human word choices increases with predictive accuracy, scale and training, and improves further after model-only calibration. Token probabilities and read-out geometry jointly predict the spectrum and its effective dimension. Controlled language assignments show that geometry follows the language law across architectures. Pretraining corpus statistics predict held-out fact acquisition without recalibration, while randomised experiments show that deeper evidence substantially delays acquisition across every tested architecture and evidence construction. Finally, the geometry prescribes minimum-disturbance local interventions, predicts their relative cost, and supports reusable control: updates learned on donor prompts transfer to unseen prompts while better preserving behaviour on reference prompts than Euclidean control. The same geometric correction improves steering, editing, attribution, dictionary learning and fine-tuning.

Related papers

• Rewriting or Reweighting? A Geometric Account in Language Models [44.66056836026845]
  Post-training can substantially alter language-model behavior.<n>We study two mechanistically distinct failures, repetition as a decoding-attractor pathology and sycophancy as a preference-related alignment failure.<n>We introduce behavioral manifold analysis, which isolates behavior-specific geometry by selecting sparse behavior-associated coordinates.
  arXiv  Detail & Related papers  (2026-08-03T07:46:28Z)
• MENTIS: What Belief Changes Under Alignment? Measuring Multi-Scale Latent Torsion in Language Models [16.071341117554038]
  We introduce MENTIS, a framework for measuring alignment-induced internal reorganization.<n>Across four 7-8B model pairs on LITMUS, our study reveals that alignment-induced change is selective rather than uniform.<n>These results suggest preference alignment leaves structured, depth-localized geometric signatures in internal computation beyond what behavior-level evaluation alone can reveal.
  arXiv  Detail & Related papers  (2026-05-31T07:05:51Z)
• The Shape of Addition: Geometric Structures of Arithmetic in Large Language Models [11.106770083374288]
  Large Language Models exhibit paradoxical hallucination in fundamental arithmetic, implying a disconnect between internal computation and discrete output.<n>We propose the Noisy Quantization Model to explain this geometry, framing arithmetic errors as Geometric Slippages caused by internal neural noise pushing a continuous, latent Carry Potential across quantization thresholds.<n>We validate these insights through a geometric consistency check method that effectively detects and corrects these quantization failures during inference.
  arXiv  Detail & Related papers  (2026-05-29T10:55:50Z)
• Draw2Think: Harnessing Geometry Reasoning through Constraint Engine Interaction [59.47026381165585]
  Vision-language models solve geometry problems with rising accuracy, yet their intermediate states remain latent and unverifiable.<n>We present Draw2Think, a framework that recasts geometric reasoning from latent spatial inference into agentic interaction with the GeoGebra constraint engine.
  arXiv  Detail & Related papers  (2026-05-20T05:46:14Z)
• Failure Modes for Deep Learning-Based Online Mapping: How to Measure and Address Them [10.503307305047487]
  We propose a framework to identify and measure the underlying failure modes by disentangling two effects: Memorization of input features and overfitting to known map geometries.<n>We introduce Fréchet distance-based reconstruction statistics that capture per-element shape fidelity without threshold tuning.<n>We formulate an MST-based sparsification strategy that reduces redundancy and improves balancing and performance while shrinking training size.
  arXiv  Detail & Related papers  (2026-03-20T11:09:11Z)
• HyperAlign: Hyperbolic Entailment Cones for Adaptive Text-to-Image Alignment Assessment [84.65251073657883]
  We propose HyperAlign, an adaptive text-to-image alignment assessment framework based on hyperbolic entailment geometry.<n>First, we extract Euclidean features using CLIP and map them to hyperbolic space.<n>Second, we design a dynamic-supervision entailment modeling mechanism that transforms discrete entailment logic into continuous geometric structure supervision.<n>Third, we propose an adaptive modulation regressor that utilizes hyperbolic geometric features to generate sample-level modulation parameters.
  arXiv  Detail & Related papers  (2026-01-08T05:41:06Z)
• GraphShaper: Geometry-aware Alignment for Improving Transfer Learning in Text-Attributed Graphs [16.624063216788638]
  We introduce textbfGraphShaper, a geometry-aware framework that enhances graph encoding through multi-geometric specialization.<n>Our approach employs expert networks tailored to different geometric spaces, dynamically computing fusion weights to adaptively integrate geometric properties.<n>It achieves 9.47% accuracy improvements on citation networks and 7.63% on social networks in zero-shot settings.
  arXiv  Detail & Related papers  (2025-10-14T02:48:50Z)
• CP$^2$: Leveraging Geometry for Conformal Prediction via Canonicalization [51.716834831684004]
  We study the problem of conformal prediction (CP) under geometric data shifts.<n>We propose integrating geometric information--such as geometric pose--into the conformal procedure to reinstate its guarantees.
  arXiv  Detail & Related papers  (2025-06-19T10:12:02Z)
• PAID: Pairwise Angular-Invariant Decomposition for Continual Test-Time Adaptation [70.98107766265636]
  This paper takes the geometric attributes of pre-trained weights as a starting point, systematically analyzing three key components: magnitude, absolute angle, and pairwise angular structure.<n>We find that the pairwise angular structure remains stable across diverse corrupted domains and encodes domain-invariant semantic information, suggesting it should be preserved during adaptation.
  arXiv  Detail & Related papers  (2025-06-03T05:18:15Z)
• Bridging Geometric States via Geometric Diffusion Bridge [79.60212414973002]
  We introduce the Geometric Diffusion Bridge (GDB), a novel generative modeling framework that accurately bridges initial and target geometric states. GDB employs an equivariant diffusion bridge derived by a modified version of Doob's $h$-transform for connecting geometric states. We show that GDB surpasses existing state-of-the-art approaches, opening up a new pathway for accurately bridging geometric states.
  arXiv  Detail & Related papers  (2024-10-31T17:59:53Z)
• Exploring Data Geometry for Continual Learning [64.4358878435983]
  We study continual learning from a novel perspective by exploring data geometry for the non-stationary stream of data. Our method dynamically expands the geometry of the underlying space to match growing geometric structures induced by new data. Experiments show that our method achieves better performance than baseline methods designed in Euclidean space.
  arXiv  Detail & Related papers  (2023-04-08T06:35:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.