Thinking in Different Spaces: Domain-Specific Latent Geometry Survives Cross-Architecture Translation

• URL: http://arxiv.org/abs/2603.20406v1
• Date: Fri, 20 Mar 2026 18:26:23 GMT
• Title: Thinking in Different Spaces: Domain-Specific Latent Geometry Survives Cross-Architecture Translation
• Abstract summary: We investigate whether independently trained language models converge to geometrically compatible latent representations.<n>We learn a linear projection matrix that maps activation vectors from a large teacher model into the coordinate system of a smaller student model.<n>We report a near-zero correlation between geometric alignment quality and behavioral correction rate.
• Score: 0.7162422068114824
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We investigate whether independently trained language models converge to geometrically compatible latent representations, and whether this compatibility can be exploited to correct model behavior at inference time without any weight updates. We learn a linear projection matrix that maps activation vectors from a large teacher model into the coordinate system of a smaller student model, then intervene on the student's residual stream during generation by substituting its internal state with the translated teacher representation. Across a fully crossed experimental matrix of 20 heterogeneous teacher-student pairings spanning mixture-of-experts, dense, code-specialized, and synthetically trained architectures, the Ridge projection consistently achieves R^2 = 0.50 on verbal reasoning and R^2 = 0.40 on mathematical reasoning, collapsing to R^2 = -0.22 under permutation control and R^2 = 0.01 under L_1 regularization. Behavioral correction rates range from 14.0% to 50.0% on TruthfulQA (mean 25.2%) and from 8.5% to 43.3% on GSM8K arithmetic reasoning (mean 25.5%), demonstrating that the method generalizes across fundamentally different reasoning domains. We report a near-zero correlation between geometric alignment quality and behavioral correction rate (r = -0.07), revealing a dissociation between representation space fidelity and output space impact. Intervention strength is architecture-specific: student models exhibit characteristic sensitivity profiles that invert across domains, with the most steerable verbal student becoming the least steerable mathematical student. Finally, a double dissociation experiment conducted across all 20 model pairings confirms without exception that projection matrices collapse catastrophically when transferred across reasoning domains (mean R^2 = -3.83 in both transfer directions), establishing domain-specific subspace geometry as a universal property of LMs.

Related papers

• Architecture-Dependent Causal Transfer of Activation States Across Large Language Models [51.56484100374058]
  Internal language activation states can be transferred causally between different large language models.<n>Injecting projected activations into a target model during generation produces a statistically significant, preregistered causal effect.<n>We conclude that end-to-end activation-state transfer between LLMs, as currently implemented, is architecture-dependent rather than universal.
  arXiv  Detail & Related papers  (2026-08-17T09:53:27Z)
• Emergent Semantic Representations in World Models through Physical Interaction without Linguistic Supervision [0.0]
  We train a VAE-based world model on random embodied exploration.<n>We find that its latent space develops spatial structure semantic that mirrors physical geometry.<n>These findings establish physical world geometry as the organizing principle of world model representations.
  arXiv  Detail & Related papers  (2026-05-22T03:31:47Z)
• Amplifying, Not Learning: Fine-Tuned AI Text Detectors Amplify a Pretrained Direction [51.56484100374058]
  Text detectors amplify a pretrained typicality axis; they do not construct an AI-vs-human boundary.<n>On raw encoders before any task supervision, projecting onto centroid(AI)-centroid(HC3) achieves NYT-vs-HC3 AUROC 0.806/0.944/0.834 across three architectures.<n>On RoBERTa-base, raw projection exceeds fine-tuning; on RoBERTa-base, full fine-tuning reduces discrimination below raw on both fluent-formal populations tested.
  arXiv  Detail & Related papers  (2026-05-20T19:08:38Z)
• Shared Emotion Geometry Across Small Language Models: A Cross-Architecture Study of Representation, Behavior, and Methodological Confounds [4.358468367889626]
  We extract 21-emotion vector sets from twelve small language models under a unified comprehension-mode pipeline at fp16 precision.<n>The five mature architectures share nearly identical 21-emotion geometry, with pairwise RDM Spearman correlations of 0.74-0.92.<n>We show that what prior work has read as a single comprehension-vs.generation method effect in fact decomposes into four distinct layers.
  arXiv  Detail & Related papers  (2026-04-13T06:27:40Z)
• LAG-XAI: A Lie-Inspired Affine Geometric Framework for Interpretable Paraphrasing in Transformer Latent Spaces [1.3788139387418392]
  This paper introduces LAG-XAI, a geometric framework that models paraphrasing as a structured affine transformation within the embedding space.<n>Experiments on the noisy PIT-2015 Twitter corpus, encoded with Sentence-BERT, reveal a "linear transparency" phenomenon.<n>The model captures approximately 80% of the non-linear baseline's effective classification capacity (AUC 0.8405), offering explicit interpretability in exchange for a marginal drop in absolute accuracy.
  arXiv  Detail & Related papers  (2026-04-07T17:02:21Z)
• Weber's Law in Transformer Magnitude Representations: Efficient Coding, Representational Geometry, and Psychophysical Laws in Language Models [0.0]
  We show that representational geometry is consistently log-compressive across three magnitude domains.<n>One model produces a human-range Weber fraction (WF = 0.20) while the other does not, and both models perform at chance on temporal and spatial discrimination.<n> causal intervention reveals a layer dissociation: early layers are functionally implicated in magnitude processing (4.1x specificity) while later layers where geometry is strongest are not causally engaged.
  arXiv  Detail & Related papers  (2026-03-21T04:38:30Z)
• Latent Semantic Manifolds in Large Language Models [0.0]
  We develop a mathematical framework that interprets Large Language Models hidden states as points on a latent semantic manifold.<n>We prove two theorems: a rate-distortion lower bound on distortion for any finite vocabulary, and a linear volume scaling law for the expressibility gap.<n>We discuss implications for architecture design, model compression, decoding strategies, and scaling laws.
  arXiv  Detail & Related papers  (2026-03-17T13:05:56Z)
• Do Foundation Models Know Geometry? Probing Frozen Features for Continuous Physical Measurement [0.0]
  Vision-language models encode geometry that their text pathway fails to express.<n>LoRA fine-tuning (r=16, 2,000 images) narrows this gap to 6.5 degrees.<n>These findings enable a single frozen backbone to function as a multi-task geometric sensor.
  arXiv  Detail & Related papers  (2026-03-06T16:48:27Z)
• Optimal Unconstrained Self-Distillation in Ridge Regression: Strict Improvements, Precise Asymptotics, and One-Shot Tuning [61.07540493350384]
  Self-distillation (SD) is the process of retraining a student on a mixture of ground-truth and the teacher's own predictions.<n>We show that for any prediction risk, the optimally mixed student improves upon the ridge teacher for every regularization level.<n>We propose a consistent one-shot tuning method to estimate $star$ without grid search, sample splitting, or refitting.
  arXiv  Detail & Related papers  (2026-02-19T17:21:15Z)
• Logit Distance Bounds Representational Similarity [18.79873056204737]
  We study a distributional distance based on logit differences and show that closeness in this distance does yield linear similarity guarantees.<n>We further show that, when model probabilities are bounded away from zero, KL divergence upper-bounds logit distance; yet the resulting bound fails to provide nontrivial control in practice.
  arXiv  Detail & Related papers  (2026-02-17T09:00:56Z)
• Seed-Induced Uniqueness in Transformer Models: Subspace Alignment Governs Subliminal Transfer [0.3805935148497361]
  We analyze subliminal transfer in Transformer models, where a teacher embeds hidden traits that can be linearly decoded by a student.<n>We find that transfer strength hinges on alignment within a trait-discriminative subspace.
  arXiv  Detail & Related papers  (2025-11-02T17:34:43Z)
• Towards Self-Supervised Covariance Estimation in Deep Heteroscedastic Regression [102.24287051757469]
  We study self-supervised covariance estimation in deep heteroscedastic regression.<n>We derive an upper bound on the 2-Wasserstein distance between normal distributions.<n>Experiments over a wide range of synthetic and real datasets demonstrate that the proposed 2-Wasserstein bound coupled with pseudo label annotations results in a computationally cheaper yet accurate deep heteroscedastic regression.
  arXiv  Detail & Related papers  (2025-02-14T22:37:11Z)
• Understanding Augmentation-based Self-Supervised Representation Learning via RKHS Approximation and Regression [53.15502562048627]
  Recent work has built the connection between self-supervised learning and the approximation of the top eigenspace of a graph Laplacian operator. This work delves into a statistical analysis of augmentation-based pretraining.
  arXiv  Detail & Related papers  (2023-06-01T15:18:55Z)
• Fundamental Limits and Tradeoffs in Invariant Representation Learning [99.2368462915979]
  Many machine learning applications involve learning representations that achieve two competing goals. Minimax game-theoretic formulation represents a fundamental tradeoff between accuracy and invariance. We provide an information-theoretic analysis of this general and important problem under both classification and regression settings.
  arXiv  Detail & Related papers  (2020-12-19T15:24:04Z)
• Rethink Maximum Mean Discrepancy for Domain Adaptation [77.2560592127872]
  This paper theoretically proves two essential facts: 1) minimizing the Maximum Mean Discrepancy equals to maximize the source and target intra-class distances respectively but jointly minimize their variance with some implicit weights, so that the feature discriminability degrades. Experiments on several benchmark datasets not only prove the validity of theoretical results but also demonstrate that our approach could perform better than the comparative state-of-art methods substantially.
  arXiv  Detail & Related papers  (2020-07-01T18:25:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.