Shape Happens: Automatic Feature Manifold Discovery in LLMs via Supervised Multi-Dimensional Scaling

• URL: http://arxiv.org/abs/2510.01025v1
• Date: Wed, 01 Oct 2025 15:30:47 GMT
• Title: Shape Happens: Automatic Feature Manifold Discovery in LLMs via Supervised Multi-Dimensional Scaling
• Authors: Federico Tiblias, Irina Bigoulaeva, Jingcheng Niu, Simone Balloccu, Iryna Gurevych,
• Abstract summary: We introduce Supervised Multi-Dimensional Scaling (SMDS), a model-agnostic method to automatically discover feature manifold.<n>We find that different features form various geometric structures such as circles, lines, and clusters.<n>Our findings shed light on the functional role of feature manifold, supporting a model of entity-based reasoning in which LMs encode and transform structured representations.
• Score: 43.70541744169649
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The linear representation hypothesis states that language models (LMs) encode concepts as directions in their latent space, forming organized, multidimensional manifolds. Prior efforts focus on discovering specific geometries for specific features, and thus lack generalization. We introduce Supervised Multi-Dimensional Scaling (SMDS), a model-agnostic method to automatically discover feature manifolds. We apply SMDS to temporal reasoning as a case study, finding that different features form various geometric structures such as circles, lines, and clusters. SMDS reveals many insights on these structures: they consistently reflect the properties of the concepts they represent; are stable across model families and sizes; actively support reasoning in models; and dynamically reshape in response to context changes. Together, our findings shed light on the functional role of feature manifolds, supporting a model of entity-based reasoning in which LMs encode and transform structured representations.

Related papers

• The Shape of Beliefs: Geometry, Dynamics, and Interventions along Representation Manifolds of Language Models' Posteriors [24.477029700560113]
  Large language models (LLMs) represent prompt-conditioned beliefs (posteriors over answers and claims)<n>We study a controlled setting in which Llama-3.2 generates samples from a normal distribution by implicitly inferring its parameters.<n>We find representations of curved "belief manifold" for these parameters form with sufficient in-context learning.
  arXiv  Detail & Related papers  (2026-02-02T16:45:05Z)
• Bridging Structure and Appearance: Topological Features for Robust Self-Supervised Segmentation [8.584363058858935]
  Self-supervised semantic segmentation methods often fail when faced with appearance ambiguities.<n>We argue that this is due to an over-reliance on unstable, appearance-based features such as shadows, glare, and local textures.<n>We propose textbfGASeg, a novel framework that bridges appearance and geometry by leveraging stable topological information.
  arXiv  Detail & Related papers  (2025-12-30T05:34:28Z)
• Geometry of Decision Making in Language Models [19.74354232642455]
  Large Language Models (LLMs) show strong generalization across diverse tasks, yet the internal decision-making processes behind their predictions remain opaque.<n>We study the geometry of hidden representations in LLMs through the lens of textitintrinsic dimension (ID)<n>We perform a large-scale study, with 28 open-weight transformer models and estimate ID across layers using multiple estimators.
  arXiv  Detail & Related papers  (2025-11-25T13:52:46Z)
• Information Structure in Mappings: An Approach to Learning, Representation, and Generalisation [3.8073142980733]
  This thesis introduces quantitative methods for identifying systematic structure in a mapping between spaces.<n>I identify structural primitives present in a mapping, along with information theoretics of each.<n>I also introduce a novel, performant, approach to estimating the entropy of vector space, that allows this analysis to be applied to models ranging in size from 1 million to 12 billion parameters.
  arXiv  Detail & Related papers  (2025-05-29T19:27:50Z)
• PRISM: Probabilistic Representation for Integrated Shape Modeling and Generation [79.46526296655776]
  PRISM is a novel approach for 3D shape generation that integrates categorical diffusion models with Statistical Shape Models (SSM) and Gaussian Mixture Models (GMM)<n>Our method employs compositional SSMs to capture part-level geometric variations and uses GMM to represent part semantics in a continuous space.<n>Our approach significantly outperforms previous methods in both quality and controllability of part-level operations.
  arXiv  Detail & Related papers  (2025-04-06T11:48:08Z)
• Geometric Origins of Bias in Deep Neural Networks: A Human Visual System Perspective [1.7315645623674356]
  Bias formation in deep neural networks (DNNs) remains a critical yet poorly understood challenge.<n>Inspired by the human visual system, we propose a geometric analysis framework linking the geometric complexity of class-specific perceptual Manifolds to model bias.<n>To support this analysis, we present the Perceptual-Manifold-Geometry library, designed for calculating the geometric properties of perceptual Manifolds.
  arXiv  Detail & Related papers  (2025-02-17T13:54:02Z)
• Analyzing Finetuning Representation Shift for Multimodal LLMs Steering [56.710375516257876]
  We propose to map hidden states to interpretable visual and textual concepts.<n>This enables us to more efficiently compare certain semantic dynamics, such as the shift from an original and fine-tuned model.<n>We also demonstrate the use of shift vectors to capture these concepts changes.
  arXiv  Detail & Related papers  (2025-01-06T13:37:13Z)
• Persistent Topological Features in Large Language Models [0.6597195879147556]
  We introduce topological descriptors that measure how topological features, $p$-dimensional holes, persist and evolve throughout the layers.<n>This offers a statistical perspective on how prompts are rearranged and their relative positions changed in the representation space.<n>As a showcase application, we use zigzag persistence to establish a criterion for layer pruning, achieving results comparable to state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-10-14T19:46:23Z)
• GeoMFormer: A General Architecture for Geometric Molecular Representation Learning [84.02083170392764]
  We introduce a novel Transformer-based molecular model called GeoMFormer to achieve this goal. We show that GeoMFormer achieves strong performance on both invariant and equivariant tasks of different types and scales.
  arXiv  Detail & Related papers  (2024-06-24T17:58:13Z)
• The Geometry of Self-supervised Learning Models and its Impact on Transfer Learning [62.601681746034956]
  Self-supervised learning (SSL) has emerged as a desirable paradigm in computer vision. We propose a data-driven geometric strategy to analyze different SSL models using local neighborhoods in the feature space induced by each.
  arXiv  Detail & Related papers  (2022-09-18T18:15:38Z)
• Closed-Form Factorization of Latent Semantics in GANs [65.42778970898534]
  A rich set of interpretable dimensions has been shown to emerge in the latent space of the Generative Adversarial Networks (GANs) trained for synthesizing images. In this work, we examine the internal representation learned by GANs to reveal the underlying variation factors in an unsupervised manner. We propose a closed-form factorization algorithm for latent semantic discovery by directly decomposing the pre-trained weights.
  arXiv  Detail & Related papers  (2020-07-13T18:05:36Z)
• Learning Bijective Feature Maps for Linear ICA [73.85904548374575]
  We show that existing probabilistic deep generative models (DGMs) which are tailor-made for image data, underperform on non-linear ICA tasks. To address this, we propose a DGM which combines bijective feature maps with a linear ICA model to learn interpretable latent structures for high-dimensional data. We create models that converge quickly, are easy to train, and achieve better unsupervised latent factor discovery than flow-based models, linear ICA, and Variational Autoencoders on images.
  arXiv  Detail & Related papers  (2020-02-18T17:58:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.