Uncovering the Latent Potential of Deep Intermediate Representations

• URL: http://arxiv.org/abs/2605.23033v1
• Date: Thu, 21 May 2026 20:58:42 GMT
• Title: Uncovering the Latent Potential of Deep Intermediate Representations
• Abstract summary: We show that task-relevant information is distributed non-monotonically across layers and cannot be recovered by nave aggregation.<n>We introduce Layer-wise Optimal Embedding Selection (LOES), a constructive spectral method that identifies task-discriminative subspaces.<n>Our method reveals how semantic factors are distributed across layers, thereby enabling cross-lingual and cross-modal interpretability analyses.
• Score: 16.988545433223933
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Foundational Models pretrained on huge amount of data learn representations that evolve across depth, forming a hierarchy of embeddings with distinct semantic content and geometric structure. Contrary to the widespread practice of using only the final layer or shallow mixtures, we show that task-relevant information is distributed non-monotonically across layers and cannot be recovered by naïve aggregation. Through a geometric and empirical study across multiple modalities, we show that effective transfer depends on identifying which layers encode task-discriminative structure and how their embeddings are geometrically organized. We introduce Layer-wise Optimal Embedding Selection (LOES), a constructive spectral method that identifies task-discriminative subspaces by minimizing residual error under orthogonality and isotropy constraints. To align fine-tuning with this selection principle, we further propose Geometric Regularization Loss (GeoReg), which enforces a simplicial structure on class manifolds and stabilizes representation geometry during fine-tuning. Across a wide range of architectures, depths, modalities, and data regimes, LOES consistently outperforms standard baselines, with gains that grow as model depth increases. Beyond accuracy, our method reveals how semantic factors are distributed across layers, thereby enabling cross-lingual and cross-modal interpretability analyses. Together, our results provide strong evidence that layerwise embedding geometry is not incidental but central to how deep models represent and transfer knowledge.

Related papers

• Aristotelian Manifolds: Leveraging Platonic Perceptual Features for Backpropagation Free Rapid Concept Learning [4.941630596191806]
  This paper formalizes and systematically characterizes Aristotelian Manifolds.<n>Our characterization reveals that semantic maturation does not follow a singular, monotonic path.<n>Ultimately, this systematic characterization demonstrates that mapping the internal geometry of frozen representations provides a robust, backpropagation-free, and interpretable framework.
  arXiv  Detail & Related papers  (2026-08-21T02:38:12Z)
• From Extrinsic to Intrinsic: Geodesic-Guided Representation Learning for 3D Geometric Data [56.41479248637621]
  We introduce a novel 3D representation learning paradigm, namely textbfPRISM, for textbfPre-training.<n>PRISM learns embeddings by textbfRecovering the textbfIntrinsic textbfSurface geodesic textbfMetric.<n> Experiments demonstrate that our approach shows satisfactory accuracy, robustness, and high efficiency in geodesic distance prediction.
  arXiv  Detail & Related papers  (2026-06-01T13:54:26Z)
• SEMASIA: A Large-Scale Dataset of Semantically Structured Latent Representations [12.185380843937196]
  We introduce SEMASIA, a large-scale collection of latent representations extracted from approximately 1,700 pretrained vision models.<n>We analyze the conceptual organization of individual latent spaces, showing consistent prototype-like clustering.<n>We perform a large-scale regression analysis of how pretraining-data complexity, specialization, transfer learning, augmentation, and model scale relate to geometric and probing properties of embeddings.
  arXiv  Detail & Related papers  (2026-05-10T11:42:36Z)
• Anisotropic Modality Align [91.23979617826926]
  Training multimodal large language models has long been limited by the scarcity of high-quality paired multimodal data.<n>Recent studies show that the shared representation space of pretrained multimodal contrastive models can serve as a bridge, enabling models to perform multimodal training with unimodal data.<n>The core obstacle lies in the persistent Modality Gap in the shared space.
  arXiv  Detail & Related papers  (2026-05-08T14:53:24Z)
• A Deep Generative Approach to Stratified Learning [3.235435741494221]
  Stratified learning is challenging due to varying dimensionality, singularities, and lack of efficient models.<n>We develop two generative frameworks for learning distributions on stratified spaces.
  arXiv  Detail & Related papers  (2026-04-12T13:56:59Z)
• Enhancing Gradient Inversion Attacks in Federated Learning via Hierarchical Feature Optimization [56.95448807869383]
  Federated Learning (FL) has emerged as a compelling paradigm for privacy-preserving distributed machine learning.<n>Recent studies find that the gradients exchanged in the FL system are also vulnerable to privacy leakage.<n>We propose textbfGradient textbfInversion over textbfFeature textbfDomains (GIFD)
  arXiv  Detail & Related papers  (2026-04-01T14:32:15Z)
• GeoGNN: Quantifying and Mitigating Semantic Drift in Text-Attributed Graphs [59.61242815508687]
  Graph neural networks (GNNs) on text--attributed graphs (TAGs) encode node texts using pretrained language models (PLMs) and propagate these embeddings through linear neighborhood aggregation.<n>This work introduces a local PCA-based metric that measures the degree of semantic drift and provides the first quantitative framework to analyze how different aggregation mechanisms affect manifold structure.
  arXiv  Detail & Related papers  (2025-11-12T06:48:43Z)
• The Neural Differential Manifold: An Architecture with Explicit Geometric Structure [8.201374511929538]
  This paper introduces the Neural Differential Manifold (NDM), a novel neural network architecture that explicitly incorporates geometric structure into its fundamental design.<n>We analyze the theoretical advantages of this approach, including its potential for more efficient optimization, enhanced continual learning, and applications in scientific discovery and controllable generative modeling.
  arXiv  Detail & Related papers  (2025-10-29T02:24:27Z)
• Surf-D: Generating High-Quality Surfaces of Arbitrary Topologies Using Diffusion Models [83.35835521670955]
  Surf-D is a novel method for generating high-quality 3D shapes as Surfaces with arbitrary topologies. We use the Unsigned Distance Field (UDF) as our surface representation to accommodate arbitrary topologies. We also propose a new pipeline that employs a point-based AutoEncoder to learn a compact and continuous latent space for accurately encoding UDF.
  arXiv  Detail & Related papers  (2023-11-28T18:56:01Z)
• On Characterizing the Evolution of Embedding Space of Neural Networks using Algebraic Topology [9.537910170141467]
  We study how the topology of feature embedding space changes as it passes through the layers of a well-trained deep neural network (DNN) through Betti numbers. We demonstrate that as depth increases, a topologically complicated dataset is transformed into a simple one, resulting in Betti numbers attaining their lowest possible value.
  arXiv  Detail & Related papers  (2023-11-08T10:45:12Z)
• Dual-constrained Deep Semi-Supervised Coupled Factorization Network with Enriched Prior [80.5637175255349]
  We propose a new enriched prior based Dual-constrained Deep Semi-Supervised Coupled Factorization Network, called DS2CF-Net. To ex-tract hidden deep features, DS2CF-Net is modeled as a deep-structure and geometrical structure-constrained neural network. Our network can obtain state-of-the-art performance for representation learning and clustering.
  arXiv  Detail & Related papers  (2020-09-08T13:10:21Z)
• Closed-Form Factorization of Latent Semantics in GANs [65.42778970898534]
  A rich set of interpretable dimensions has been shown to emerge in the latent space of the Generative Adversarial Networks (GANs) trained for synthesizing images. In this work, we examine the internal representation learned by GANs to reveal the underlying variation factors in an unsupervised manner. We propose a closed-form factorization algorithm for latent semantic discovery by directly decomposing the pre-trained weights.
  arXiv  Detail & Related papers  (2020-07-13T18:05:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.