Convergence of Spectral Principal Paths: How Deep Networks Distill Linear Representations from Noisy Inputs

• URL: http://arxiv.org/abs/2506.08543v1
• Date: Tue, 10 Jun 2025 08:08:52 GMT
• Title: Convergence of Spectral Principal Paths: How Deep Networks Distill Linear Representations from Noisy Inputs
• Authors: Bowei Tian, Xuntao Lyu, Meng Liu, Hongyi Wang, Ang Li,
• Abstract summary: High-level representations have become a central focus in enhancing AI transparency and control, shifting attention from individual neurons or circuits to structured semantic directions that align with human-interpretable concepts.<n>Motivated by the Linear Representation Hypothesis (LRH), we propose the Input-Space Linearity Hypothesis (ISLH), which posits that concept-aligned directions originate in the input space and are selectively amplified with increasing depth.<n>We then introduce the Spectral Principal Path (SPP) framework, which formalizes how deep networks progressively distill linear representations along a small set of dominant spectral directions.
• Score: 17.987141330832582
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: High-level representations have become a central focus in enhancing AI transparency and control, shifting attention from individual neurons or circuits to structured semantic directions that align with human-interpretable concepts. Motivated by the Linear Representation Hypothesis (LRH), we propose the Input-Space Linearity Hypothesis (ISLH), which posits that concept-aligned directions originate in the input space and are selectively amplified with increasing depth. We then introduce the Spectral Principal Path (SPP) framework, which formalizes how deep networks progressively distill linear representations along a small set of dominant spectral directions. Building on this framework, we further demonstrate the multimodal robustness of these representations in Vision-Language Models (VLMs). By bridging theoretical insights with empirical validation, this work advances a structured theory of representation formation in deep networks, paving the way for improving AI robustness, fairness, and transparency.

Related papers

• Why Representation Engineering Works: A Theoretical and Empirical Study in Vision-Language Models [17.987141330832582]
  We develop a theoretical framework that explains the stability of neural activity across layers using the principal eigenvector.<n>This work transforms Representation Engineering (RepE) into a structured theoretical framework, opening new directions for improving AI robustness, fairness, and transparency.
  arXiv  Detail & Related papers  (2025-03-25T20:32:15Z)
• Approaching Deep Learning through the Spectral Dynamics of Weights [41.948042468042374]
  spectral dynamics of weights -- the behavior of singular values and vectors during optimization -- to clarify and unify several phenomena in deep learning. We identify a consistent bias in optimization across various experiments, from small-scale grokking'' to large-scale tasks like image classification with ConvNets, image generation with UNets, speech recognition with LSTMs, and language modeling with Transformers.
  arXiv  Detail & Related papers  (2024-08-21T17:48:01Z)
• Learned Image Reasoning Prior Penetrates Deep Unfolding Network for Panchromatic and Multi-Spectral Image Fusion [45.28120834593148]
  We propose a novel model-driven deep unfolding framework with image reasoning prior tailored for the pan-sharpening task. Our framework is motivated by the content reasoning ability of masked autoencoders with insightful designs. The uniqueness of our framework is that the holistic learning process is explicitly integrated with the inherent physical mechanism underlying the pan-sharpening task.
  arXiv  Detail & Related papers  (2023-08-30T15:15:31Z)
• Unsupervised Interpretable Basis Extraction for Concept-Based Visual Explanations [53.973055975918655]
  We show that, intermediate layer representations become more interpretable when transformed to the bases extracted with our method. We compare the bases extracted with our method with the bases derived with a supervised approach and find that, in one aspect, the proposed unsupervised approach has a strength that constitutes a limitation of the supervised one and give potential directions for future research.
  arXiv  Detail & Related papers  (2023-03-19T00:37:19Z)
• Guiding the PLMs with Semantic Anchors as Intermediate Supervision: Towards Interpretable Semantic Parsing [57.11806632758607]
  We propose to incorporate the current pretrained language models with a hierarchical decoder network. By taking the first-principle structures as the semantic anchors, we propose two novel intermediate supervision tasks. We conduct intensive experiments on several semantic parsing benchmarks and demonstrate that our approach can consistently outperform the baselines.
  arXiv  Detail & Related papers  (2022-10-04T07:27:29Z)
• USegScene: Unsupervised Learning of Depth, Optical Flow and Ego-Motion with Semantic Guidance and Coupled Networks [31.600708674008384]
  USegScene is a framework for semantically guided unsupervised learning of depth, optical flow and ego-motion estimation for stereo camera images. We present results on the popular KITTI dataset and show that our approach outperforms other methods by a large margin.
  arXiv  Detail & Related papers  (2022-07-15T13:25:47Z)
• Discovering "Semantics" in Super-Resolution Networks [54.45509260681529]
  Super-resolution (SR) is a fundamental and representative task of low-level vision area. It is generally thought that the features extracted from the SR network have no specific semantic information. Can we find any "semantics" in SR networks?
  arXiv  Detail & Related papers  (2021-08-01T09:12:44Z)
• The Principles of Deep Learning Theory [19.33681537640272]
  This book develops an effective theory approach to understanding deep neural networks of practical relevance. We explain how these effectively-deep networks learn nontrivial representations from training. We show that the depth-to-width ratio governs the effective model complexity of the ensemble of trained networks.
  arXiv  Detail & Related papers  (2021-06-18T15:00:00Z)
• Variational Structured Attention Networks for Deep Visual Representation Learning [49.80498066480928]
  We propose a unified deep framework to jointly learn both spatial attention maps and channel attention in a principled manner. Specifically, we integrate the estimation and the interaction of the attentions within a probabilistic representation learning framework. We implement the inference rules within the neural network, thus allowing for end-to-end learning of the probabilistic and the CNN front-end parameters.
  arXiv  Detail & Related papers  (2021-03-05T07:37:24Z)
• Closed-Form Factorization of Latent Semantics in GANs [65.42778970898534]
  A rich set of interpretable dimensions has been shown to emerge in the latent space of the Generative Adversarial Networks (GANs) trained for synthesizing images. In this work, we examine the internal representation learned by GANs to reveal the underlying variation factors in an unsupervised manner. We propose a closed-form factorization algorithm for latent semantic discovery by directly decomposing the pre-trained weights.
  arXiv  Detail & Related papers  (2020-07-13T18:05:36Z)
• Dynamic Hierarchical Mimicking Towards Consistent Optimization Objectives [73.15276998621582]
  We propose a generic feature learning mechanism to advance CNN training with enhanced generalization ability. Partially inspired by DSN, we fork delicately designed side branches from the intermediate layers of a given neural network. Experiments on both category and instance recognition tasks demonstrate the substantial improvements of our proposed method.
  arXiv  Detail & Related papers  (2020-03-24T09:56:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.