Directly Constructing Low-Dimensional Solution Subspaces in Deep Neural Networks

• URL: http://arxiv.org/abs/2512.23410v1
• Date: Mon, 29 Dec 2025 12:13:15 GMT
• Title: Directly Constructing Low-Dimensional Solution Subspaces in Deep Neural Networks
• Abstract summary: This work provides a novel paradigm for the Trainspace Sub-Native Distillation.<n>By defining the negligible subspace directly, we provide a search for student degradation.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While it is well-established that the weight matrices and feature manifolds of deep neural networks exhibit a low Intrinsic Dimension (ID), current state-of-the-art models still rely on massive high-dimensional widths. This redundancy is not required for representation, but is strictly necessary to solve the non-convex optimization search problem-finding a global minimum, which remains intractable for compact networks. In this work, we propose a constructive approach to bypass this optimization bottleneck. By decoupling the solution geometry from the ambient search space, we empirically demonstrate across ResNet-50, ViT, and BERT that the classification head can be compressed by even huge factors of 16 with negligible performance degradation. This motivates Subspace-Native Distillation as a novel paradigm: by defining the target directly in this constructed subspace, we provide a stable geometric coordinate system for student models, potentially allowing them to circumvent the high-dimensional search problem entirely and realize the vision of Train Big, Deploy Small.

Related papers

• Learning in Low-Dimensional Subspaces: Orthogonal Bottlenecks for Reinforcement Learning [47.218458651895766]
  We present a representation-level prior that inserts a fixed orthonormal projection to constrain encoder features to a low-dimensional subspace.<n>Under a linear realizability assumption, we prove that when the bottleneck dimension exceeds the intrinsic rank of the optimal value function in feature space, the bottleneck preserves expressivity.<n>We find that across both single and multi-task benchmarks, baseline performance is either matched or improved once the bottleneck dimension exceeds a small task-dependent threshold.
  arXiv  Detail & Related papers  (2026-05-25T16:31:33Z)
• Compositional Sparsity as an Inductive Bias for Neural Architecture Design [35.06894725394093]
  We identify the structural priors that enable Deep Neural Networks to overcome the curse of dimensionality.<n>We present an interpretable pipeline in which abstraction emerges through hierarchical composition.<n>Across a broad suite of real-world datasets, HNNs consistently match or outperform dense baselines while using far fewer parameters.
  arXiv  Detail & Related papers  (2026-05-14T12:26:50Z)
• SAND: Spatially Adaptive Network Depth for Fast Sampling of Neural Implicit Surfaces [91.71563221166417]
  Implicit neural representations are powerful for geometric modeling, but their practical use is often limited by the high computational cost.<n>We propose an efficient neural implicit geometry representation framework with spatially adaptive network depth.
  arXiv  Detail & Related papers  (2026-04-15T14:14:17Z)
• The Inductive Bias of Convolutional Neural Networks: Locality and Weight Sharing Reshape Implicit Regularization [57.37943479039033]
  We study how architectural inductive bias reshapes the implicit regularization induced by the edge-of-stability phenomenon in gradient descent.<n>We show that locality and weight sharing fundamentally change this picture.
  arXiv  Detail & Related papers  (2026-03-05T04:50:51Z)
• Learning Topology-Driven Multi-Subspace Fusion for Grassmannian Deep Network [31.003374497881968]
  Grassmannian manifold offers a powerful carrier for geometric representation learning.<n>We propose a topology-driven multi-subspace fusion framework that enables adaptive subspace collaboration on the Grassmannian.<n>Our work advances geometric deep learning and adapts the proven multi-channel interaction philosophy of Euclidean networks to non-Euclidean domains.
  arXiv  Detail & Related papers  (2025-11-09T10:33:13Z)
• Connecting Independently Trained Modes via Layer-Wise Connectivity [0.4094848360328623]
  We propose a new empirical algorithm for connecting independently trained modes that generalizes beyond traditional architectures.<n>In addition to broader applicability, the proposed method yields more consistent connectivity paths across independently trained mode pairs.
  arXiv  Detail & Related papers  (2025-05-05T12:16:55Z)
• How Low Can You Go? Searching for the Intrinsic Dimensionality of Complex Networks using Metric Node Embeddings [1.7061868168035932]
  Low-dimensional embeddings are essential for machine learning tasks involving graphs.<n>We prove that lower dimensional embeddings are possible when using Euclidean metric embeddings.<n>For the first time, we demonstrate that even large-scale networks can be effectively embedded in very low-dimensional spaces.
  arXiv  Detail & Related papers  (2025-03-03T16:37:38Z)
• High-Dimensional Bayesian Optimization via Random Projection of Manifold Subspaces [0.0]
  A common framework to tackle this problem is to assume that the objective function depends on a limited set of features that lie on a low-dimensional manifold embedded in the high-dimensional ambient space.<n>This paper proposes a new approach for BO in high dimensions by exploiting a new representation of the objective function.<n>Our approach enables efficient optimizing of BO's acquisition function in the low-dimensional space, with the advantage of projecting back to the original high-dimensional space compared to existing works in the same setting.
  arXiv  Detail & Related papers  (2024-12-21T09:41:24Z)
• Efficient Compression of Overparameterized Deep Models through Low-Dimensional Learning Dynamics [10.673414267895355]
  We present a novel approach for compressing over parameterized models. Our algorithm improves the training efficiency by more than 2x, without compromising generalization.
  arXiv  Detail & Related papers  (2023-11-08T23:57:03Z)
• Large-scale global optimization of ultra-high dimensional non-convex landscapes based on generative neural networks [0.0]
  We present an algorithm manage ultra-high dimensional optimization. based on a deep generative network. We show that our method performs better with fewer function evaluations compared to state-of-the-art algorithm.
  arXiv  Detail & Related papers  (2023-07-09T00:05:59Z)
• Data-Free Learning of Reduced-Order Kinematics [54.85157881323157]
  We produce a low-dimensional map whose image parameterizes a diverse yet low-energy submanifold of configurations. We represent subspaces as neural networks that map a low-dimensional latent vector to the full configuration space. This formulation is effective across a very general range of physical systems.
  arXiv  Detail & Related papers  (2023-05-05T20:53:36Z)
• Iterative Soft Shrinkage Learning for Efficient Image Super-Resolution [91.3781512926942]
  Image super-resolution (SR) has witnessed extensive neural network designs from CNN to transformer architectures. This work investigates the potential of network pruning for super-resolution iteration to take advantage of off-the-shelf network designs and reduce the underlying computational overhead. We propose a novel Iterative Soft Shrinkage-Percentage (ISS-P) method by optimizing the sparse structure of a randomly network at each and tweaking unimportant weights with a small amount proportional to the magnitude scale on-the-fly.
  arXiv  Detail & Related papers  (2023-03-16T21:06:13Z)
• Path Regularization: A Convexity and Sparsity Inducing Regularization for Parallel ReLU Networks [75.33431791218302]
  We study the training problem of deep neural networks and introduce an analytic approach to unveil hidden convexity in the optimization landscape. We consider a deep parallel ReLU network architecture, which also includes standard deep networks and ResNets as its special cases.
  arXiv  Detail & Related papers  (2021-10-18T18:00:36Z)
• DDCNet: Deep Dilated Convolutional Neural Network for Dense Prediction [0.0]
  A receptive field (ERF) and a higher resolution of spatial features within a network are essential for providing higher-resolution dense estimates. We present a systemic approach to design network architectures that can provide a larger receptive field while maintaining a higher spatial feature resolution.
  arXiv  Detail & Related papers  (2021-07-09T23:15:34Z)
• Modeling from Features: a Mean-field Framework for Over-parameterized Deep Neural Networks [54.27962244835622]
  This paper proposes a new mean-field framework for over- parameterized deep neural networks (DNNs) In this framework, a DNN is represented by probability measures and functions over its features in the continuous limit. We illustrate the framework via the standard DNN and the Residual Network (Res-Net) architectures.
  arXiv  Detail & Related papers  (2020-07-03T01:37:16Z)
• High-dimensional Convolutional Networks for Geometric Pattern Recognition [75.43345656210992]
  We present high-dimensional convolutional networks (ConvNets) for pattern recognition problems. We first study the effectiveness of convolutional networks in detecting linear subspaces in high-dimensional spaces with up to 32 dimensions. We then apply high-dimensional ConvNets to 3D registration under rigid motions and image correspondence estimation.
  arXiv  Detail & Related papers  (2020-05-17T01:46:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.