Feature learning from non-Gaussian inputs: the case of Independent Component Analysis in high dimensions

• URL: http://arxiv.org/abs/2503.23896v1
• Date: Mon, 31 Mar 2025 09:46:47 GMT
• Title: Feature learning from non-Gaussian inputs: the case of Independent Component Analysis in high dimensions
• Abstract summary: We investigate the interplay between data structure in feature learning for the most popular ICA algorithm, FastICA, and gradient descent (SGD)<n>We show that vanilla online SGD outperforms FastICA, and prove that the optimal sample complexity $ngtrsim d2$ can be reached by smoothing the loss.
• Score: 7.2136602534376015
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deep neural networks learn structured features from complex, non-Gaussian inputs, but the mechanisms behind this process remain poorly understood. Our work is motivated by the observation that the first-layer filters learnt by deep convolutional neural networks from natural images resemble those learnt by independent component analysis (ICA), a simple unsupervised method that seeks the most non-Gaussian projections of its inputs. This similarity suggests that ICA provides a simple, yet principled model for studying feature learning. Here, we leverage this connection to investigate the interplay between data structure and optimisation in feature learning for the most popular ICA algorithm, FastICA, and stochastic gradient descent (SGD), which is used to train deep networks. We rigorously establish that FastICA requires at least $n\gtrsim d^4$ samples to recover a single non-Gaussian direction from $d$-dimensional inputs on a simple synthetic data model. We show that vanilla online SGD outperforms FastICA, and prove that the optimal sample complexity $n \gtrsim d^2$ can be reached by smoothing the loss, albeit in a data-dependent way. We finally demonstrate the existence of a search phase for FastICA on ImageNet, and discuss how the strong non-Gaussianity of said images compensates for the poor sample complexity of FastICA.

Related papers

• A Fourier perspective on the learning dynamics of neural networks: from sample complexities to mechanistic insights [8.973965016201822]
  We show that simple neural networks trained on image classification tasks first rely on amplitude information.<n>We then exploit phase information, which encodes edges and higher-order correlations.<n>We show both experimentally and theoretically that power-law spectra can dramatically accelerate the speed of learning phase information.
  arXiv  Detail & Related papers  (2026-05-16T10:03:41Z)
• A Compression Perspective on Simplicity Bias [20.205677906026924]
  We show how simplicity bias governs feature selection in neural networks.<n>We identify distinct data regimes where increasing data promotes robustness by ruling out trivial shortcuts.<n>We validate our theory on a semi-synthetic benchmark showing that the feature selection of neural networks follows the same trajectory of solutions as optimal two-part compression.
  arXiv  Detail & Related papers  (2026-03-26T19:02:49Z)
• Neural network optimization strategies and the topography of the loss landscape [45.88028371034407]
  We investigate neural network learning by gradient descent (SGD)<n>We use several computational tools to investigate neural network parameters obtained by these two optimization methods.
  arXiv  Detail & Related papers  (2026-02-24T17:49:13Z)
• Gradient Descent as a Perceptron Algorithm: Understanding Dynamics and Implicit Acceleration [67.12978375116599]
  We show that the steps of gradient descent (GD) reduce to those of generalized perceptron algorithms.<n>This helps explain the optimization dynamics and the implicit acceleration phenomenon observed in neural networks.
  arXiv  Detail & Related papers  (2025-12-12T14:16:35Z)
• Cyber Orbits of Large Scale Network Traffic [4.556941519392727]
  Physical analogies for complicated network behavior can be a useful aid to understanding these newly discovered network phenomena.<n>The probability of observing a source at a temporal distance'' $r(t)$ at time $t$ is $p(t) propto 1/r(t)2$.<n>Applying this physical analogy to the observed source correlations in the Anonymized Network Sensing Graph Challenge data leads to an elegant cyber orbit analogy.
  arXiv  Detail & Related papers  (2025-08-23T00:21:29Z)
• Global Convergence and Rich Feature Learning in $L$-Layer Infinite-Width Neural Networks under $μ$P Parametrization [66.03821840425539]
  In this paper, we investigate the training dynamics of $L$-layer neural networks using the tensor gradient program (SGD) framework.<n>We show that SGD enables these networks to learn linearly independent features that substantially deviate from their initial values.<n>This rich feature space captures relevant data information and ensures that any convergent point of the training process is a global minimum.
  arXiv  Detail & Related papers  (2025-03-12T17:33:13Z)
• Simplicity Bias via Global Convergence of Sharpness Minimization [43.658859631741024]
  We show that label noise SGD always minimizes the sharpness on the manifold of models with zero loss for two-layer networks. We also find a novel property of the trace of Hessian of the loss at approximate stationary points on the manifold of zero loss.
  arXiv  Detail & Related papers  (2024-10-21T18:10:37Z)
• Sliding down the stairs: how correlated latent variables accelerate learning with neural networks [8.107431208836426]
  We show that correlations between latent variables along directions encoded in different input cumulants speed up learning from higher-order correlations. Our results are confirmed in simulations of two-layer neural networks.
  arXiv  Detail & Related papers  (2024-04-12T17:01:25Z)
• Neural Network Pruning by Gradient Descent [7.427858344638741]
  We introduce a novel and straightforward neural network pruning framework that incorporates the Gumbel-Softmax technique. We demonstrate its exceptional compression capability, maintaining high accuracy on the MNIST dataset with only 0.15% of the original network parameters. We believe our method opens a promising new avenue for deep learning pruning and the creation of interpretable machine learning systems.
  arXiv  Detail & Related papers  (2023-11-21T11:12:03Z)
• Gradient-Based Feature Learning under Structured Data [57.76552698981579]
  In the anisotropic setting, the commonly used spherical gradient dynamics may fail to recover the true direction. We show that appropriate weight normalization that is reminiscent of batch normalization can alleviate this issue. In particular, under the spiked model with a suitably large spike, the sample complexity of gradient-based training can be made independent of the information exponent.
  arXiv  Detail & Related papers  (2023-09-07T16:55:50Z)
• Deep neural networks have an inbuilt Occam's razor [4.330196787806085]
  We show that structured data combined with an intrinsic Occam's razor-like inductive bias towards simple functions counteracts the exponential growth of functions with complexity.<n>This analysis reveals that structured data, combined with an intrinsic Occam's razor-like inductive bias towards (Kolmogorov) simple functions that is strong enough to counteract the exponential growth of functions with complexity, is a key to the success of DNNs.
  arXiv  Detail & Related papers  (2023-04-13T16:58:21Z)
• Joint Edge-Model Sparse Learning is Provably Efficient for Graph Neural Networks [89.28881869440433]
  This paper provides the first theoretical characterization of joint edge-model sparse learning for graph neural networks (GNNs) It proves analytically that both sampling important nodes and pruning neurons with the lowest-magnitude can reduce the sample complexity and improve convergence without compromising the test accuracy.
  arXiv  Detail & Related papers  (2023-02-06T16:54:20Z)
• Neural networks trained with SGD learn distributions of increasing complexity [78.30235086565388]
  We show that neural networks trained using gradient descent initially classify their inputs using lower-order input statistics. We then exploit higher-order statistics only later during training. We discuss the relation of DSB to other simplicity biases and consider its implications for the principle of universality in learning.
  arXiv  Detail & Related papers  (2022-11-21T15:27:22Z)
• Deep Architecture Connectivity Matters for Its Convergence: A Fine-Grained Analysis [94.64007376939735]
  We theoretically characterize the impact of connectivity patterns on the convergence of deep neural networks (DNNs) under gradient descent training. We show that by a simple filtration on "unpromising" connectivity patterns, we can trim down the number of models to evaluate.
  arXiv  Detail & Related papers  (2022-05-11T17:43:54Z)
• Data-driven emergence of convolutional structure in neural networks [83.4920717252233]
  We show how fully-connected neural networks solving a discrimination task can learn a convolutional structure directly from their inputs. By carefully designing data models, we show that the emergence of this pattern is triggered by the non-Gaussian, higher-order local structure of the inputs.
  arXiv  Detail & Related papers  (2022-02-01T17:11:13Z)
• Generalized Approach to Matched Filtering using Neural Networks [4.535489275919893]
  We make a key observation on the relationship between the emerging deep learning and the traditional techniques. matched filtering is formally equivalent to a particular neural network. We show that the proposed neural network architecture can outperform matched filtering.
  arXiv  Detail & Related papers  (2021-04-08T17:59:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.