Self-supervised local learning rules learn the hidden hierarchical structure of high-dimensional data

• URL: http://arxiv.org/abs/2605.18557v1
• Date: Mon, 18 May 2026 15:37:28 GMT
• Title: Self-supervised local learning rules learn the hidden hierarchical structure of high-dimensional data
• Abstract summary: We study how deep neural networks learn the intrinsic hierarchical structure of high-dimensional data.<n>We focus on two types of local learning rules that avoid both a long convergence time and the use of a symmetric error network.
• Score: 6.184770966699031
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: The brain learns abstract representations of high-dimensional sensory input, but the plasticity rules that enable such learning are unknown. We study biologically plausible algorithms on the Random Hierarchy Model (RHM), an artificial dataset designed to investigate how deep neural networks learn the intrinsic hierarchical structure of high-dimensional data. We focus on two types of local learning rules that avoid both a long convergence time and the use of a symmetric error network. The first type uses direct feedback signals to approximate error propagation from the output layer. The second type uses layerwise self-supervised contrastive or non-contrastive loss functions that do not explicitly approximate errors at the output layer. We show that all rules of the first type fail to solve the tasks of the RHM and trace this failure back to input-specific nonlinearities (`masking') that are implemented in full backpropagation and are essential for learning complex tasks. However, algorithms of the second type are able to learn the hierarchical hidden structure of the RHM tasks and are as data-efficient as supervised backpropagation training, while being compatible with known rules of synaptic plasticity in cortex.

Related papers

• Supervised Hebbian learning in Deep Counterstream Associative Networks [0.0]
  Modern machine learning applications employ deep neural networks training with the error backpropagation algorithm.<n>This algorithm is very effective, but lacks biological realism.<n>I propose supervised counterstream learning in deep associative networks.
  arXiv  Detail & Related papers  (2026-06-28T17:44:13Z)
• Provable Learning of Random Hierarchy Models and Hierarchical Shallow-to-Deep Chaining [58.69016084278948]
  We consider a hierarchical context-free grammar introduced by arXiv:2307.02129 and conjectured to separate deep and shallow networks.<n>We prove that, under mild conditions, a deep convolutional network can be efficiently trained to learn this function class.
  arXiv  Detail & Related papers  (2026-01-27T16:19:54Z)
• Repetita Iuvant: Data Repetition Allows SGD to Learn High-Dimensional Multi-Index Functions [20.036783417617652]
  We investigate the training dynamics of two-layer shallow neural networks trained with gradient-based algorithms.<n>We show that a simple modification of the idealized single-pass gradient descent training scenario drastically improves its computational efficiency.<n>Our results highlight the ability of networks to learn relevant structures from data alone without any pre-processing.
  arXiv  Detail & Related papers  (2024-05-24T11:34:31Z)
• Self-Supervised Training with Autoencoders for Visual Anomaly Detection [61.62861063776813]
  We focus on a specific use case in anomaly detection where the distribution of normal samples is supported by a lower-dimensional manifold. We adapt a self-supervised learning regime that exploits discriminative information during training but focuses on the submanifold of normal examples. We achieve a new state-of-the-art result on the MVTec AD dataset -- a challenging benchmark for visual anomaly detection in the manufacturing domain.
  arXiv  Detail & Related papers  (2022-06-23T14:16:30Z)
• Stacked unsupervised learning with a network architecture found by supervised meta-learning [4.209801809583906]
  Stacked unsupervised learning seems more biologically plausible than backpropagation. But SUL has fallen far short of backpropagation in practical applications. We show an SUL algorithm that can perform completely unsupervised clustering of MNIST digits.
  arXiv  Detail & Related papers  (2022-06-06T16:17:20Z)
• Data-driven emergence of convolutional structure in neural networks [83.4920717252233]
  We show how fully-connected neural networks solving a discrimination task can learn a convolutional structure directly from their inputs. By carefully designing data models, we show that the emergence of this pattern is triggered by the non-Gaussian, higher-order local structure of the inputs.
  arXiv  Detail & Related papers  (2022-02-01T17:11:13Z)
• Biologically Plausible Training Mechanisms for Self-Supervised Learning in Deep Networks [14.685237010856953]
  We develop biologically plausible training mechanisms for self-supervised learning (SSL) in deep networks. We show that learning can be performed with one of two more plausible alternatives to backpagation.
  arXiv  Detail & Related papers  (2021-09-30T12:56:57Z)
• Unsupervised Deep Learning by Injecting Low-Rank and Sparse Priors [5.5586788751870175]
  We focus on employing sparsity-inducing priors in deep learning to encourage the network to concisely capture the nature of high-dimensional data. We demonstrate unsupervised learning of U-Net for background subtraction using low-rank and sparse priors.
  arXiv  Detail & Related papers  (2021-06-21T08:41:02Z)
• A neural anisotropic view of underspecification in deep learning [60.119023683371736]
  We show that the way neural networks handle the underspecification of problems is highly dependent on the data representation. Our results highlight that understanding the architectural inductive bias in deep learning is fundamental to address the fairness, robustness, and generalization of these systems.
  arXiv  Detail & Related papers  (2021-04-29T14:31:09Z)
• Topological obstructions in neural networks learning [67.8848058842671]
  We study global properties of the loss gradient function flow. We use topological data analysis of the loss function and its Morse complex to relate local behavior along gradient trajectories with global properties of the loss surface.
  arXiv  Detail & Related papers  (2020-12-31T18:53:25Z)
• A Novel Anomaly Detection Algorithm for Hybrid Production Systems based on Deep Learning and Timed Automata [73.38551379469533]
  DAD:DeepAnomalyDetection is a new approach for automatic model learning and anomaly detection in hybrid production systems. It combines deep learning and timed automata for creating behavioral model from observations. The algorithm has been applied to few data sets including two from real systems and has shown promising results.
  arXiv  Detail & Related papers  (2020-10-29T08:27:43Z)
• Understanding Self-supervised Learning with Dual Deep Networks [74.92916579635336]
  We propose a novel framework to understand contrastive self-supervised learning (SSL) methods that employ dual pairs of deep ReLU networks. We prove that in each SGD update of SimCLR with various loss functions, the weights at each layer are updated by a emphcovariance operator. To further study what role the covariance operator plays and which features are learned in such a process, we model data generation and augmentation processes through a emphhierarchical latent tree model (HLTM)
  arXiv  Detail & Related papers  (2020-10-01T17:51:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.