Can Local Learning Match Self-Supervised Backpropagation?

• URL: http://arxiv.org/abs/2601.21683v1
• Date: Thu, 29 Jan 2026 13:15:57 GMT
• Title: Can Local Learning Match Self-Supervised Backpropagation?
• Authors: Wu S. Zihan, Ariane Delrocq, Wulfram Gerstner, Guillaume Bellec,
• Abstract summary: We develop a theory for deep linear networks to establish a link between global and local rules.<n>We then develop novel variants of local-SSL algorithms to approximate global BP-SSL in deep non-linear convolutional neural networks.<n>Variants that improve the similarity between gradient updates of local-SSL with those of global BP-SSL also show better performance on image datasets.
• Score: 6.184770966699031
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: While end-to-end self-supervised learning with backpropagation (global BP-SSL) has become central for training modern AI systems, theories of local self-supervised learning (local-SSL) have struggled to build functional representations in deep neural networks. To establish a link between global and local rules, we first develop a theory for deep linear networks: we identify conditions for local-SSL algorithms (like Forward-forward or CLAPP) to implement exactly the same weight update as a global BP-SSL. Starting from the theoretical insights, we then develop novel variants of local-SSL algorithms to approximate global BP-SSL in deep non-linear convolutional neural networks. Variants that improve the similarity between gradient updates of local-SSL with those of global BP-SSL also show better performance on image datasets (CIFAR-10, STL-10, and Tiny ImageNet). The best local-SSL rule with the CLAPP loss function matches the performance of a comparable global BP-SSL with InfoNCE or CPC-like loss functions, and improves upon state-of-the-art for local SSL on these benchmarks.

Related papers

• Stochastic Layer-wise Learning: Scalable and Efficient Alternative to Backpropagation [1.0285749562751982]
  Backpropagation underpins modern deep learning, yet its reliance on global synchronization limits scalability and incurs high memory costs.<n>In contrast, fully local learning rules are more efficient but often struggle to maintain the cross-layer coordination needed for coherent global learning.<n>We introduce Layer-wise Learning (SLL), a layer-wise training algorithm that decomposes the global objective into coordinated layer-local updates.
  arXiv  Detail & Related papers  (2025-05-08T12:32:29Z)
• Towards Interpretable Deep Local Learning with Successive Gradient Reconciliation [70.43845294145714]
  Relieving the reliance of neural network training on a global back-propagation (BP) has emerged as a notable research topic. We propose a local training strategy that successively regularizes the gradient reconciliation between neighboring modules. Our method can be integrated into both local-BP and BP-free settings.
  arXiv  Detail & Related papers  (2024-06-07T19:10:31Z)
• LW-FedSSL: Resource-efficient Layer-wise Federated Self-supervised Learning [14.937324556028548]
  We propose the Layer-Wise Federated Self-Supervised Learning approach, which allows edge devices to incrementally train a small part of the model at a time.<n>Since only a portion of the model is active for training at any given time, LW-FedSSL significantly reduces computational requirements.<n>It can achieve up to a $3.34 times$ reduction in memory usage, $4.20 times$ fewer computational operations (GFLOPs), and a $5.07 times$ lower communication cost.
  arXiv  Detail & Related papers  (2024-01-22T01:57:31Z)
• OpenLDN: Learning to Discover Novel Classes for Open-World Semi-Supervised Learning [110.40285771431687]
  Semi-supervised learning (SSL) is one of the dominant approaches to address the annotation bottleneck of supervised learning. Recent SSL methods can effectively leverage a large repository of unlabeled data to improve performance while relying on a small set of labeled data. This work introduces OpenLDN that utilizes a pairwise similarity loss to discover novel classes.
  arXiv  Detail & Related papers  (2022-07-05T18:51:05Z)
• An Entropy-guided Reinforced Partial Convolutional Network for Zero-Shot Learning [77.72330187258498]
  We propose a novel Entropy-guided Reinforced Partial Convolutional Network (ERPCNet) ERPCNet extracts and aggregates localities based on semantic relevance and visual correlations without human-annotated regions. It not only discovers global-cooperative localities dynamically but also converges faster for policy gradient optimization.
  arXiv  Detail & Related papers  (2021-11-03T11:13:13Z)
• Biologically Plausible Training Mechanisms for Self-Supervised Learning in Deep Networks [14.685237010856953]
  We develop biologically plausible training mechanisms for self-supervised learning (SSL) in deep networks. We show that learning can be performed with one of two more plausible alternatives to backpagation.
  arXiv  Detail & Related papers  (2021-09-30T12:56:57Z)
• Self-Supervised Learning of Graph Neural Networks: A Unified Review [50.71341657322391]
  Self-supervised learning is emerging as a new paradigm for making use of large amounts of unlabeled samples. We provide a unified review of different ways of training graph neural networks (GNNs) using SSL. Our treatment of SSL methods for GNNs sheds light on the similarities and differences of various methods, setting the stage for developing new methods and algorithms.
  arXiv  Detail & Related papers  (2021-02-22T03:43:45Z)
• End-to-end Generative Zero-shot Learning via Few-shot Learning [76.9964261884635]
  State-of-the-art approaches to Zero-Shot Learning (ZSL) train generative nets to synthesize examples conditioned on the provided metadata. We introduce an end-to-end generative ZSL framework that uses such an approach as a backbone and feeds its synthesized output to a Few-Shot Learning algorithm.
  arXiv  Detail & Related papers  (2021-02-08T17:35:37Z)
• Interventional Few-Shot Learning [88.31112565383457]
  We propose a novel Few-Shot Learning paradigm: Interventional Few-Shot Learning. Code is released at https://github.com/yue-zhongqi/ifsl.
  arXiv  Detail & Related papers  (2020-09-28T01:16:54Z)
• Semi-supervised source localization with deep generative modeling [27.344649091365067]
  We propose a semi-supervised localization approach based on deep generative modeling with variational autoencoders (VAEs) VAE-SSL can outperform both SRP-PHAT and CNN in label-limited scenarios.
  arXiv  Detail & Related papers  (2020-05-27T04:59:52Z)
• Semi-supervised Medical Image Classification with Global Latent Mixing [8.330337646455957]
  Computer-aided diagnosis via deep learning relies on large-scale annotated data sets. Semi-supervised learning mitigates this challenge by leveraging unlabeled data. We present a novel SSL approach that trains the neural network on linear mixing of labeled and unlabeled data.
  arXiv  Detail & Related papers  (2020-05-22T14:49:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.