Cumulative-Goodness Free-Riding in Forward-Forward Networks: Real, Repairable, but Not Accuracy-Dominant

• URL: http://arxiv.org/abs/2605.06240v1
• Date: Thu, 07 May 2026 13:24:58 GMT
• Title: Cumulative-Goodness Free-Riding in Forward-Forward Networks: Real, Repairable, but Not Accuracy-Dominant
• Abstract summary: Forward-Forward (FF) training allows each layer to learn from a local goodness criterion.<n>In cumulative-goodness variants, later layers can inherit a task that earlier layers have already partially separated.<n>We formalize this phenomenon as layer free-riding.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Forward-Forward (FF) training allows each layer to learn from a local goodness criterion. In cumulative-goodness variants, however, later layers can inherit a task that earlier layers have already partially separated. We formalize this phenomenon as layer free-riding: under the softplus FF criterion, the class-discrimination gradient reaching block $d$ decays exponentially with the positive margin accumulated by preceding blocks. We then study three local remedies -- per-block, hardness-gated, and depth-scaled -- that recover current-layer separation measures without relying on backpropagated gradients. On CIFAR-10 and CIFAR-100, these remedies dramatically improve layer-separation statistics, with $4\times$--$45\times$ gains in deeper layers, while changing accuracy by less than one percentage point for non-degenerate training procedures. Tiny ImageNet provides a tougher cross-dataset check for our selected block-wise configuration and reveals the same qualitative gap between layer-health diagnostics and final accuracy. Calibration experiments further show that architecture and augmentation choices have a larger effect on final accuracy than the training-rule modifications studied here. Cumulative free-riding is therefore a real and repairable optimization pathology. Nonetheless, for the FF training rules, architectures, and datasets we study, it is not the dominant factor limiting achievable accuracy.

Related papers

• Gradient Concentration, Not Weight Saliency, Explains Representation-Level Class Unlearning [3.33554314544636]
  We perform the first controlled ablation of the saliency masking mechanism used by SalUn.<n>We find that representation-level forgetting is governed by gradient concentration and representation geometry rather than by the specific identity of saliency-selected weights.
  arXiv  Detail & Related papers  (2026-07-23T14:17:51Z)
• Scaling Adaptive Depth with Norm-Agnostic Residual Networks [9.219329760729629]
  NAG is a norm-agnostic residual architecture that separates magnitude from directional information in the residual stream.<n>NAG introduces only a negligible number of additional parameters and relies on simple operations that are easily kernel-fusible.<n>We show that this architecture outperforms baseline Transformers, with gains that increase substantially as depth grows.
  arXiv  Detail & Related papers  (2026-06-15T02:05:37Z)
• Enhancing Gradient Inversion Attacks in Federated Learning via Hierarchical Feature Optimization [56.95448807869383]
  Federated Learning (FL) has emerged as a compelling paradigm for privacy-preserving distributed machine learning.<n>Recent studies find that the gradients exchanged in the FL system are also vulnerable to privacy leakage.<n>We propose textbfGradient textbfInversion over textbfFeature textbfDomains (GIFD)
  arXiv  Detail & Related papers  (2026-04-01T14:32:15Z)
• OrthoGrad Improves Neural Calibration [0.0]
  $perp$Grad constrains descent directions to address overconfidence.<n>$perp$Grad is a geometry-aware modification to optimization.
  arXiv  Detail & Related papers  (2025-06-04T22:12:46Z)
• Determining Layer-wise Sparsity for Large Language Models Through a Theoretical Perspective [55.90119819642064]
  We address the challenge of determining the layer-wise sparsity rates of large language models (LLMs) through a theoretical perspective.<n>This refers to the cumulative effect of reconstruction errors throughout the sparsification process.<n>We derive a simple yet effective approach to layer-wise sparsity allocation that mitigates this issue.
  arXiv  Detail & Related papers  (2025-02-20T17:51:10Z)
• A Layer Selection Approach to Test Time Adaptation [0.2968738145616401]
  Test Time Adaptation (TTA) addresses the problem of distribution shift by adapting a pretrained model to a new domain during inference.<n>We propose GALA, a novel layer selection criterion to identify the most beneficial updates to perform during TTA.
  arXiv  Detail & Related papers  (2024-04-04T19:55:11Z)
• PALM: Pushing Adaptive Learning Rate Mechanisms for Continual Test-Time Adaptation [6.181548939188321]
  Real-world vision models in dynamic environments face rapid shifts in domain distributions, leading to decreased recognition performance.<n>We propose continuous test-time adaptation (CTTA) to adjust a pre-trained source discriminative model to these changing domains.<n>We conduct extensive image classification experiments on CIFAR-10C, CIFAR-100C, and ImageNet-C, demonstrating the superior efficacy of our method compared to prior approaches.
  arXiv  Detail & Related papers  (2024-03-15T19:35:10Z)
• Scaling Forward Gradient With Local Losses [117.22685584919756]
  Forward learning is a biologically plausible alternative to backprop for learning deep neural networks. We show that it is possible to substantially reduce the variance of the forward gradient by applying perturbations to activations rather than weights. Our approach matches backprop on MNIST and CIFAR-10 and significantly outperforms previously proposed backprop-free algorithms on ImageNet.
  arXiv  Detail & Related papers  (2022-10-07T03:52:27Z)
• Test-time Batch Statistics Calibration for Covariate Shift [66.7044675981449]
  We propose to adapt the deep models to the novel environment during inference. We present a general formulation $alpha$-BN to calibrate the batch statistics. We also present a novel loss function to form a unified test time adaptation framework Core.
  arXiv  Detail & Related papers  (2021-10-06T08:45:03Z)
• Semantic Perturbations with Normalizing Flows for Improved Generalization [62.998818375912506]
  We show that perturbations in the latent space can be used to define fully unsupervised data augmentations. We find that our latent adversarial perturbations adaptive to the classifier throughout its training are most effective.
  arXiv  Detail & Related papers  (2021-08-18T03:20:00Z)
• Unbiased Risk Estimators Can Mislead: A Case Study of Learning with Complementary Labels [92.98756432746482]
  We study a weakly supervised problem called learning with complementary labels. We show that the quality of gradient estimation matters more in risk minimization. We propose a novel surrogate complementary loss(SCL) framework that trades zero bias with reduced variance.
  arXiv  Detail & Related papers  (2020-07-05T04:19:37Z)
• A block coordinate descent optimizer for classification problems exploiting convexity [0.0]
  We introduce a coordinate descent method to deep linear networks for classification tasks that exploits convexity of the cross-entropy loss in the weights of the hidden layer. By alternating between a second-order method to find globally optimal parameters for the linear layer and gradient descent to the hidden layers, we ensure an optimal fit of the adaptive basis to data throughout training.
  arXiv  Detail & Related papers  (2020-06-17T19:49:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.