REPAIR: REnormalizing Permuted Activations for Interpolation Repair

• URL: http://arxiv.org/abs/2211.08403v3
• Date: Mon, 25 Sep 2023 17:21:00 GMT
• Title: REPAIR: REnormalizing Permuted Activations for Interpolation Repair
• Abstract summary: We show that if the permutation invariance of neural networks is taken into account, there is likely no loss barrier to the linear variance between SGD solutions. In particular, we report a 74% barrier reduction for ResNet50 on ImageNet and 90% barrier reduction for ResNet18 on CIFAR10.
• Score: 42.167567335873954
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: In this paper we look into the conjecture of Entezari et al. (2021) which states that if the permutation invariance of neural networks is taken into account, then there is likely no loss barrier to the linear interpolation between SGD solutions. First, we observe that neuron alignment methods alone are insufficient to establish low-barrier linear connectivity between SGD solutions due to a phenomenon we call variance collapse: interpolated deep networks suffer a collapse in the variance of their activations, causing poor performance. Next, we propose REPAIR (REnormalizing Permuted Activations for Interpolation Repair) which mitigates variance collapse by rescaling the preactivations of such interpolated networks. We explore the interaction between our method and the choice of normalization layer, network width, and depth, and demonstrate that using REPAIR on top of neuron alignment methods leads to 60%-100% relative barrier reduction across a wide variety of architecture families and tasks. In particular, we report a 74% barrier reduction for ResNet50 on ImageNet and 90% barrier reduction for ResNet18 on CIFAR10.

Related papers

• Per-Loss Adapters for Gradient Conflict in Physics-Informed Neural Networks [3.1861308132183375]
  We show that PINN gradient conflict is not a uniform failure mode with one universal remedy.<n>We identify distinct PINN gradient-conflict regimes, each associated with a different intervention class.
  arXiv  Detail & Related papers  (2026-05-11T07:46:07Z)
• Shapley-Guided Neural Repair Approach via Derivative-Free Optimization [21.274208005071277]
  Existing approaches mainly involve retraining, optimization, constraint, or search algorithms.<n>We propose SHARPEN, integrating interpretable fault localization with a derivative-free optimization strategy.
  arXiv  Detail & Related papers  (2026-04-01T03:05:48Z)
• Stable Nonconvex-Nonconcave Training via Linear Interpolation [51.668052890249726]
  This paper presents a theoretical analysis of linearahead as a principled method for stabilizing (large-scale) neural network training. We argue that instabilities in the optimization process are often caused by the nonmonotonicity of the loss landscape and show how linear can help by leveraging the theory of nonexpansive operators.
  arXiv  Detail & Related papers  (2023-10-20T12:45:12Z)
• On permutation symmetries in Bayesian neural network posteriors: a variational perspective [8.310462710943971]
  We show that there is essentially no loss barrier between the local solutions of gradient descent. This raises questions for approximate inference in Bayesian neural networks. We propose a matching algorithm to search for linearly connected solutions.
  arXiv  Detail & Related papers  (2023-10-16T08:26:50Z)
• Filter Pruning For CNN With Enhanced Linear Representation Redundancy [3.853146967741941]
  We present a data-driven loss function term calculated from the correlation matrix of different feature maps in the same layer, named CCM-loss. CCM-loss provides us with another universal transcendental mathematical tool besides L*-norm regularization. In our new strategy, we mainly focus on the consistency and integrality of the information flow in the network.
  arXiv  Detail & Related papers  (2023-10-10T06:27:30Z)
• Overcoming Distribution Mismatch in Quantizing Image Super-Resolution Networks [53.23803932357899]
  quantization leads to accuracy loss in image super-resolution (SR) networks. Existing works address this distribution mismatch problem by dynamically adapting quantization ranges during test time. We propose a new quantization-aware training scheme that effectively Overcomes the Distribution Mismatch problem in SR networks.
  arXiv  Detail & Related papers  (2023-07-25T08:50:01Z)
• Differentially private training of residual networks with scale normalisation [64.60453677988517]
  We investigate the optimal choice of replacement layer for Batch Normalisation (BN) in residual networks (ResNets) We study the phenomenon of scale mixing in residual blocks, whereby the activations on the two branches are scaled differently.
  arXiv  Detail & Related papers  (2022-03-01T09:56:55Z)
• Image Superresolution using Scale-Recurrent Dense Network [30.75380029218373]
  Recent advances in the design of convolutional neural network (CNN) have yielded significant improvements in the performance of image super-resolution (SR) We propose a scale recurrent SR architecture built upon units containing series of dense connections within a residual block (Residual Dense Blocks (RDBs)) Our scale recurrent design delivers competitive performance for higher scale factors while being parametrically more efficient as compared to current state-of-the-art approaches.
  arXiv  Detail & Related papers  (2022-01-28T09:18:43Z)
• Edge Rewiring Goes Neural: Boosting Network Resilience via Policy Gradient [62.660451283548724]
  ResiNet is a reinforcement learning framework to discover resilient network topologies against various disasters and attacks. We show that ResiNet achieves a near-optimal resilience gain on multiple graphs while balancing the utility, with a large margin compared to existing approaches.
  arXiv  Detail & Related papers  (2021-10-18T06:14:28Z)
• Pruning Neural Networks with Interpolative Decompositions [5.377278489623063]
  We introduce a principled approach to neural network pruning that casts the problem as a structured low-rank matrix approximation. We demonstrate how to prune a neural network by first building a set of primitives to prune a single fully connected or convolution layer. We achieve an accuracy of 93.62 $pm$ 0.36% using VGG-16 on CIFAR-10, with a 51% FLOPS reduction.
  arXiv  Detail & Related papers  (2021-07-30T20:13:49Z)
• Optimizing Mode Connectivity via Neuron Alignment [84.26606622400423]
  Empirically, the local minima of loss functions can be connected by a learned curve in model space along which the loss remains nearly constant. We propose a more general framework to investigate effect of symmetry on landscape connectivity by accounting for the weight permutations of networks being connected.
  arXiv  Detail & Related papers  (2020-09-05T02:25:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.