Related papers: Simultaneous linear connectivity of neural networks modulo permutation

Simultaneous linear connectivity of neural networks modulo permutation

URL: http://arxiv.org/abs/2404.06498v1
Date: Tue, 9 Apr 2024 17:50:38 GMT
Title: Simultaneous linear connectivity of neural networks modulo permutation
Abstract summary: Recent work has argued that permutation symmetries are only sources permuted appropriately. In contrast, the claim "strong" that for each pair of networks exists one permutation that connects it with the other networks.
Score: 38.833555803442835
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Abstract: Neural networks typically exhibit permutation symmetries which contribute to the non-convexity of the networks' loss landscapes, since linearly interpolating between two permuted versions of a trained network tends to encounter a high loss barrier. Recent work has argued that permutation symmetries are the only sources of non-convexity, meaning there are essentially no such barriers between trained networks if they are permuted appropriately. In this work, we refine these arguments into three distinct claims of increasing strength. We show that existing evidence only supports "weak linear connectivity"-that for each pair of networks belonging to a set of SGD solutions, there exist (multiple) permutations that linearly connect it with the other networks. In contrast, the claim "strong linear connectivity"-that for each network, there exists one permutation that simultaneously connects it with the other networks-is both intuitively and practically more desirable. This stronger claim would imply that the loss landscape is convex after accounting for permutation, and enable linear interpolation between three or more independently trained models without increased loss. In this work, we introduce an intermediate claim-that for certain sequences of networks, there exists one permutation that simultaneously aligns matching pairs of networks from these sequences. Specifically, we discover that a single permutation aligns sequences of iteratively trained as well as iteratively pruned networks, meaning that two networks exhibit low loss barriers at each step of their optimization and sparsification trajectories respectively. Finally, we provide the first evidence that strong linear connectivity may be possible under certain conditions, by showing that barriers decrease with increasing network width when interpolating among three networks.

Related papers

Unrolled Neural Networks for Constrained Optimization [83.29547301151177]
Our framework comprises two coupled neural networks that jointly approximate the saddle point of the Lagrangian.<n>We numerically evaluate the framework on mixed-integer quadratic programs and power allocation in wireless networks.
arXiv Detail & Related papers (2026-01-24T03:12:41Z)
Embedding principle of homogeneous neural network for classification problem [4.418361486640713]
This paper investigates the relationship between Karush-Kuhn-Tucker (KKT) points across networks of different widths generated via neuron splitting.<n>We introduce and formalize the textbfKKT point embedding principle, establishing that KKT points of a homogeneous network's max-margin problem can be embedded into the KKT points of a larger network's problem.
arXiv Detail & Related papers (2025-05-18T13:43:22Z)
Exploring the loss landscape of regularized neural networks via convex duality [42.48510370193192]
We discuss several aspects of the loss landscape of regularized neural networks. We first characterize the solution set of the convex problem using its dual and further characterize all stationary points. We show that the solution set characterization and connectivity results can be extended to different architectures.
arXiv Detail & Related papers (2024-11-12T11:41:38Z)
When Are Bias-Free ReLU Networks Like Linear Networks? [7.197469507060226]
We show that bias-free ReLU networks have the same learning dynamics as linear networks. While bias-free ReLU networks are more expressive than their two-layer counterparts, they still share a number of similarities with deep linear networks.
arXiv Detail & Related papers (2024-06-18T13:43:58Z)
On permutation symmetries in Bayesian neural network posteriors: a variational perspective [8.310462710943971]
We show that there is essentially no loss barrier between the local solutions of gradient descent. This raises questions for approximate inference in Bayesian neural networks. We propose a matching algorithm to search for linearly connected solutions.
arXiv Detail & Related papers (2023-10-16T08:26:50Z)
Hierarchical Multi-Marginal Optimal Transport for Network Alignment [52.206006379563306]
Multi-network alignment is an essential prerequisite for joint learning on multiple networks. We propose a hierarchical multi-marginal optimal transport framework named HOT for multi-network alignment. Our proposed HOT achieves significant improvements over the state-of-the-art in both effectiveness and scalability.
arXiv Detail & Related papers (2023-10-06T02:35:35Z)
Randomly Initialized One-Layer Neural Networks Make Data Linearly Separable [1.2277343096128712]
Given sufficient width, a randomly one-layer neural network can transform two sets into two linearly separable sets without any training. This paper contributes by establishing that, given sufficient width, a randomly one-layer neural network can transform two sets into two linearly separable sets without any training.
arXiv Detail & Related papers (2022-05-24T01:38:43Z)
The Sample Complexity of One-Hidden-Layer Neural Networks [57.6421258363243]
We study a class of scalar-valued one-hidden-layer networks, and inputs bounded in Euclidean norm. We prove that controlling the spectral norm of the hidden layer weight matrix is insufficient to get uniform convergence guarantees. We analyze two important settings where a mere spectral norm control turns out to be sufficient.
arXiv Detail & Related papers (2022-02-13T07:12:02Z)
Training invariances and the low-rank phenomenon: beyond linear networks [44.02161831977037]
We show that when one trains a deep linear network with logistic or exponential loss on linearly separable data, the weights converge to rank-$1$ matrices. This is the first time a low-rank phenomenon is proven rigorously for nonlinear ReLU-activated feedforward networks. Our proof relies on a specific decomposition of the network into a multilinear function and another ReLU network whose weights are constant under a certain parameter directional convergence.
arXiv Detail & Related papers (2022-01-28T07:31:19Z)
Artificial Neural Networks generated by Low Discrepancy Sequences [59.51653996175648]
We generate artificial neural networks as random walks on a dense network graph. Such networks can be trained sparse from scratch, avoiding the expensive procedure of training a dense network and compressing it afterwards. We demonstrate that the artificial neural networks generated by low discrepancy sequences can achieve an accuracy within reach of their dense counterparts at a much lower computational complexity.
arXiv Detail & Related papers (2021-03-05T08:45:43Z)
PlueckerNet: Learn to Register 3D Line Reconstructions [57.20244406275875]
This paper proposes a neural network based method to solve the problem of Aligning two partially-overlapped 3D line reconstructions in Euclidean space. Experiments on both indoor and outdoor datasets show that the registration (rotation and translation) precision of our method outperforms baselines significantly.
arXiv Detail & Related papers (2020-12-02T11:31:56Z)
Optimizing Mode Connectivity via Neuron Alignment [84.26606622400423]
Empirically, the local minima of loss functions can be connected by a learned curve in model space along which the loss remains nearly constant. We propose a more general framework to investigate effect of symmetry on landscape connectivity by accounting for the weight permutations of networks being connected.
arXiv Detail & Related papers (2020-09-05T02:25:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.