Modularity based linkage model for neuroevolution
- URL: http://arxiv.org/abs/2306.01227v1
- Date: Fri, 2 Jun 2023 01:32:49 GMT
- Title: Modularity based linkage model for neuroevolution
- Authors: Yukai Qiao and Marcus Gallagher
- Abstract summary: Crossover between neural networks is considered disruptive due to the strong functional dependency between connection weights.
We propose a modularity-based linkage model at the weight level to preserve functionally dependent communities.
Our algorithm finds better, more functionally dependent linkage which leads to more successful crossover and better performance.
- Score: 4.9444321684311925
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Crossover between neural networks is considered disruptive due to the strong
functional dependency between connection weights. We propose a modularity-based
linkage model at the weight level to preserve functionally dependent
communities (building blocks) in neural networks during mixing. A proximity
matrix is built by estimating the dependency between weights, then a community
detection algorithm maximizing modularity is run on the graph described by such
matrix. The resulting communities/groups of parameters are considered to be
mutually independent and used as crossover masks in an optimal mixing EA. A
variant is tested with an operator that neutralizes the permutation problem of
neural networks to a degree. Experiments were performed on 8 and 10-bit parity
problems as the intrinsic hierarchical nature of the dependencies in these
problems are challenging to learn. The results show that our algorithm finds
better, more functionally dependent linkage which leads to more successful
crossover and better performance.
Related papers
- Message Passing Variational Autoregressive Network for Solving Intractable Ising Models [6.261096199903392]
Many deep neural networks have been used to solve Ising models, including autoregressive neural networks, convolutional neural networks, recurrent neural networks, and graph neural networks.
Here we propose a variational autoregressive architecture with a message passing mechanism, which can effectively utilize the interactions between spin variables.
The new network trained under an annealing framework outperforms existing methods in solving several prototypical Ising spin Hamiltonians, especially for larger spin systems at low temperatures.
arXiv Detail & Related papers (2024-04-09T11:27:07Z) - What to Do When Your Discrete Optimization Is the Size of a Neural
Network? [24.546550334179486]
Machine learning applications using neural networks involve solving discrete optimization problems.
classical approaches used in discrete settings do not scale well to large neural networks.
We take continuation path (CP) methods to represent using purely the former and Monte Carlo (MC) methods to represent the latter.
arXiv Detail & Related papers (2024-02-15T21:57:43Z) - Nonlinear subspace clustering by functional link neural networks [20.972039615938193]
Subspace clustering based on a feed-forward neural network has been demonstrated to provide better clustering accuracy than some advanced subspace clustering algorithms.
We employ a functional link neural network to transform data samples into a nonlinear domain.
We introduce a convex combination subspace clustering scheme, which combines a linear subspace clustering method with the functional link neural network subspace clustering approach.
arXiv Detail & Related papers (2024-02-03T06:01:21Z) - Neural Dependencies Emerging from Learning Massive Categories [94.77992221690742]
This work presents two astonishing findings on neural networks learned for large-scale image classification.
1) Given a well-trained model, the logits predicted for some category can be directly obtained by linearly combining the predictions of a few other categories.
2) Neural dependencies exist not only within a single model, but even between two independently learned models.
arXiv Detail & Related papers (2022-11-21T09:42:15Z) - Deep Architecture Connectivity Matters for Its Convergence: A
Fine-Grained Analysis [94.64007376939735]
We theoretically characterize the impact of connectivity patterns on the convergence of deep neural networks (DNNs) under gradient descent training.
We show that by a simple filtration on "unpromising" connectivity patterns, we can trim down the number of models to evaluate.
arXiv Detail & Related papers (2022-05-11T17:43:54Z) - Physics-informed neural network simulation of multiphase poroelasticity
using stress-split sequential training [0.0]
We present a framework for solving problems governed by partial differential equations (PDEs) based on elastic networks.
We find that the approach converges to solve problems of porosci, Barry-Scier's injection-production problem, and a two-phase drainage problem.
arXiv Detail & Related papers (2021-10-06T20:09:09Z) - Efficient Model-Based Multi-Agent Mean-Field Reinforcement Learning [89.31889875864599]
We propose an efficient model-based reinforcement learning algorithm for learning in multi-agent systems.
Our main theoretical contributions are the first general regret bounds for model-based reinforcement learning for MFC.
We provide a practical parametrization of the core optimization problem.
arXiv Detail & Related papers (2021-07-08T18:01:02Z) - Optimizing Mode Connectivity via Neuron Alignment [84.26606622400423]
Empirically, the local minima of loss functions can be connected by a learned curve in model space along which the loss remains nearly constant.
We propose a more general framework to investigate effect of symmetry on landscape connectivity by accounting for the weight permutations of networks being connected.
arXiv Detail & Related papers (2020-09-05T02:25:23Z) - Multipole Graph Neural Operator for Parametric Partial Differential
Equations [57.90284928158383]
One of the main challenges in using deep learning-based methods for simulating physical systems is formulating physics-based data.
We propose a novel multi-level graph neural network framework that captures interaction at all ranges with only linear complexity.
Experiments confirm our multi-graph network learns discretization-invariant solution operators to PDEs and can be evaluated in linear time.
arXiv Detail & Related papers (2020-06-16T21:56:22Z) - Model Fusion via Optimal Transport [64.13185244219353]
We present a layer-wise model fusion algorithm for neural networks.
We show that this can successfully yield "one-shot" knowledge transfer between neural networks trained on heterogeneous non-i.i.d. data.
arXiv Detail & Related papers (2019-10-12T22:07:15Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.