Clusterability in Neural Networks

• URL: http://arxiv.org/abs/2103.03386v1
• Date: Thu, 4 Mar 2021 23:53:53 GMT
• Title: Clusterability in Neural Networks
• Authors: Daniel Filan, Stephen Casper, Shlomi Hod, Cody Wild, Andrew Critch, Stuart Russell
• Abstract summary: We find that a trained neural network is typically more clusterable than randomly networks, and often clusterable relative to random networks with the same distribution of weights. We also exhibit novel methods to promote clusterability in neural network training, and find that in multi-layer perceptrons they lead to more clusterable networks with little reduction in accuracy.
• Score: 9.190168301432811
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The learned weights of a neural network have often been considered devoid of scrutable internal structure. In this paper, however, we look for structure in the form of clusterability: how well a network can be divided into groups of neurons with strong internal connectivity but weak external connectivity. We find that a trained neural network is typically more clusterable than randomly initialized networks, and often clusterable relative to random networks with the same distribution of weights. We also exhibit novel methods to promote clusterability in neural network training, and find that in multi-layer perceptrons they lead to more clusterable networks with little reduction in accuracy. Understanding and controlling the clusterability of neural networks will hopefully render their inner workings more interpretable to engineers by facilitating partitioning into meaningful clusters.

Related papers

• Semantic representations emerge in biologically inspired ensembles of cross-supervising neural networks [1.5346678870160888]
  We present a model of representation learning by ensembles of neural networks.<n>Each network learns to encode stimuli into an abstract representation space by cross-supervising interactions with other networks.<n>We find that performance is optimal for small receptive fields, and that sparse connectivity between networks is nearly as accurate as all-to-all interactions.
  arXiv  Detail & Related papers  (2025-10-16T09:30:22Z)
• Coding schemes in neural networks learning classification tasks [52.22978725954347]
  We investigate fully-connected, wide neural networks learning classification tasks. We show that the networks acquire strong, data-dependent features. Surprisingly, the nature of the internal representations depends crucially on the neuronal nonlinearity.
  arXiv  Detail & Related papers  (2024-06-24T14:50:05Z)
• Stable Learning Using Spiking Neural Networks Equipped With Affine Encoders and Decoders [2.0072624123275533]
  We study the learning problem associated with spiking neural networks. We focus on spiking neural networks composed of simple spiking neurons having only positive synaptic weights. In particular, we show in theory and simulations that affine spiking neural networks are capable of approximating shallow ReLU neural networks.
  arXiv  Detail & Related papers  (2024-04-06T08:17:07Z)
• Taming Binarized Neural Networks and Mixed-Integer Programs [2.7624021966289596]
  We show that binarized neural networks admit a tame representation. This makes it possible to use the framework of Bolte et al. for implicit differentiation. This approach could also be used for a broader class of mixed-integer programs.
  arXiv  Detail & Related papers  (2023-10-05T21:04:16Z)
• How neural networks learn to classify chaotic time series [77.34726150561087]
  We study the inner workings of neural networks trained to classify regular-versus-chaotic time series. We find that the relation between input periodicity and activation periodicity is key for the performance of LKCNN models.
  arXiv  Detail & Related papers  (2023-06-04T08:53:27Z)
• Connected Hidden Neurons (CHNNet): An Artificial Neural Network for Rapid Convergence [0.6218519716921521]
  We propose a more robust model of artificial neural networks where the hidden neurons, residing in the same hidden layer, are interconnected that leads to rapid convergence. With the experimental study of our proposed model in deep networks, we demonstrate that the model results in a noticeable increase in convergence rate compared to the conventional feed-forward neural network.
  arXiv  Detail & Related papers  (2023-05-17T14:00:38Z)
• Rank Diminishing in Deep Neural Networks [71.03777954670323]
  Rank of neural networks measures information flowing across layers. It is an instance of a key structural condition that applies across broad domains of machine learning. For neural networks, however, the intrinsic mechanism that yields low-rank structures remains vague and unclear.
  arXiv  Detail & Related papers  (2022-06-13T12:03:32Z)
• Building Compact and Robust Deep Neural Networks with Toeplitz Matrices [93.05076144491146]
  This thesis focuses on the problem of training neural networks which are compact, easy to train, reliable and robust to adversarial examples. We leverage the properties of structured matrices from the Toeplitz family to build compact and secure neural networks.
  arXiv  Detail & Related papers  (2021-09-02T13:58:12Z)
• Provably Training Neural Network Classifiers under Fairness Constraints [70.64045590577318]
  We show that overparametrized neural networks could meet the constraints. Key ingredient of building a fair neural network classifier is establishing no-regret analysis for neural networks.
  arXiv  Detail & Related papers  (2020-12-30T18:46:50Z)
• Finding trainable sparse networks through Neural Tangent Transfer [16.092248433189816]
  In deep learning, trainable sparse networks that perform well on a specific task are usually constructed using label-dependent pruning criteria. In this article, we introduce Neural Tangent Transfer, a method that instead finds trainable sparse networks in a label-free manner.
  arXiv  Detail & Related papers  (2020-06-15T08:58:01Z)
• On Hiding Neural Networks Inside Neural Networks [30.20287788363144]
  We show that this excess capacity provides an opportunity for embedding secret machine learning models within a trained neural network. Our novel framework hides the existence of a secret neural network with arbitrary desired functionality within a carrier network.
  arXiv  Detail & Related papers  (2020-02-24T05:18:29Z)
• ResiliNet: Failure-Resilient Inference in Distributed Neural Networks [56.255913459850674]
  We introduce ResiliNet, a scheme for making inference in distributed neural networks resilient to physical node failures. Failout simulates physical node failure conditions during training using dropout, and is specifically designed to improve the resiliency of distributed neural networks.
  arXiv  Detail & Related papers  (2020-02-18T05:58:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.