Sparsifying networks by traversing Geodesics

• URL: http://arxiv.org/abs/2012.09605v1
• Date: Sat, 12 Dec 2020 21:39:19 GMT
• Title: Sparsifying networks by traversing Geodesics
• Authors: Guruprasad Raghavan, Matt Thomson
• Abstract summary: In this paper, we attempt to solve certain open questions in ML, by viewing them through the lens of geometry. We propose a mathematical framework to evaluate geodesics in the functional space, to find high-performance paths from a dense network to its sparser counterpart.
• Score: 6.09170287691728
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The geometry of weight spaces and functional manifolds of neural networks play an important role towards 'understanding' the intricacies of ML. In this paper, we attempt to solve certain open questions in ML, by viewing them through the lens of geometry, ultimately relating it to the discovery of points or paths of equivalent function in these spaces. We propose a mathematical framework to evaluate geodesics in the functional space, to find high-performance paths from a dense network to its sparser counterpart. Our results are obtained on VGG-11 trained on CIFAR-10 and MLP's trained on MNIST. Broadly, we demonstrate that the framework is general, and can be applied to a wide variety of problems, ranging from sparsification to alleviating catastrophic forgetting.

Related papers

• Landscaping Linear Mode Connectivity [76.39694196535996]
  linear mode connectivity (LMC) has garnered interest from both theoretical and practical fronts. We take a step towards understanding it by providing a model of how the loss landscape needs to behave topographically for LMC.
  arXiv  Detail & Related papers  (2024-06-24T03:53:30Z)
• Asymptotics of Learning with Deep Structured (Random) Features [9.366617422860543]
  For a large class of feature maps we provide a tight characterisation of the test error associated with learning the readout layer. In some cases our results can capture feature maps learned by deep, finite-width neural networks trained under gradient descent.
  arXiv  Detail & Related papers  (2024-02-21T18:35:27Z)
• Riemannian Residual Neural Networks [58.925132597945634]
  We show how to extend the residual neural network (ResNet) ResNets have become ubiquitous in machine learning due to their beneficial learning properties, excellent empirical results, and easy-to-incorporate nature when building varied neural networks.
  arXiv  Detail & Related papers  (2023-10-16T02:12:32Z)
• Equivariant Architectures for Learning in Deep Weight Spaces [54.61765488960555]
  We present a novel network architecture for learning in deep weight spaces. It takes as input a concatenation of weights and biases of a pre-trainedvariant. We show how these layers can be implemented using three basic operations.
  arXiv  Detail & Related papers  (2023-01-30T10:50:33Z)
• Sheaf Neural Networks with Connection Laplacians [3.3414557160889076]
  Sheaf Neural Network (SNN) is a type of Graph Neural Network (GNN) that operates on a sheaf, an object that equips a graph with vector spaces over its nodes and edges and linear maps between these spaces. Previous works proposed two diametrically opposed approaches: manually constructing the sheaf based on domain knowledge and learning the sheaf end-to-end using gradient-based methods. In this work, we propose a novel way of computing sheaves drawing inspiration from Riemannian geometry. We show that this approach achieves promising results with less computational overhead when compared to previous SNN models.
  arXiv  Detail & Related papers  (2022-06-17T11:39:52Z)
• FuNNscope: Visual microscope for interactively exploring the loss landscape of fully connected neural networks [77.34726150561087]
  We show how to explore high-dimensional landscape characteristics of neural networks. We generalize observations on small neural networks to more complex systems. An interactive dashboard opens up a number of possible application networks.
  arXiv  Detail & Related papers  (2022-04-09T16:41:53Z)
• Geometry-Aware Hierarchical Bayesian Learning on Manifolds [5.182379239800725]
  We propose a hierarchical Bayesian learning model for learning on manifold-valued vision data. We first introduce a kernel with the properties of geometry-awareness and intra- Kernel convolution. We then use a Gaussian process regression to organize the inputs and finally implement a hierarchical Bayesian network for the feature aggregation.
  arXiv  Detail & Related papers  (2021-10-30T05:47:05Z)
• Localized Persistent Homologies for more Effective Deep Learning [60.78456721890412]
  We introduce an approach that relies on a new filtration function to account for location during network training. We demonstrate experimentally on 2D images of roads and 3D image stacks of neuronal processes that networks trained in this manner are better at recovering the topology of the curvilinear structures they extract.
  arXiv  Detail & Related papers  (2021-10-12T19:28:39Z)
• Solving hybrid machine learning tasks by traversing weight space geodesics [6.09170287691728]
  Machine learning problems have an intrinsic geometric structure as central objects including a neural network's weight space. We introduce a geometric framework that unifies a range machine learning objectives and can be applied to multiple classes neural network architectures.
  arXiv  Detail & Related papers  (2021-06-05T04:37:03Z)
• A neural anisotropic view of underspecification in deep learning [60.119023683371736]
  We show that the way neural networks handle the underspecification of problems is highly dependent on the data representation. Our results highlight that understanding the architectural inductive bias in deep learning is fundamental to address the fairness, robustness, and generalization of these systems.
  arXiv  Detail & Related papers  (2021-04-29T14:31:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.