Unveiling Invariances via Neural Network Pruning

• URL: http://arxiv.org/abs/2309.08171v1
• Date: Fri, 15 Sep 2023 05:38:33 GMT
• Title: Unveiling Invariances via Neural Network Pruning
• Authors: Derek Xu, Yizhou Sun, Wei Wang
• Abstract summary: Invariance describes transformations that do not alter data's underlying semantics. Modern networks are handcrafted to handle well-known invariances. We propose a framework to learn novel network architectures that capture data-dependent invariances via pruning.
• Score: 44.47186380630998
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Invariance describes transformations that do not alter data's underlying semantics. Neural networks that preserve natural invariance capture good inductive biases and achieve superior performance. Hence, modern networks are handcrafted to handle well-known invariances (ex. translations). We propose a framework to learn novel network architectures that capture data-dependent invariances via pruning. Our learned architectures consistently outperform dense neural networks on both vision and tabular datasets in both efficiency and effectiveness. We demonstrate our framework on multiple deep learning models across 3 vision and 40 tabular datasets.

Related papers

• Graph Neural Networks for Learning Equivariant Representations of Neural Networks [55.04145324152541]
  We propose to represent neural networks as computational graphs of parameters. Our approach enables a single model to encode neural computational graphs with diverse architectures. We showcase the effectiveness of our method on a wide range of tasks, including classification and editing of implicit neural representations.
  arXiv  Detail & Related papers  (2024-03-18T18:01:01Z)
• Revisiting Data Augmentation for Rotational Invariance in Convolutional Neural Networks [0.29127054707887967]
  We investigate how best to include rotational invariance in a CNN for image classification. Our experiments show that networks trained with data augmentation alone can classify rotated images nearly as well as in the normal unrotated case.
  arXiv  Detail & Related papers  (2023-10-12T15:53:24Z)
• Interpretable Mesomorphic Networks for Tabular Data [25.76214343259399]
  We propose a new class of interpretable neural networks that are both deep and linear at the same time. We optimize deep hypernetworks to generate explainable linear models on a per-instance basis.
  arXiv  Detail & Related papers  (2023-05-22T14:41:17Z)
• Exploring explicit coarse-grained structure in artificial neural networks [0.0]
  We propose to employ the hierarchical coarse-grained structure in the artificial neural networks explicitly to improve the interpretability without degrading performance. One is a neural network called TaylorNet, which aims to approximate the general mapping from input data to output result in terms of Taylor series directly. The other is a new setup for data distillation, which can perform multi-level abstraction of the input dataset and generate new data.
  arXiv  Detail & Related papers  (2022-11-03T13:06:37Z)
• Transfer Learning with Deep Tabular Models [66.67017691983182]
  We show that upstream data gives tabular neural networks a decisive advantage over GBDT models. We propose a realistic medical diagnosis benchmark for tabular transfer learning. We propose a pseudo-feature method for cases where the upstream and downstream feature sets differ.
  arXiv  Detail & Related papers  (2022-06-30T14:24:32Z)
• Deep invariant networks with differentiable augmentation layers [87.22033101185201]
  Methods for learning data augmentation policies require held-out data and are based on bilevel optimization problems. We show that our approach is easier and faster to train than modern automatic data augmentation techniques.
  arXiv  Detail & Related papers  (2022-02-04T14:12:31Z)
• Dynamic Inference with Neural Interpreters [72.90231306252007]
  We present Neural Interpreters, an architecture that factorizes inference in a self-attention network as a system of modules. inputs to the model are routed through a sequence of functions in a way that is end-to-end learned. We show that Neural Interpreters perform on par with the vision transformer using fewer parameters, while being transferrable to a new task in a sample efficient manner.
  arXiv  Detail & Related papers  (2021-10-12T23:22:45Z)
• A Bayesian Approach to Invariant Deep Neural Networks [14.807284992678762]
  We show that our model outperforms other non-invariant architectures, when trained on datasets that contain specific invariances. The same holds true when no data augmentation is performed.
  arXiv  Detail & Related papers  (2021-07-20T07:33:58Z)
• Diversity inducing Information Bottleneck in Model Ensembles [73.80615604822435]
  In this paper, we target the problem of generating effective ensembles of neural networks by encouraging diversity in prediction. We explicitly optimize a diversity inducing adversarial loss for learning latent variables and thereby obtain diversity in the output predictions necessary for modeling multi-modal data. Compared to the most competitive baselines, we show significant improvements in classification accuracy, under a shift in the data distribution.
  arXiv  Detail & Related papers  (2020-03-10T03:10:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.