MLDS: A Dataset for Weight-Space Analysis of Neural Networks

• URL: http://arxiv.org/abs/2104.10555v1
• Date: Wed, 21 Apr 2021 14:24:26 GMT
• Title: MLDS: A Dataset for Weight-Space Analysis of Neural Networks
• Authors: John Clemens
• Abstract summary: We present MLDS, a new dataset consisting of thousands of trained neural networks with carefully controlled parameters. This dataset enables new insights into both model-to-model and model-to-training-data relationships.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Neural networks are powerful models that solve a variety of complex real-world problems. However, the stochastic nature of training and large number of parameters in a typical neural model makes them difficult to evaluate via inspection. Research shows this opacity can hide latent undesirable behavior, be it from poorly representative training data or via malicious intent to subvert the behavior of the network, and that this behavior is difficult to detect via traditional indirect evaluation criteria such as loss. Therefore, it is time to explore direct ways to evaluate a trained neural model via its structure and weights. In this paper we present MLDS, a new dataset consisting of thousands of trained neural networks with carefully controlled parameters and generated via a global volunteer-based distributed computing platform. This dataset enables new insights into both model-to-model and model-to-training-data relationships. We use this dataset to show clustering of models in weight-space with identical training data and meaningful divergence in weight-space with even a small change to the training data, suggesting that weight-space analysis is a viable and effective alternative to loss for evaluating neural networks.

Related papers

• Transferable Post-training via Inverse Value Learning [83.75002867411263]
  We propose modeling changes at the logits level during post-training using a separate neural network (i.e., the value network) After training this network on a small base model using demonstrations, this network can be seamlessly integrated with other pre-trained models during inference. We demonstrate that the resulting value network has broad transferability across pre-trained models of different parameter sizes.
  arXiv  Detail & Related papers  (2024-10-28T13:48:43Z)
• Data Augmentations in Deep Weight Spaces [89.45272760013928]
  We introduce a novel augmentation scheme based on the Mixup method. We evaluate the performance of these techniques on existing benchmarks as well as new benchmarks we generate.
  arXiv  Detail & Related papers  (2023-11-15T10:43:13Z)
• Epistemic Modeling Uncertainty of Rapid Neural Network Ensembles for Adaptive Learning [0.0]
  A new type of neural network is presented using the rapid neural network paradigm. It is found that the proposed emulator embedded neural network trains near-instantaneously, typically without loss of prediction accuracy.
  arXiv  Detail & Related papers  (2023-09-12T22:34:34Z)
• Iterative self-transfer learning: A general methodology for response time-history prediction based on small dataset [0.0]
  An iterative self-transfer learningmethod for training neural networks based on small datasets is proposed in this study. The results show that the proposed method can improve the model performance by near an order of magnitude on small datasets.
  arXiv  Detail & Related papers  (2023-06-14T18:48:04Z)
• Towards Robust Dataset Learning [90.2590325441068]
  We propose a principled, tri-level optimization to formulate the robust dataset learning problem. Under an abstraction model that characterizes robust vs. non-robust features, the proposed method provably learns a robust dataset.
  arXiv  Detail & Related papers  (2022-11-19T17:06:10Z)
• Data-driven emergence of convolutional structure in neural networks [83.4920717252233]
  We show how fully-connected neural networks solving a discrimination task can learn a convolutional structure directly from their inputs. By carefully designing data models, we show that the emergence of this pattern is triggered by the non-Gaussian, higher-order local structure of the inputs.
  arXiv  Detail & Related papers  (2022-02-01T17:11:13Z)
• Gone Fishing: Neural Active Learning with Fisher Embeddings [55.08537975896764]
  There is an increasing need for active learning algorithms that are compatible with deep neural networks. This article introduces BAIT, a practical representation of tractable, and high-performing active learning algorithm for neural networks.
  arXiv  Detail & Related papers  (2021-06-17T17:26:31Z)
• Locally Sparse Networks for Interpretable Predictions [7.362415721170984]
  We propose a framework for training locally sparse neural networks where the local sparsity is learned via a sample-specific gating mechanism. The sample-specific sparsity is predicted via a textitgating network, which is trained in tandem with the textitprediction network. We demonstrate that our method outperforms state-of-the-art models when predicting the target function with far fewer features per instance.
  arXiv  Detail & Related papers  (2021-06-11T15:46:50Z)
• Estimating informativeness of samples with Smooth Unique Information [108.25192785062367]
  We measure how much a sample informs the final weights and how much it informs the function computed by the weights. We give efficient approximations of these quantities using a linearized network. We apply these measures to several problems, such as dataset summarization.
  arXiv  Detail & Related papers  (2021-01-17T10:29:29Z)
• An analytic theory of shallow networks dynamics for hinge loss classification [14.323962459195771]
  We study the training dynamics of a simple type of neural network: a single hidden layer trained to perform a classification task. We specialize our theory to the prototypical case of a linearly separable dataset and a linear hinge loss. This allow us to address in a simple setting several phenomena appearing in modern networks such as slowing down of training dynamics, crossover between rich and lazy learning, and overfitting.
  arXiv  Detail & Related papers  (2020-06-19T16:25:29Z)
• Learning Queuing Networks by Recurrent Neural Networks [0.0]
  We propose a machine-learning approach to derive performance models from data. We exploit a deterministic approximation of their average dynamics in terms of a compact system of ordinary differential equations. This allows for an interpretable structure of the neural network, which can be trained from system measurements to yield a white-box parameterized model.
  arXiv  Detail & Related papers  (2020-02-25T10:56:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.