Learning on Model Weights using Tree Experts

• URL: http://arxiv.org/abs/2410.13569v2
• Date: Thu, 05 Dec 2024 11:50:24 GMT
• Title: Learning on Model Weights using Tree Experts
• Authors: Eliahu Horwitz, Bar Cavia, Jonathan Kahana, Yedid Hoshen,
• Abstract summary: We show how to train neural networks that use other networks as input. ProbeX is the first probing method specifically designed to learn from the weights of a single model layer. We demonstrate the effectiveness of ProbeX by predicting the categories in a model's training dataset based only on its weights.
• Score: 39.90685550999956
• License:
• Abstract: The increasing availability of public models begs the question: can we train neural networks that use other networks as input? Such models allow us to study different aspects of a given neural network, for example, determining the categories in a model's training dataset. However, machine learning on model weights is challenging as they often exhibit significant variation unrelated to the models' semantic properties (nuisance variation). Here, we identify a key property of real-world models: most public models belong to a small set of Model Trees, where all models within a tree are fine-tuned from a common ancestor (e.g., a foundation model). Importantly, we find that within each tree there is less nuisance variation between models. Concretely, while learning across Model Trees requires complex architectures, even a linear classifier trained on a single model layer often works within trees. While effective, these linear classifiers are computationally expensive, especially when dealing with larger models that have many parameters. To address this, we introduce Probing Experts (ProbeX), a theoretically motivated and lightweight method. Notably, ProbeX is the first probing method specifically designed to learn from the weights of a single hidden model layer. We demonstrate the effectiveness of ProbeX by predicting the categories in a model's training dataset based only on its weights. Excitingly, ProbeX can also map the weights of Stable Diffusion into a shared weight-language embedding space, enabling zero-shot model classification.

Related papers

• Soup-of-Experts: Pretraining Specialist Models via Parameters Averaging [23.44999968321367]
  Soup-of-Experts can instantiate a model at test time for any domain weights with minimal computational cost and without re-training the model. We demonstrate how our approach obtains small specialized models on several language modeling tasks quickly.
  arXiv  Detail & Related papers  (2025-02-03T20:33:20Z)
• Exploring space efficiency in a tree-based linear model for extreme multi-label classification [11.18858602369985]
  Extreme multi-label classification (XMC) aims to identify relevant subsets from numerous labels. Among the various approaches for XMC, tree-based linear models are effective due to their superior efficiency and simplicity. In this work, we conduct both theoretical and empirical analyses on the space to store a tree model under the assumption of sparse data.
  arXiv  Detail & Related papers  (2024-10-12T15:02:40Z)
• On the Origin of Llamas: Model Tree Heritage Recovery [39.08927346274156]
  We introduce the task of Model Tree Heritage Recovery (MoTHer Recovery) for discovering Model Trees in neural networks. Our hypothesis is that model weights encode this information, the challenge is to decode the underlying tree structure given the weights. MoTHer recovery holds exciting long-term applications akin to indexing the internet by search engines.
  arXiv  Detail & Related papers  (2024-05-28T17:59:51Z)
• BEND: Bagging Deep Learning Training Based on Efficient Neural Network Diffusion [56.9358325168226]
  We propose a Bagging deep learning training algorithm based on Efficient Neural network Diffusion (BEND) Our approach is simple but effective, first using multiple trained model weights and biases as inputs to train autoencoder and latent diffusion model. Our proposed BEND algorithm can consistently outperform the mean and median accuracies of both the original trained model and the diffused model.
  arXiv  Detail & Related papers  (2024-03-23T08:40:38Z)
• A Dynamical Model of Neural Scaling Laws [79.59705237659547]
  We analyze a random feature model trained with gradient descent as a solvable model of network training and generalization. Our theory shows how the gap between training and test loss can gradually build up over time due to repeated reuse of data.
  arXiv  Detail & Related papers  (2024-02-02T01:41:38Z)
• Initializing Models with Larger Ones [76.41561758293055]
  We introduce weight selection, a method for initializing smaller models by selecting a subset of weights from a pretrained larger model. Our experiments demonstrate that weight selection can significantly enhance the performance of small models and reduce their training time.
  arXiv  Detail & Related papers  (2023-11-30T18:58:26Z)
• Dataless Knowledge Fusion by Merging Weights of Language Models [51.8162883997512]
  Fine-tuning pre-trained language models has become the prevalent paradigm for building downstream NLP models. This creates a barrier to fusing knowledge across individual models to yield a better single model. We propose a dataless knowledge fusion method that merges models in their parameter space.
  arXiv  Detail & Related papers  (2022-12-19T20:46:43Z)
• Part-Based Models Improve Adversarial Robustness [57.699029966800644]
  We show that combining human prior knowledge with end-to-end learning can improve the robustness of deep neural networks. Our model combines a part segmentation model with a tiny classifier and is trained end-to-end to simultaneously segment objects into parts. Our experiments indicate that these models also reduce texture bias and yield better robustness against common corruptions and spurious correlations.
  arXiv  Detail & Related papers  (2022-09-15T15:41:47Z)
• Revealing Secrets From Pre-trained Models [2.0249686991196123]
  Transfer-learning has been widely adopted in many emerging deep learning algorithms. We show that pre-trained models and fine-tuned models have significantly high similarities in weight values. We propose a new model extraction attack that reveals the model architecture and the pre-trained model used by the black-box victim model.
  arXiv  Detail & Related papers  (2022-07-19T20:19:03Z)
• Neural Basis Models for Interpretability [33.51591891812176]
  Generalized Additive Models (GAMs) are an inherently interpretable class of models. We propose an entirely new subfamily of GAMs that utilize basis decomposition of shape functions. A small number of basis functions are shared among all features, and are learned jointly for a given task.
  arXiv  Detail & Related papers  (2022-05-27T17:31:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.