ZooD: Exploiting Model Zoo for Out-of-Distribution Generalization

• URL: http://arxiv.org/abs/2210.09236v1
• Date: Mon, 17 Oct 2022 16:31:57 GMT
• Title: ZooD: Exploiting Model Zoo for Out-of-Distribution Generalization
• Authors: Qishi Dong, Awais Muhammad, Fengwei Zhou, Chuanlong Xie, Tianyang Hu, Yongxin Yang, Sung-Ho Bae, Zhenguo Li
• Abstract summary: We propose ZooD, a paradigm for PTMs ranking and ensemble with feature selection. We evaluate our paradigm on a diverse model zoo consisting of 35 models for various Out-of-Distribution (OoD) tasks.
• Score: 65.58562481279023
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent advances on large-scale pre-training have shown great potentials of leveraging a large set of Pre-Trained Models (PTMs) for improving Out-of-Distribution (OoD) generalization, for which the goal is to perform well on possible unseen domains after fine-tuning on multiple training domains. However, maximally exploiting a zoo of PTMs is challenging since fine-tuning all possible combinations of PTMs is computationally prohibitive while accurate selection of PTMs requires tackling the possible data distribution shift for OoD tasks. In this work, we propose ZooD, a paradigm for PTMs ranking and ensemble with feature selection. Our proposed metric ranks PTMs by quantifying inter-class discriminability and inter-domain stability of the features extracted by the PTMs in a leave-one-domain-out cross-validation manner. The top-K ranked models are then aggregated for the target OoD task. To avoid accumulating noise induced by model ensemble, we propose an efficient variational EM algorithm to select informative features. We evaluate our paradigm on a diverse model zoo consisting of 35 models for various OoD tasks and demonstrate: (i) model ranking is better correlated with fine-tuning ranking than previous methods and up to 9859x faster than brute-force fine-tuning; (ii) OoD generalization after model ensemble with feature selection outperforms the state-of-the-art methods and the accuracy on most challenging task DomainNet is improved from 46.5\% to 50.6\%. Furthermore, we provide the fine-tuning results of 35 PTMs on 7 OoD datasets, hoping to help the research of model zoo and OoD generalization. Code will be available at https://gitee.com/mindspore/models/tree/master/research/cv/zood.

Related papers

• Steering Masked Discrete Diffusion Models via Discrete Denoising Posterior Prediction [88.65168366064061]
  We introduce Discrete Denoising Posterior Prediction (DDPP), a novel framework that casts the task of steering pre-trained MDMs as a problem of probabilistic inference. Our framework leads to a family of three novel objectives that are all simulation-free, and thus scalable. We substantiate our designs via wet-lab validation, where we observe transient expression of reward-optimized protein sequences.
  arXiv  Detail & Related papers  (2024-10-10T17:18:30Z)
• FissionFusion: Fast Geometric Generation and Hierarchical Souping for Medical Image Analysis [0.7751705157998379]
  The scarcity of well-annotated medical datasets requires leveraging transfer learning from broader datasets like ImageNet or pre-trained models like CLIP. Model soups averages multiple fine-tuned models aiming to improve performance on In-Domain (ID) tasks and enhance robustness against Out-of-Distribution (OOD) datasets. We propose a hierarchical merging approach that involves local and global aggregation of models at various levels.
  arXiv  Detail & Related papers  (2024-03-20T06:48:48Z)
• Pruning Large Language Models via Accuracy Predictor [0.0]
  Large language models (LLMs) containing tens of billions of parameters (or even more) have demonstrated impressive capabilities in various NLP tasks. We propose a novel pruning approach: firstly, a training set of a certain number of architecture-accuracy pairs is established, and then a non-neural model is trained as an accuracy predictor.
  arXiv  Detail & Related papers  (2023-09-18T06:38:24Z)
• Model Spider: Learning to Rank Pre-Trained Models Efficiently [42.56392378060269]
  Model Spider learns to construct tokens and measure the fitness score between a model-task pair via their tokens. Model Spider balances efficiency and selection ability, making PTM selection like a spider preying on a web.
  arXiv  Detail & Related papers  (2023-06-06T17:58:12Z)
• Cross-Modal Fine-Tuning: Align then Refine [83.37294254884446]
  ORCA is a cross-modal fine-tuning framework that extends the applicability of a single large-scale pretrained model to diverse modalities. We show that ORCA obtains state-of-the-art results on 3 benchmarks containing over 60 datasets from 12 modalities.
  arXiv  Detail & Related papers  (2023-02-11T16:32:28Z)
• An Automated Question-Answering Framework Based on Evolution Algorithm [19.054115603616513]
  We propose an automated Question-Answering framework, which could adjust network architecture for multiple datasets. Our framework achieves 78.9 EM and 86.1 F1 on SQuAD 1.1, 69.9 EM and 72.5 F1 on SQuAD 2.0.
  arXiv  Detail & Related papers  (2022-01-26T08:13:24Z)
• Training Experimentally Robust and Interpretable Binarized Regression Models Using Mixed-Integer Programming [3.179831861897336]
  We present a model-based approach to training robust and interpretable binarized regression models for multiclass classification tasks. Our MIP model balances the optimization of prediction margin and model size by using a weighted objective. We show the effectiveness of training robust and interpretable binarized regression models using MIP.
  arXiv  Detail & Related papers  (2021-12-01T11:53:08Z)
• Ensemble of Averages: Improving Model Selection and Boosting Performance in Domain Generalization [63.28279815753543]
  In Domain Generalization (DG) settings, models trained on a given set of training domains have notoriously chaotic performance on shifted test domains. We first show that a simple protocol for averaging model parameters along the optimization path, starting early during training, significantly boosts domain generalizationity. We show that an ensemble of independently trained models also has a chaotic behavior in the DG setting.
  arXiv  Detail & Related papers  (2021-10-21T00:08:17Z)
• Ranking and Tuning Pre-trained Models: A New Paradigm of Exploiting Model Hubs [136.4492678691406]
  We propose a new paradigm of exploiting model hubs by ranking and tuning pre-trained models. The best ranked PTM can be fine-tuned and deployed if we have no preference for the model's architecture. The tuning part introduces a novel method for multiple PTMs tuning, which surpasses dedicated methods.
  arXiv  Detail & Related papers  (2021-10-20T12:59:23Z)
• AvgOut: A Simple Output-Probability Measure to Eliminate Dull Responses [97.50616524350123]
  We build dialogue models that are dynamically aware of what utterances or tokens are dull without any feature-engineering. The first model, MinAvgOut, directly maximizes the diversity score through the output distributions of each batch. The second model, Label Fine-Tuning (LFT), prepends to the source sequence a label continuously scaled by the diversity score to control the diversity level. The third model, RL, adopts Reinforcement Learning and treats the diversity score as a reward signal.
  arXiv  Detail & Related papers  (2020-01-15T18:32:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.