Improving Heterogeneous Model Reuse by Density Estimation
- URL: http://arxiv.org/abs/2305.13871v1
- Date: Tue, 23 May 2023 09:46:54 GMT
- Title: Improving Heterogeneous Model Reuse by Density Estimation
- Authors: Anke Tang, Yong Luo, Han Hu, Fengxiang He, Kehua Su, Bo Du, Yixin
Chen, Dacheng Tao
- Abstract summary: This paper studies multiparty learning, aiming to learn a model using the private data of different participants.
Model reuse is a promising solution for multiparty learning, assuming that a local model has been trained for each party.
- Score: 105.97036205113258
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: This paper studies multiparty learning, aiming to learn a model using the
private data of different participants. Model reuse is a promising solution for
multiparty learning, assuming that a local model has been trained for each
party. Considering the potential sample selection bias among different parties,
some heterogeneous model reuse approaches have been developed. However,
although pre-trained local classifiers are utilized in these approaches, the
characteristics of the local data are not well exploited. This motivates us to
estimate the density of local data and design an auxiliary model together with
the local classifiers for reuse. To address the scenarios where some local
models are not well pre-trained, we further design a multiparty cross-entropy
loss for calibration. Upon existing works, we address a challenging problem of
heterogeneous model reuse from a decision theory perspective and take advantage
of recent advances in density estimation. Experimental results on both
synthetic and benchmark data demonstrate the superiority of the proposed
method.
Related papers
- Model Assessment and Selection under Temporal Distribution Shift [1.024113475677323]
We develop an adaptive rolling window approach to estimate the generalization error of a given model.
We also integrate pairwise comparisons into a single-elimination tournament, achieving near-optimal model selection from a collection of candidates.
arXiv Detail & Related papers (2024-02-13T18:54:08Z) - Spectral Co-Distillation for Personalized Federated Learning [69.97016362754319]
We propose a novel distillation method based on model spectrum information to better capture generic versus personalized representations.
We also introduce a co-distillation framework that establishes a two-way bridge between generic and personalized model training.
We demonstrate the outperformance and efficacy of our proposed spectral co-distillation method, as well as our wait-free training protocol.
arXiv Detail & Related papers (2024-01-29T16:01:38Z) - Federated Skewed Label Learning with Logits Fusion [23.062650578266837]
Federated learning (FL) aims to collaboratively train a shared model across multiple clients without transmitting their local data.
We propose FedBalance, which corrects the optimization bias among local models by calibrating their logits.
Our method can gain 13% higher average accuracy compared with state-of-the-art methods.
arXiv Detail & Related papers (2023-11-14T14:37:33Z) - Pre-trained Recommender Systems: A Causal Debiasing Perspective [19.712997823535066]
We develop a generic recommender that captures universal interaction patterns by training on generic user-item interaction data extracted from different domains.
Our empirical studies show that the proposed model could significantly improve the recommendation performance in zero- and few-shot learning settings.
arXiv Detail & Related papers (2023-10-30T03:37:32Z) - Dual Student Networks for Data-Free Model Stealing [79.67498803845059]
Two main challenges are estimating gradients of the target model without access to its parameters, and generating a diverse set of training samples.
We propose a Dual Student method where two students are symmetrically trained in order to provide the generator a criterion to generate samples that the two students disagree on.
We show that our new optimization framework provides more accurate gradient estimation of the target model and better accuracies on benchmark classification datasets.
arXiv Detail & Related papers (2023-09-18T18:11:31Z) - Dataless Knowledge Fusion by Merging Weights of Language Models [51.8162883997512]
Fine-tuning pre-trained language models has become the prevalent paradigm for building downstream NLP models.
This creates a barrier to fusing knowledge across individual models to yield a better single model.
We propose a dataless knowledge fusion method that merges models in their parameter space.
arXiv Detail & Related papers (2022-12-19T20:46:43Z) - Investigating Ensemble Methods for Model Robustness Improvement of Text
Classifiers [66.36045164286854]
We analyze a set of existing bias features and demonstrate there is no single model that works best for all the cases.
By choosing an appropriate bias model, we can obtain a better robustness result than baselines with a more sophisticated model design.
arXiv Detail & Related papers (2022-10-28T17:52:10Z) - Optimal Model Averaging: Towards Personalized Collaborative Learning [0.0]
In federated learning, differences in the data or objectives between the participating nodes motivate approaches to train a personalized machine learning model for each node.
One such approach is weighted averaging between a locally trained model and the global model.
We find that there is always some positive amount of model averaging that reduces the expected squared error compared to the local model.
arXiv Detail & Related papers (2021-10-25T13:33:20Z) - LOGAN: Local Group Bias Detection by Clustering [86.38331353310114]
We argue that evaluating bias at the corpus level is not enough for understanding how biases are embedded in a model.
We propose LOGAN, a new bias detection technique based on clustering.
Experiments on toxicity classification and object classification tasks show that LOGAN identifies bias in a local region.
arXiv Detail & Related papers (2020-10-06T16:42:51Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.