Vertical Federated Learning Hybrid Local Pre-training

• URL: http://arxiv.org/abs/2405.11884v2
• Date: Tue, 21 May 2024 07:46:03 GMT
• Title: Vertical Federated Learning Hybrid Local Pre-training
• Authors: Wenguo Li, Xinling Guo, Xu Jiao, Tiancheng Huang, Xiaoran Yan, Yao Yang,
• Abstract summary: We propose a novel VFL Hybrid Local Pre-training (VFLHLP) approach for Vertical Federated Learning (VFL) VFLHLP first pre-trains local networks on the local data of participating parties. Then it utilizes these pre-trained networks to adjust the sub-model for the labeled party or enhance representation learning for other parties during downstream federated learning on aligned data.
• Score: 4.31644387824845
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vertical Federated Learning (VFL), which has a broad range of real-world applications, has received much attention in both academia and industry. Enterprises aspire to exploit more valuable features of the same users from diverse departments to boost their model prediction skills. VFL addresses this demand and concurrently secures individual parties from exposing their raw data. However, conventional VFL encounters a bottleneck as it only leverages aligned samples, whose size shrinks with more parties involved, resulting in data scarcity and the waste of unaligned data. To address this problem, we propose a novel VFL Hybrid Local Pre-training (VFLHLP) approach. VFLHLP first pre-trains local networks on the local data of participating parties. Then it utilizes these pre-trained networks to adjust the sub-model for the labeled party or enhance representation learning for other parties during downstream federated learning on aligned data, boosting the performance of federated models. The experimental results on real-world advertising datasets, demonstrate that our approach achieves the best performance over baseline methods by large margins. The ablation study further illustrates the contribution of each technique in VFLHLP to its overall performance.

Related papers

• FedAli: Personalized Federated Learning with Aligned Prototypes through Optimal Transport [9.683642138601464]
  Federated Learning (FL) enables collaborative, personalized model training across multiple devices without sharing raw data. We introduce the Alignment with Prototypes layers, which align incoming embeddings closer to learnable prototypes. We evaluate FedAli on heterogeneous sensor-based human activity recognition and vision benchmark datasets, demonstrating that it outperforms existing FL strategies.
  arXiv  Detail & Related papers  (2024-11-15T21:35:21Z)
• De-VertiFL: A Solution for Decentralized Vertical Federated Learning [7.877130417748362]
  This work introduces De-VertiFL, a novel solution for training models in a decentralized VFL setting. De-VertiFL contributes by introducing a new network architecture distribution, an innovative knowledge exchange scheme, and a distributed federated training process. The results demonstrate that De-VertiFL generally surpasses state-of-the-art methods in F1-score performance, while maintaining a decentralized and privacy-preserving framework.
  arXiv  Detail & Related papers  (2024-10-08T15:31:10Z)
• Distributionally Robust Alignment for Medical Federated Vision-Language Pre-training Under Data Heterogeneity [4.84693589377679]
  We propose Federated Distributionally Robust Alignment (FedDRA) for medical vision-language pre-training. FedDRA achieves robust vision-language alignment under heterogeneous conditions. Our method also adapts well to various medical pre-training methods.
  arXiv  Detail & Related papers  (2024-04-05T01:17:25Z)
• Contrastive encoder pre-training-based clustered federated learning for heterogeneous data [17.580390632874046]
  Federated learning (FL) enables distributed clients to collaboratively train a global model while preserving their data privacy. We propose contrastive pre-training-based clustered federated learning (CP-CFL) to improve the model convergence and overall performance of FL systems.
  arXiv  Detail & Related papers  (2023-11-28T05:44:26Z)
• VFedMH: Vertical Federated Learning for Training Multiple Heterogeneous Models [53.30484242706966]
  This paper proposes a novel approach called Vertical federated learning for training multiple Heterogeneous models (VFedMH) To protect the participants' local embedding values, we propose an embedding protection method based on lightweight blinding factors. Experiments are conducted to demonstrate that VFedMH can simultaneously train multiple heterogeneous models with heterogeneous optimization and outperform some recent methods in model performance.
  arXiv  Detail & Related papers  (2023-10-20T09:22:51Z)
• Tackling Computational Heterogeneity in FL: A Few Theoretical Insights [68.8204255655161]
  We introduce and analyse a novel aggregation framework that allows for formalizing and tackling computational heterogeneous data. Proposed aggregation algorithms are extensively analyzed from a theoretical, and an experimental prospective.
  arXiv  Detail & Related papers  (2023-07-12T16:28:21Z)
• DPP-based Client Selection for Federated Learning with Non-IID Data [97.1195165400568]
  This paper proposes a client selection (CS) method to tackle the communication bottleneck of federated learning (FL) We first analyze the effect of CS in FL and show that FL training can be accelerated by adequately choosing participants to diversify the training dataset in each round of training. We leverage data profiling and determinantal point process (DPP) sampling techniques to develop an algorithm termed Federated Learning with DPP-based Participant Selection (FL-DP$3$S)
  arXiv  Detail & Related papers  (2023-03-30T13:14:54Z)
• Integrating Local Real Data with Global Gradient Prototypes for Classifier Re-Balancing in Federated Long-Tailed Learning [60.41501515192088]
  Federated Learning (FL) has become a popular distributed learning paradigm that involves multiple clients training a global model collaboratively. The data samples usually follow a long-tailed distribution in the real world, and FL on the decentralized and long-tailed data yields a poorly-behaved global model. In this work, we integrate the local real data with the global gradient prototypes to form the local balanced datasets.
  arXiv  Detail & Related papers  (2023-01-25T03:18:10Z)
• Vertical Semi-Federated Learning for Efficient Online Advertising [50.18284051956359]
  Semi-VFL (Vertical Semi-Federated Learning) is proposed to achieve a practical industry application fashion for VFL. We build an inference-efficient single-party student model applicable to the whole sample space. New representation distillation methods are designed to extract cross-party feature correlations for both the overlapped and non-overlapped data.
  arXiv  Detail & Related papers  (2022-09-30T17:59:27Z)
• FedDM: Iterative Distribution Matching for Communication-Efficient Federated Learning [87.08902493524556]
  Federated learning(FL) has recently attracted increasing attention from academia and industry. We propose FedDM to build the global training objective from multiple local surrogate functions. In detail, we construct synthetic sets of data on each client to locally match the loss landscape from original data.
  arXiv  Detail & Related papers  (2022-07-20T04:55:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.