Adapter-based Selective Knowledge Distillation for Federated Multi-domain Meeting Summarization

• URL: http://arxiv.org/abs/2308.03275v1
• Date: Mon, 7 Aug 2023 03:34:01 GMT
• Title: Adapter-based Selective Knowledge Distillation for Federated Multi-domain Meeting Summarization
• Authors: Xiachong Feng, Xiaocheng Feng, Xiyuan Du, Min-Yen Kan, Bing Qin
• Abstract summary: Meeting summarization has emerged as a promising technique for providing users with condensed summaries. We propose adapter-based Federated Selective Knowledge Distillation (AdaFedSelecKD) for training performant client models.
• Score: 36.916155654985936
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Meeting summarization has emerged as a promising technique for providing users with condensed summaries. However, existing work has focused on training models on centralized data, neglecting real-world scenarios where meeting data are infeasible to collect centrally, due to their sensitive nature. This gap motivates us to explore federated learning for meeting summarization. Two critical challenges impede progress. First, state-of-the-art summarizers are based on parameter-heavy pre-trained models. Exchanging such a model's parameters across clients imposes large bandwidth costs. Second, as real-world meeting data belong to various domains and are distributed across clients, they are instances of non-identically and independently distributed (non-IID). IID assumptions do not hold, which changes which forms of learning algorithms best apply. To address this, we propose Adapter-based Federated Selective Knowledge Distillation (AdaFedSelecKD) for training performant client models. Specifically, we develop an adapter-based summarization model where two adapters cooperatively facilitate learning using fewer parameters to reduce communication costs. Then, we devise a selective knowledge distillation strategy, assisting clients in robustly handling domain-focused modelling on their own data, while leveraging global parameters based on non-IID data. Extensive experiments on the QMSum benchmark demonstrate AdaFedSelecKD can achieve comparable performance with powerful centralized training methods, and shows its generalizability and robustness.

Related papers

• FedReMa: Improving Personalized Federated Learning via Leveraging the Most Relevant Clients [13.98392319567057]
  Federated Learning (FL) is a distributed machine learning paradigm that achieves a globally robust model through decentralized computation and periodic model synthesis. Despite their wide adoption, existing FL and PFL works have yet to comprehensively address the class-imbalance issue. We propose FedReMa, an efficient PFL algorithm that can tackle class-imbalance by utilizing an adaptive inter-client co-learning approach.
  arXiv  Detail & Related papers  (2024-11-04T05:44:28Z)
• ConDa: Fast Federated Unlearning with Contribution Dampening [46.074452659791575]
  ConDa is a framework that performs efficient unlearning by tracking down the parameters which affect the global model for each client. We perform experiments on multiple datasets and demonstrate that ConDa is effective to forget a client's data.
  arXiv  Detail & Related papers  (2024-10-05T12:45:35Z)
• Federated Learning with Projected Trajectory Regularization [65.6266768678291]
  Federated learning enables joint training of machine learning models from distributed clients without sharing their local data. One key challenge in federated learning is to handle non-identically distributed data across the clients. We propose a novel federated learning framework with projected trajectory regularization (FedPTR) for tackling the data issue.
  arXiv  Detail & Related papers  (2023-12-22T02:12:08Z)
• One-Shot Federated Learning with Classifier-Guided Diffusion Models [44.604485649167216]
  One-shot federated learning (OSFL) has gained attention in recent years due to its low communication cost. In this paper, we explore the novel opportunities that diffusion models bring to OSFL and propose FedCADO. FedCADO generates data that complies with clients' distributions and subsequently training the aggregated model on the server.
  arXiv  Detail & Related papers  (2023-11-15T11:11:25Z)
• Elastically-Constrained Meta-Learner for Federated Learning [3.032797107899338]
  Federated learning is an approach to collaboratively machine learning models for multiple parties that prohibit data sharing. One of the challenges in federated learning is non-constrained data between clients, as a model can not fit data distribution for all clients.
  arXiv  Detail & Related papers  (2023-06-29T05:58:47Z)
• Personalizing Federated Learning with Over-the-Air Computations [84.8089761800994]
  Federated edge learning is a promising technology to deploy intelligence at the edge of wireless networks in a privacy-preserving manner. Under such a setting, multiple clients collaboratively train a global generic model under the coordination of an edge server. This paper presents a distributed training paradigm that employs analog over-the-air computation to address the communication bottleneck.
  arXiv  Detail & Related papers  (2023-02-24T08:41:19Z)
• Optimizing Server-side Aggregation For Robust Federated Learning via Subspace Training [80.03567604524268]
  Non-IID data distribution across clients and poisoning attacks are two main challenges in real-world federated learning systems. We propose SmartFL, a generic approach that optimize the server-side aggregation process. We provide theoretical analyses of the convergence and generalization capacity for SmartFL.
  arXiv  Detail & Related papers  (2022-11-10T13:20:56Z)
• Straggler-Resilient Personalized Federated Learning [55.54344312542944]
  Federated learning allows training models from samples distributed across a large network of clients while respecting privacy and communication restrictions. We develop a novel algorithmic procedure with theoretical speedup guarantees that simultaneously handles two of these hurdles. Our method relies on ideas from representation learning theory to find a global common representation using all clients' data and learn a user-specific set of parameters leading to a personalized solution for each client.
  arXiv  Detail & Related papers  (2022-06-05T01:14:46Z)
• Federated Multi-Target Domain Adaptation [99.93375364579484]
  Federated learning methods enable us to train machine learning models on distributed user data while preserving its privacy. We consider a more practical scenario where the distributed client data is unlabeled, and a centralized labeled dataset is available on the server. We propose an effective DualAdapt method to address the new challenges.
  arXiv  Detail & Related papers  (2021-08-17T17:53:05Z)
• Towards Fair Federated Learning with Zero-Shot Data Augmentation [123.37082242750866]
  Federated learning has emerged as an important distributed learning paradigm, where a server aggregates a global model from many client-trained models while having no access to the client data. We propose a novel federated learning system that employs zero-shot data augmentation on under-represented data to mitigate statistical heterogeneity and encourage more uniform accuracy performance across clients in federated networks. We study two variants of this scheme, Fed-ZDAC (federated learning with zero-shot data augmentation at the clients) and Fed-ZDAS (federated learning with zero-shot data augmentation at the server).
  arXiv  Detail & Related papers  (2021-04-27T18:23:54Z)
• Adaptive Distillation for Decentralized Learning from Heterogeneous Clients [9.261720698142097]
  We propose a new decentralized learning method called Decentralized Learning via Adaptive Distillation (DLAD) The proposed DLAD aggregates the outputs of the client models while adaptively emphasizing those with higher confidence in given distillation samples. Our extensive experimental evaluation on multiple public datasets demonstrates the effectiveness of the proposed method.
  arXiv  Detail & Related papers  (2020-08-18T14:25:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.