Multi-task Federated Learning with Encoder-Decoder Structure: Enabling Collaborative Learning Across Different Tasks

• URL: http://arxiv.org/abs/2504.09800v1
• Date: Mon, 14 Apr 2025 02:01:39 GMT
• Title: Multi-task Federated Learning with Encoder-Decoder Structure: Enabling Collaborative Learning Across Different Tasks
• Authors: Jingxuan Zhou, Weidong Bao, Ji Wang, Dayu Zhang, Xiongtao Zhang, Yaohong Zhang,
• Abstract summary: We propose a multi-task federated learning with encoder-decoder structure (M-Fed)<n>We leverage this structure to share intra-task knowledge through traditional federated learning methods and extract general knowledge from the encoder to achieve cross-task knowledge sharing.<n>Our method is lightweight and modular, demonstrating innovation compared to previous research.
• Score: 4.949677105074088
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Federated learning has been extensively studied and applied due to its ability to ensure data security in distributed environments while building better models. However, clients participating in federated learning still face limitations, as clients with different structures or tasks cannot participate in learning together. In view of this, constructing a federated learning framework that allows collaboration between clients with different model structures and performing different tasks, enabling them to share valuable knowledge to enhance model efficiency, holds significant practical implications for the widespread application of federated learning. To achieve this goal, we propose a multi-task federated learning with encoder-decoder structure (M-Fed). Specifically, given the widespread adoption of the encoder-decoder architecture in current models, we leverage this structure to share intra-task knowledge through traditional federated learning methods and extract general knowledge from the encoder to achieve cross-task knowledge sharing. The training process is similar to traditional federated learning, and we incorporate local decoder and global decoder information into the loss function. The local decoder iteratively updates and gradually approaches the global decoder until sufficient cross-task knowledge sharing is achieved. Our method is lightweight and modular, demonstrating innovation compared to previous research. It enables clients performing different tasks to share general knowledge while maintaining the efficiency of traditional federated learning systems. We conducted experiments on two widely used benchmark datasets to verify the feasibility of M-Fed and compared it with traditional methods. The experimental results demonstrate the effectiveness of M-Fed in multi-task federated learning.

Related papers

• Recognizing Unseen Objects via Multimodal Intensive Knowledge Graph Propagation [68.13453771001522]
  We propose a multimodal intensive ZSL framework that matches regions of images with corresponding semantic embeddings. We conduct extensive experiments and evaluate our model on large-scale real-world data.
  arXiv  Detail & Related papers  (2023-06-14T13:07:48Z)
• Pre-training Multi-task Contrastive Learning Models for Scientific Literature Understanding [52.723297744257536]
  Pre-trained language models (LMs) have shown effectiveness in scientific literature understanding tasks. We propose a multi-task contrastive learning framework, SciMult, to facilitate common knowledge sharing across different literature understanding tasks.
  arXiv  Detail & Related papers  (2023-05-23T16:47:22Z)
• Concept Discovery for Fast Adapatation [42.81705659613234]
  We introduce concept discovery to the few-shot learning problem, where we achieve more effective adaptation by meta-learning the structure among the data features. Our proposed method Concept-Based Model-Agnostic Meta-Learning (COMAML) has been shown to achieve consistent improvements in the structured data for both synthesized datasets and real-world datasets.
  arXiv  Detail & Related papers  (2023-01-19T02:33:58Z)
• Leveraging Different Learning Styles for Improved Knowledge Distillation in Biomedical Imaging [0.9208007322096533]
  Our work endeavors to leverage the concept of knowledge diversification to improve the performance of model compression techniques like Knowledge Distillation (KD) and Mutual Learning (ML) We use a single-teacher and two-student network in a unified framework that not only allows for the transfer of knowledge from teacher to students (KD) but also encourages collaborative learning between students (ML) Unlike the conventional approach, where the teacher shares the same knowledge in the form of predictions or feature representations with the student network, our proposed approach employs a more diversified strategy by training one student with predictions and the other with feature maps from the teacher.
  arXiv  Detail & Related papers  (2022-12-06T12:40:45Z)
• Multi-Task and Transfer Learning for Federated Learning Applications [5.224306534441244]
  Federated learning enables applications benefiting distributed and private datasets of a large number of potential data-holding clients. We propose to train a deep neural network model with more generalized layers closer to the input and more personalized layers to the output. We provide simulation results to highlight particular scenarios in which meta-learning-based federated learning proves to be useful.
  arXiv  Detail & Related papers  (2022-07-17T11:48:11Z)
• EEML: Ensemble Embedded Meta-learning [5.9514420658483935]
  We propose an ensemble embedded meta-learning algorithm (EEML) that explicitly utilizes multi-model-ensemble to organize prior knowledge into diverse specific experts. We rely on a task embedding cluster mechanism to deliver diverse tasks to matching experts in training process and instruct how experts collaborate in test phase. The experimental results show that the proposed method outperforms recent state-of-the-arts easily in few-shot learning problem.
  arXiv  Detail & Related papers  (2022-06-18T12:37:17Z)
• A Unified Continuous Learning Framework for Multi-modal Knowledge Discovery and Pre-training [73.7507857547549]
  We propose to unify knowledge discovery and multi-modal pre-training in a continuous learning framework. For knowledge discovery, a pre-trained model is used to identify cross-modal links on a graph. For model pre-training, the knowledge graph is used as the external knowledge to guide the model updating.
  arXiv  Detail & Related papers  (2022-06-11T16:05:06Z)
• Towards a Universal Continuous Knowledge Base [49.95342223987143]
  We propose a method for building a continuous knowledge base that can store knowledge imported from multiple neural networks. Experiments on text classification show promising results. We import the knowledge from multiple models to the knowledge base, from which the fused knowledge is exported back to a single model.
  arXiv  Detail & Related papers  (2020-12-25T12:27:44Z)
• Practical One-Shot Federated Learning for Cross-Silo Setting [114.76232507580067]
  One-shot federated learning is a promising approach to make federated learning applicable in cross-silo setting. We propose a practical one-shot federated learning algorithm named FedKT. By utilizing the knowledge transfer technique, FedKT can be applied to any classification models and can flexibly achieve differential privacy guarantees.
  arXiv  Detail & Related papers  (2020-10-02T14:09:10Z)
• Learning From Multiple Experts: Self-paced Knowledge Distillation for Long-tailed Classification [106.08067870620218]
  We propose a self-paced knowledge distillation framework, termed Learning From Multiple Experts (LFME) We refer to these models as 'Experts', and the proposed LFME framework aggregates the knowledge from multiple 'Experts' to learn a unified student model. We conduct extensive experiments and demonstrate that our method is able to achieve superior performances compared to state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-01-06T12:57:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.