Query-based Knowledge Transfer for Heterogeneous Learning Environments

• URL: http://arxiv.org/abs/2504.09205v1
• Date: Sat, 12 Apr 2025 13:09:39 GMT
• Title: Query-based Knowledge Transfer for Heterogeneous Learning Environments
• Authors: Norah Alballa, Wenxuan Zhang, Ziquan Liu, Ahmed M. Abdelmoniem, Mohamed Elhoseiny, Marco Canini,
• Abstract summary: We propose a novel framework called Query-based Knowledge Transfer (QKT)<n>QKT enables tailored knowledge acquisition to fulfill specific client needs without direct data exchange.<n>Our experiments show that QKT significantly outperforms existing collaborative learning methods.
• Score: 50.45210784447839
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Decentralized collaborative learning under data heterogeneity and privacy constraints has rapidly advanced. However, existing solutions like federated learning, ensembles, and transfer learning, often fail to adequately serve the unique needs of clients, especially when local data representation is limited. To address this issue, we propose a novel framework called Query-based Knowledge Transfer (QKT) that enables tailored knowledge acquisition to fulfill specific client needs without direct data exchange. QKT employs a data-free masking strategy to facilitate communication-efficient query-focused knowledge transfer while refining task-specific parameters to mitigate knowledge interference and forgetting. Our experiments, conducted on both standard and clinical benchmarks, show that QKT significantly outperforms existing collaborative learning methods by an average of 20.91\% points in single-class query settings and an average of 14.32\% points in multi-class query scenarios. Further analysis and ablation studies reveal that QKT effectively balances the learning of new and existing knowledge, showing strong potential for its application in decentralized learning.

Related papers

• Accurate Forgetting for Heterogeneous Federated Continual Learning [89.08735771893608]
  We propose a new concept accurate forgetting (AF) and develop a novel generative-replay methodMethodwhich selectively utilizes previous knowledge in federated networks. We employ a probabilistic framework based on a normalizing flow model to quantify the credibility of previous knowledge.
  arXiv  Detail & Related papers  (2025-02-20T02:35:17Z)
• FedAuxHMTL: Federated Auxiliary Hard-Parameter Sharing Multi-Task Learning for Network Edge Traffic Classification [9.816810723612653]
  This paper introduces a new framework for federated auxiliary hard- parameter sharing multi-task learning, namely, FedAuxHMTL. It incorporates model parameter exchanges between edge server and base stations, enabling base stations from distributed areas to participate in the FedAuxHMTL process. Empirical experiments are conducted to validate and demonstrate the FedAuxHMTL's effectiveness in terms of accuracy, total global loss, communication costs, computing time, and energy consumption.
  arXiv  Detail & Related papers  (2024-04-11T16:23:28Z)
• Global and Local Prompts Cooperation via Optimal Transport for Federated Learning [13.652593797756774]
  We present Federated Prompts Cooperation via Optimal Transport (FedOTP), which introduces efficient collaborative prompt learning strategies to capture diverse category traits on a per-client basis. Specifically, for each client, we learn a global prompt to extract consensus knowledge among clients, and a local prompt to capture client-specific category characteristics. Unbalanced Optimal Transport is then employed to align local visual features with these prompts, striking a balance between global consensus and local personalization.
  arXiv  Detail & Related papers  (2024-02-29T11:43:04Z)
• FedLALR: Client-Specific Adaptive Learning Rates Achieve Linear Speedup for Non-IID Data [54.81695390763957]
  Federated learning is an emerging distributed machine learning method. We propose a heterogeneous local variant of AMSGrad, named FedLALR, in which each client adjusts its learning rate. We show that our client-specified auto-tuned learning rate scheduling can converge and achieve linear speedup with respect to the number of clients.
  arXiv  Detail & Related papers  (2023-09-18T12:35:05Z)
• Knowledge-Aware Federated Active Learning with Non-IID Data [75.98707107158175]
  We propose a federated active learning paradigm to efficiently learn a global model with limited annotation budget. The main challenge faced by federated active learning is the mismatch between the active sampling goal of the global model on the server and that of the local clients. We propose Knowledge-Aware Federated Active Learning (KAFAL), which consists of Knowledge-Specialized Active Sampling (KSAS) and Knowledge-Compensatory Federated Update (KCFU)
  arXiv  Detail & Related papers  (2022-11-24T13:08:43Z)
• Feature Correlation-guided Knowledge Transfer for Federated Self-supervised Learning [19.505644178449046]
  We propose a novel and general method named Federated Self-supervised Learning with Feature-correlation based Aggregation (FedFoA) Our insight is to utilize feature correlation to align the feature mappings and calibrate the local model updates across clients during their local training process. We prove that FedFoA is a model-agnostic training framework and can be easily compatible with state-of-the-art unsupervised FL methods.
  arXiv  Detail & Related papers  (2022-11-14T13:59:50Z)
• Federated Self-supervised Learning for Heterogeneous Clients [20.33482170846688]
  We propose a unified and systematic framework, emphHeterogeneous Self-supervised Federated Learning (Hetero-SSFL) for enabling self-supervised learning with federation on heterogeneous clients. The proposed framework allows representation learning across all the clients without imposing architectural constraints or requiring presence of labeled data. We empirically demonstrate that our proposed approach outperforms the state of the art methods by a significant margin.
  arXiv  Detail & Related papers  (2022-05-25T05:07:44Z)
• Addressing Client Drift in Federated Continual Learning with Adaptive Optimization [10.303676184878896]
  We outline a framework for performing Federated Continual Learning (FCL) by using NetTailor as a candidate continual learning approach. We show that adaptive federated optimization can reduce the adverse impact of client drift and showcase its effectiveness on CIFAR100, MiniImagenet, and Decathlon benchmarks.
  arXiv  Detail & Related papers  (2022-03-24T20:00:03Z)
• CosSGD: Nonlinear Quantization for Communication-efficient Federated Learning [62.65937719264881]
  Federated learning facilitates learning across clients without transferring local data on these clients to a central server. We propose a nonlinear quantization for compressed gradient descent, which can be easily utilized in federated learning. Our system significantly reduces the communication cost by up to three orders of magnitude, while maintaining convergence and accuracy of the training process.
  arXiv  Detail & Related papers  (2020-12-15T12:20:28Z)
• Federated Continual Learning with Weighted Inter-client Transfer [79.93004004545736]
  We propose a novel federated continual learning framework, Federated Weighted Inter-client Transfer (FedWeIT) FedWeIT decomposes the network weights into global federated parameters and sparse task-specific parameters, and each client receives selective knowledge from other clients. We validate our FedWeIT against existing federated learning and continual learning methods, and our model significantly outperforms them with a large reduction in the communication cost.
  arXiv  Detail & Related papers  (2020-03-06T13:33:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.