Toward Enhancing Representation Learning in Federated Multi-Task Settings

• URL: http://arxiv.org/abs/2602.01626v1
• Date: Mon, 02 Feb 2026 04:39:36 GMT
• Title: Toward Enhancing Representation Learning in Federated Multi-Task Settings
• Authors: Mehdi Setayesh, Mahdi Beitollahi, Yasser H. Khalil, Hongliang Li,
• Abstract summary: Federated multi-task learning (FMTL) seeks to collaboratively train customized models for users with different tasks.<n>We propose Muscle loss, a contrastive learning objective that simultaneously aligns representations from all participating models.<n>We develop FedMuscle, a practical and communication-efficient FMTL algorithm that naturally handles both model and task.
• Score: 13.483094713610322
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Federated multi-task learning (FMTL) seeks to collaboratively train customized models for users with different tasks while preserving data privacy. Most existing approaches assume model congruity (i.e., the use of fully or partially homogeneous models) across users, which limits their applicability in realistic settings. To overcome this limitation, we aim to learn a shared representation space across tasks rather than shared model parameters. To this end, we propose Muscle loss, a novel contrastive learning objective that simultaneously aligns representations from all participating models. Unlike existing multi-view or multi-model contrastive methods, which typically align models pairwise, Muscle loss can effectively capture dependencies across tasks because its minimization is equivalent to the maximization of mutual information among all the models' representations. Building on this principle, we develop FedMuscle, a practical and communication-efficient FMTL algorithm that naturally handles both model and task heterogeneity. Experiments on diverse image and language tasks demonstrate that FedMuscle consistently outperforms state-of-the-art baselines, delivering substantial improvements and robust performance across heterogeneous settings.

Related papers

• Towards Unified Modeling in Federated Multi-Task Learning via Subspace Decoupling [23.642760378344335]
  Federated Multi-Task Learning (FMTL) enables multiple clients performing heterogeneous tasks without exchanging their local data.<n>Most existing FMTL methods focus on building personalized models for each client and unable to support the aggregation of multiple heterogeneous tasks into a unified model.<n>We propose FedDEA, an update-structure-aware aggregation method specifically designed for multi-task model integration.
  arXiv  Detail & Related papers  (2025-05-30T03:53:21Z)
• Not All Clients Are Equal: Collaborative Model Personalization on Heterogeneous Multi-Modal Clients [59.52341877720199]
  We propose FedMosaic, a method that enables knowledge sharing across heterogeneous architectures without huge computational cost.<n>To mimic the real-world task diversity, we propose a multi-modal PFL benchmark spanning 40 distinct tasks with distribution shifts over time.<n>The empirical study shows that FedMosaic outperforms the state-of-the-art PFL methods.
  arXiv  Detail & Related papers  (2025-05-20T09:17:07Z)
• Modeling Multi-Task Model Merging as Adaptive Projective Gradient Descent [72.10987117380584]
  Merging multiple expert models offers a promising approach for performing multi-task learning without accessing their original data.<n>We find existing methods discard task-specific information that, while causing conflicts, is crucial for performance.<n>Our approach consistently outperforms previous methods, achieving state-of-the-art results across diverse architectures and tasks in both vision and NLP domains.
  arXiv  Detail & Related papers  (2025-01-02T12:45:21Z)
• Task Groupings Regularization: Data-Free Meta-Learning with Heterogeneous Pre-trained Models [83.02797560769285]
  Data-Free Meta-Learning (DFML) aims to derive knowledge from a collection of pre-trained models without accessing their original data.<n>Current methods often overlook the heterogeneity among pre-trained models, which leads to performance degradation due to task conflicts.
  arXiv  Detail & Related papers  (2024-05-26T13:11:55Z)
• Merging Multi-Task Models via Weight-Ensembling Mixture of Experts [64.94129594112557]
  Merging Transformer-based models trained on different tasks into a single unified model can execute all the tasks concurrently. Previous methods, exemplified by task arithmetic, have been proven to be both effective and scalable. We propose to merge most of the parameters while upscaling the Transformer layers to a weight-ensembling mixture of experts (MoE) module.
  arXiv  Detail & Related papers  (2024-02-01T08:58:57Z)
• Concrete Subspace Learning based Interference Elimination for Multi-task Model Fusion [86.6191592951269]
  Merging models fine-tuned from common extensively pretrained large model but specialized for different tasks has been demonstrated as a cheap and scalable strategy to construct a multitask model that performs well across diverse tasks. We propose the CONtinuous relaxation dis (Concrete) subspace learning method to identify a common lowdimensional subspace and utilize its shared information track interference problem without sacrificing performance.
  arXiv  Detail & Related papers  (2023-12-11T07:24:54Z)
• DEPHN: Different Expression Parallel Heterogeneous Network using virtual gradient optimization for Multi-task Learning [1.0705399532413615]
  Recommendation system algorithm based on multi-task learning (MTL) is the major method for Internet operators to understand users and predict their behaviors. Traditional models use shared-bottom models and gating experts to realize shared representation learning and information differentiation. We propose an Different Expression Parallel Heterogeneous Network (DEPHN) to model multiple tasks simultaneously.
  arXiv  Detail & Related papers  (2023-07-24T04:29:00Z)
• FedBone: Towards Large-Scale Federated Multi-Task Learning [13.835972363413884]
  In real-world applications, visual and natural language tasks typically require large-scale models to extract high-level abstract features. Existing HFML methods disregard the impact of gradient conflicts on multi-task optimization. We propose an innovative framework called FedBone, which enables the construction of large-scale models with better generalization.
  arXiv  Detail & Related papers  (2023-06-30T08:19:38Z)
• Task-Feature Collaborative Learning with Application to Personalized Attribute Prediction [166.87111665908333]
  We propose a novel multi-task learning method called Task-Feature Collaborative Learning (TFCL) Specifically, we first propose a base model with a heterogeneous block-diagonal structure regularizer to leverage the collaborative grouping of features and tasks. As a practical extension, we extend the base model by allowing overlapping features and differentiating the hard tasks.
  arXiv  Detail & Related papers  (2020-04-29T02:32:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.