Extra Clients at No Extra Cost: Overcome Data Heterogeneity in Federated Learning with Filter Decomposition

• URL: http://arxiv.org/abs/2503.08652v1
• Date: Tue, 11 Mar 2025 17:42:36 GMT
• Title: Extra Clients at No Extra Cost: Overcome Data Heterogeneity in Federated Learning with Filter Decomposition
• Authors: Wei Chen, Qiang Qiu,
• Abstract summary: We propose a technique for decomposing a convolutional filter in federated learning (FL) into a linear combination of filter subspace elements.<n>This simple technique transforms global filter aggregation in FL into aggregating filter atoms and their atom coefficients.<n> Empirical results on benchmark datasets demonstrate that our filter decomposition technique substantially improves the accuracy of FL methods.
• Score: 25.658632928800962
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Data heterogeneity is one of the major challenges in federated learning (FL), which results in substantial client variance and slow convergence. In this study, we propose a novel solution: decomposing a convolutional filter in FL into a linear combination of filter subspace elements, i.e., filter atoms. This simple technique transforms global filter aggregation in FL into aggregating filter atoms and their atom coefficients. The key advantage here involves mathematically generating numerous cross-terms by expanding the product of two weighted sums from filter atom and atom coefficient. These cross-terms effectively emulate many additional latent clients, significantly reducing model variance, which is validated by our theoretical analysis and empirical observation. Furthermore, our method permits different training schemes for filter atoms and atom coefficients for highly adaptive model personalization and communication efficiency. Empirical results on benchmark datasets demonstrate that our filter decomposition technique substantially improves the accuracy of FL methods, confirming its efficacy in addressing data heterogeneity.

Related papers

• Understanding Dataset Distillation via Spectral Filtering [69.07076441512612]
  We introduce UniDD, a spectral filtering framework that unifies diverse DD objectives.<n>UniDD reveals that the essence of DD fundamentally lies in matching frequency-specific features.<n>To address this limitation, we propose Curriculum Frequency Matching (CFM), which gradually adjusts the filter parameter to cover both low- and high-frequency information.
  arXiv  Detail & Related papers  (2025-03-03T06:22:34Z)
• Adversarial Transform Particle Filters [11.330617592263744]
  The particle filter (PF) and the ensemble Kalman filter (EnKF) are widely used for approximate inference in state-space models.<n>We propose the Adversarial Transform Particle Filter (ATPF), a novel filtering framework that combines the strengths of the PF and the EnKF through adversarial learning.
  arXiv  Detail & Related papers  (2025-02-10T05:31:35Z)
• Permutation Invariant Learning with High-Dimensional Particle Filters [8.878254892409005]
  Sequential learning in deep models often suffers from challenges such as catastrophic forgetting and loss of plasticity. We introduce a novel permutation-invariant learning framework based on high-dimensional particle filters.
  arXiv  Detail & Related papers  (2024-10-30T05:06:55Z)
• Large Convolutional Model Tuning via Filter Subspace [28.223665047553016]
  We propose to fine-tune pre-trained models by adjusting only filter atoms, which are responsible for spatial-only convolution.<n>We show that such a simple scheme surpasses previous tuning baselines for both discriminate and generative tasks.
  arXiv  Detail & Related papers  (2024-03-01T04:16:08Z)
• Filter Pruning for Efficient CNNs via Knowledge-driven Differential Filter Sampler [103.97487121678276]
  Filter pruning simultaneously accelerates the computation and reduces the memory overhead of CNNs. We propose a novel Knowledge-driven Differential Filter Sampler(KDFS) with Masked Filter Modeling(MFM) framework for filter pruning.
  arXiv  Detail & Related papers  (2023-07-01T02:28:41Z)
• Filter Pruning via Filters Similarity in Consecutive Layers [20.29555787754269]
  Filter pruning is widely adopted to compress and accelerate the Convolutional Neural Networks (CNNs) We intuitively propose a novel pruning method by explicitly leveraging the Filters Similarity in Consecutive Layers (FSCL) Experiments demonstrate the effectiveness of FSCL, and it yields remarkable improvement over state-of-the-art on accuracy, FLOPs and parameter reduction.
  arXiv  Detail & Related papers  (2023-04-26T09:18:38Z)
• Computational Doob's h-transforms for Online Filtering of Discretely Observed Diffusions [65.74069050283998]
  We propose a computational framework to approximate Doob's $h$-transforms. The proposed approach can be orders of magnitude more efficient than state-of-the-art particle filters.
  arXiv  Detail & Related papers  (2022-06-07T15:03:05Z)
• Batch Normalization Tells You Which Filter is Important [49.903610684578716]
  We propose a simple yet effective filter pruning method by evaluating the importance of each filter based on the BN parameters of pre-trained CNNs. The experimental results on CIFAR-10 and ImageNet demonstrate that the proposed method can achieve outstanding performance.
  arXiv  Detail & Related papers  (2021-12-02T12:04:59Z)
• Training Compact CNNs for Image Classification using Dynamic-coded Filter Fusion [139.71852076031962]
  We present a novel filter pruning method, dubbed dynamic-coded filter fusion (DCFF) We derive compact CNNs in a computation-economical and regularization-free manner for efficient image classification. Our DCFF derives a compact VGGNet-16 with only 72.77M FLOPs and 1.06M parameters while reaching top-1 accuracy of 93.47%.
  arXiv  Detail & Related papers  (2021-07-14T18:07:38Z)
• FILTRA: Rethinking Steerable CNN by Filter Transform [59.412570807426135]
  The problem of steerable CNN has been studied from aspect of group representation theory. We show that kernel constructed by filter transform can also be interpreted in the group representation theory. This interpretation help complete the puzzle of steerable CNN theory and provides a novel and simple approach to implement steerable convolution operators.
  arXiv  Detail & Related papers  (2021-05-25T03:32:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.