Handling Data Heterogeneity via Architectural Design for Federated Visual Recognition

• URL: http://arxiv.org/abs/2310.15165v1
• Date: Mon, 23 Oct 2023 17:59:16 GMT
• Title: Handling Data Heterogeneity via Architectural Design for Federated Visual Recognition
• Authors: Sara Pieri, Jose Renato Restom, Samuel Horvath, Hisham Cholakkal
• Abstract summary: We study 19 visual recognition models from five different architectural families on four challenging FL datasets. Our findings emphasize the importance of architectural design for computer vision tasks in practical scenarios.
• Score: 16.50490537786593
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Federated Learning (FL) is a promising research paradigm that enables the collaborative training of machine learning models among various parties without the need for sensitive information exchange. Nonetheless, retaining data in individual clients introduces fundamental challenges to achieving performance on par with centrally trained models. Our study provides an extensive review of federated learning applied to visual recognition. It underscores the critical role of thoughtful architectural design choices in achieving optimal performance, a factor often neglected in the FL literature. Many existing FL solutions are tested on shallow or simple networks, which may not accurately reflect real-world applications. This practice restricts the transferability of research findings to large-scale visual recognition models. Through an in-depth analysis of diverse cutting-edge architectures such as convolutional neural networks, transformers, and MLP-mixers, we experimentally demonstrate that architectural choices can substantially enhance FL systems' performance, particularly when handling heterogeneous data. We study 19 visual recognition models from five different architectural families on four challenging FL datasets. We also re-investigate the inferior performance of convolution-based architectures in the FL setting and analyze the influence of normalization layers on the FL performance. Our findings emphasize the importance of architectural design for computer vision tasks in practical scenarios, effectively narrowing the performance gap between federated and centralized learning. Our source code is available at https://github.com/sarapieri/fed_het.git.

Related papers

• Can We Theoretically Quantify the Impacts of Local Updates on the Generalization Performance of Federated Learning? [50.03434441234569]
  Federated Learning (FL) has gained significant popularity due to its effectiveness in training machine learning models across diverse sites without requiring direct data sharing. While various algorithms have shown that FL with local updates is a communication-efficient distributed learning framework, the generalization performance of FL with local updates has received comparatively less attention.
  arXiv  Detail & Related papers  (2024-09-05T19:00:18Z)
• FedConv: Enhancing Convolutional Neural Networks for Handling Data Heterogeneity in Federated Learning [34.37155882617201]
  Federated learning (FL) is an emerging paradigm in machine learning, where a shared model is collaboratively learned using data from multiple devices. We systematically investigate the impact of different architectural elements, such as activation functions and normalization layers, on the performance within heterogeneous FL. Our findings indicate that with strategic architectural modifications, pure CNNs can achieve a level of robustness that either matches or even exceeds that of ViTs.
  arXiv  Detail & Related papers  (2023-10-06T17:57:50Z)
• FS-Real: Towards Real-World Cross-Device Federated Learning [60.91678132132229]
  Federated Learning (FL) aims to train high-quality models in collaboration with distributed clients while not uploading their local data. There is still a considerable gap between the flourishing FL research and real-world scenarios, mainly caused by the characteristics of heterogeneous devices and its scales. We propose an efficient and scalable prototyping system for real-world cross-device FL, FS-Real.
  arXiv  Detail & Related papers  (2023-03-23T15:37:17Z)
• FedCV: A Federated Learning Framework for Diverse Computer Vision Tasks [38.012182901565616]
  Federated Learning (FL) is a distributed learning paradigm that can learn a global or personalized model from decentralized datasets on edge devices. FL has rarely been demonstrated effectively in advanced computer vision tasks such as object detection and image segmentation. We provide non-I.I.D. benchmarking datasets, models, and various reference FL algorithms.
  arXiv  Detail & Related papers  (2021-11-22T09:26:08Z)
• Mobility-Aware Cluster Federated Learning in Hierarchical Wireless Networks [81.83990083088345]
  We develop a theoretical model to characterize the hierarchical federated learning (HFL) algorithm in wireless networks. Our analysis proves that the learning performance of HFL deteriorates drastically with highly-mobile users. To circumvent these issues, we propose a mobility-aware cluster federated learning (MACFL) algorithm.
  arXiv  Detail & Related papers  (2021-08-20T10:46:58Z)
• Rethinking Architecture Design for Tackling Data Heterogeneity in Federated Learning [53.73083199055093]
  We show that attention-based architectures (e.g., Transformers) are fairly robust to distribution shifts. Our experiments show that replacing convolutional networks with Transformers can greatly reduce catastrophic forgetting of previous devices.
  arXiv  Detail & Related papers  (2021-06-10T21:04:18Z)
• FedScale: Benchmarking Model and System Performance of Federated Learning [4.1617240682257925]
  FedScale is a set of challenging and realistic benchmark datasets for federated learning (FL) research. FedScale is open-source with permissive licenses and actively maintained.
  arXiv  Detail & Related papers  (2021-05-24T15:55:27Z)
• Multi-Perspective LSTM for Joint Visual Representation Learning [81.21490913108835]
  We present a novel LSTM cell architecture capable of learning both intra- and inter-perspective relationships available in visual sequences captured from multiple perspectives. Our architecture adopts a novel recurrent joint learning strategy that uses additional gates and memories at the cell level. We show that by using the proposed cell to create a network, more effective and richer visual representations are learned for recognition tasks.
  arXiv  Detail & Related papers  (2021-05-06T16:44:40Z)
• On the Impact of Device and Behavioral Heterogeneity in Federated Learning [5.038980064083677]
  Federated learning (FL) is becoming a popular paradigm for collaborative learning over distributed, private datasets owned by non-trusting entities. This paper describes the challenge of performing training over largely heterogeneous datasets, devices, and networks. We conduct an empirical study spanning close to 1.5K unique configurations on five popular FL benchmarks.
  arXiv  Detail & Related papers  (2021-02-15T12:04:38Z)
• Self-supervised Cross-silo Federated Neural Architecture Search [13.971827232338716]
  We present Self-supervised Vertical Federated Neural Architecture Search (SS-VFNAS) for automating Vertical Federated Learning (VFL) In the proposed framework, each party first conducts NAS using self-supervised approach to find a local optimal architecture with its own data. We demonstrate experimentally that our approach has superior performance, communication efficiency and privacy compared to Federated NAS.
  arXiv  Detail & Related papers  (2021-01-28T09:57:30Z)
• Edge-assisted Democratized Learning Towards Federated Analytics [67.44078999945722]
  We show the hierarchical learning structure of the proposed edge-assisted democratized learning mechanism, namely Edge-DemLearn. We also validate Edge-DemLearn as a flexible model training mechanism to build a distributed control and aggregation methodology in regions.
  arXiv  Detail & Related papers  (2020-12-01T11:46:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.