Efficient Image Representation Learning with Federated Sampled Softmax
- URL: http://arxiv.org/abs/2203.04888v1
- Date: Wed, 9 Mar 2022 17:00:32 GMT
- Title: Efficient Image Representation Learning with Federated Sampled Softmax
- Authors: Sagar M. Waghmare, Hang Qi, Huizhong Chen, Mikhail Sirotenko and Tomer
Meron
- Abstract summary: Federated sampled softmax (FedSS) is a resource-efficient approach for learning image representation with Federated Learning.
We show that our method significantly reduces the number of parameters transferred to and optimized by the client devices.
- Score: 2.5557803548119464
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Learning image representations on decentralized data can bring many benefits
in cases where data cannot be aggregated across data silos. Softmax cross
entropy loss is highly effective and commonly used for learning image
representations. Using a large number of classes has proven to be particularly
beneficial for the descriptive power of such representations in centralized
learning. However, doing so on decentralized data with Federated Learning is
not straightforward as the demand on FL clients' computation and communication
increases proportionally to the number of classes. In this work we introduce
federated sampled softmax (FedSS), a resource-efficient approach for learning
image representation with Federated Learning. Specifically, the FL clients
sample a set of classes and optimize only the corresponding model parameters
with respect to a sampled softmax objective that approximates the global full
softmax objective. We examine the loss formulation and empirically show that
our method significantly reduces the number of parameters transferred to and
optimized by the client devices, while performing on par with the standard full
softmax method. This work creates a possibility for efficiently learning image
representations on decentralized data with a large number of classes under the
federated setting.
Related papers
- Federated Learning Across Decentralized and Unshared Archives for Remote Sensing Image Classification [2.725507329935916]
Federated learning (FL) enables the collaboration of multiple deep learning models to learn from decentralized data archives (i.e., clients) without accessing data on clients.
Although FL offers ample opportunities in knowledge discovery from distributed image archives, it is seldom considered in remote sensing (RS)
We present a comparative study of state-of-the-art FL algorithms for RS image classification problems.
arXiv Detail & Related papers (2023-11-10T15:58:53Z) - FedLALR: Client-Specific Adaptive Learning Rates Achieve Linear Speedup
for Non-IID Data [54.81695390763957]
Federated learning is an emerging distributed machine learning method.
We propose a heterogeneous local variant of AMSGrad, named FedLALR, in which each client adjusts its learning rate.
We show that our client-specified auto-tuned learning rate scheduling can converge and achieve linear speedup with respect to the number of clients.
arXiv Detail & Related papers (2023-09-18T12:35:05Z) - Towards Instance-adaptive Inference for Federated Learning [80.38701896056828]
Federated learning (FL) is a distributed learning paradigm that enables multiple clients to learn a powerful global model by aggregating local training.
In this paper, we present a novel FL algorithm, i.e., FedIns, to handle intra-client data heterogeneity by enabling instance-adaptive inference in the FL framework.
Our experiments show that our FedIns outperforms state-of-the-art FL algorithms, e.g., a 6.64% improvement against the top-performing method with less than 15% communication cost on Tiny-ImageNet.
arXiv Detail & Related papers (2023-08-11T09:58:47Z) - Joint Adaptive Representations for Image-Language Learning [59.40890927221377]
We propose a recipe for image-language learning, which produces effective models, outperforming bigger and more expensive ones, often trained on orders of magnitude larger datasets.
Our key finding is the joint learning of a compact vision and language representation, which adaptively and iteratively fuses the multi-modal features.
With only 40M training examples and with 39 GFLOPs our lightweight model outperforms many times larger state-of-the-art models of 2-20x more FLOPs and using bigger datasets some of which with close to 1B training examples.
arXiv Detail & Related papers (2023-05-31T15:02:02Z) - PFSL: Personalized & Fair Split Learning with Data & Label Privacy for
thin clients [0.5144809478361603]
PFSL is a new framework of distributed split learning where a large number of thin clients perform transfer learning in parallel.
We implement a lightweight step of personalization of client models to provide high performance for their respective data distributions.
Our accuracy far exceeds that of current algorithms SL and is very close to that of centralized learning on several real-life benchmarks.
arXiv Detail & Related papers (2023-03-19T10:38:29Z) - FedDM: Iterative Distribution Matching for Communication-Efficient
Federated Learning [87.08902493524556]
Federated learning(FL) has recently attracted increasing attention from academia and industry.
We propose FedDM to build the global training objective from multiple local surrogate functions.
In detail, we construct synthetic sets of data on each client to locally match the loss landscape from original data.
arXiv Detail & Related papers (2022-07-20T04:55:18Z) - Few-Max: Few-Shot Domain Adaptation for Unsupervised Contrastive
Representation Learning [7.748713051083396]
Contrastive self-supervised learning methods learn to map data points such as images into non-parametric representation space without requiring labels.
We propose a domain adaption method for self-supervised contrastive learning, termed Few-Max, to address the issue of adaptation to a target distribution under few-shot learning.
We evaluate Few-Max on a range of source and target datasets, including ImageNet, VisDA, and fastMRI, on which Few-Max consistently outperforms other approaches.
arXiv Detail & Related papers (2022-06-21T06:46:19Z) - An Expectation-Maximization Perspective on Federated Learning [75.67515842938299]
Federated learning describes the distributed training of models across multiple clients while keeping the data private on-device.
In this work, we view the server-orchestrated federated learning process as a hierarchical latent variable model where the server provides the parameters of a prior distribution over the client-specific model parameters.
We show that with simple Gaussian priors and a hard version of the well known Expectation-Maximization (EM) algorithm, learning in such a model corresponds to FedAvg, the most popular algorithm for the federated learning setting.
arXiv Detail & Related papers (2021-11-19T12:58:59Z) - Exploiting Shared Representations for Personalized Federated Learning [54.65133770989836]
We propose a novel federated learning framework and algorithm for learning a shared data representation across clients and unique local heads for each client.
Our algorithm harnesses the distributed computational power across clients to perform many local-updates with respect to the low-dimensional local parameters for every update of the representation.
This result is of interest beyond federated learning to a broad class of problems in which we aim to learn a shared low-dimensional representation among data distributions.
arXiv Detail & Related papers (2021-02-14T05:36:25Z) - Personalized Federated Learning with Moreau Envelopes [16.25105865597947]
Federated learning (FL) is a decentralized and privacy-preserving machine learning technique.
One challenge associated with FL is statistical diversity among clients.
We propose an algorithm for personalized FL (FedFedMe) using envelopes regularized loss function.
arXiv Detail & Related papers (2020-06-16T00:55:23Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.