FedRSClip: Federated Learning for Remote Sensing Scene Classification Using Vision-Language Models

• URL: http://arxiv.org/abs/2501.02461v1
• Date: Sun, 05 Jan 2025 07:10:27 GMT
• Title: FedRSClip: Federated Learning for Remote Sensing Scene Classification Using Vision-Language Models
• Authors: Hui Lin, Chao Zhang, Danfeng Hong, Kexin Dong, Congcong Wen,
• Abstract summary: We propose FedRSCLIP, the first federated learning framework for remote sensing image classification based on a VLM, specifically CLIP. FedRSCLIP addresses the challenges of data heterogeneity and large-scale model transmission in federated environments by introducing Prompt Learning. To validate the effectiveness of our proposed model, we construct a Fed-RSIC dataset based on three existing remote sensing image classification datasets.
• Score: 23.830133838392964
• License:
• Abstract: Remote sensing data is often distributed across multiple institutions, and due to privacy concerns and data-sharing restrictions, leveraging large-scale datasets in a centralized training framework is challenging. Federated learning offers a promising solution by enabling collaborative model training across distributed data sources without requiring data centralization. However, current Vision-Language Models (VLMs), which typically contain billions of parameters, pose significant communication challenges for traditional federated learning approaches based on model parameter updates, as they would incur substantial communication costs. In this paper, we propose FedRSCLIP, the first federated learning framework designed for remote sensing image classification based on a VLM, specifically CLIP. FedRSCLIP addresses the challenges of data heterogeneity and large-scale model transmission in federated environments by introducing Prompt Learning, which optimizes only a small set of tunable parameters. The framework introduces a dual-prompt mechanism, comprising Shared Prompts for global knowledge sharing and Private Prompts for client-specific adaptation. To maintain semantic coherence between shared and private prompts, we propose the Dual Prompt Alignment Constraint to balance global consistency and local adaptability across diverse client distributions. Additionally, to enhance cross-modal representation learning, we introduce the Cross-Modal Feature Alignment Constraint to align multimodal features between text and image prompts. To validate the effectiveness of our proposed model, we construct a Fed-RSIC dataset based on three existing remote sensing image classification datasets, specifically designed to simulate various federated learning configurations. Experimental results demonstrate the effectiveness and superiority of FedRSCLIP in remote sensing image classification.

Related papers

• Personalized federated learning based on feature fusion [2.943623084019036]
  Federated learning enables distributed clients to collaborate on training while storing their data locally to protect client privacy. We propose a personalized federated learning approach called pFedPM. In our process, we replace traditional gradient uploading with feature uploading, which helps reduce communication costs and allows for heterogeneous client models.
  arXiv  Detail & Related papers  (2024-06-24T12:16:51Z)
• Leveraging Foundation Models for Multi-modal Federated Learning with Incomplete Modality [41.79433449873368]
  We propose a novel multi-modal federated learning method, Federated Multi-modal contrastiVe training with Pre-trained completion (FedMVP) FedMVP integrates the large-scale pre-trained models to enhance the federated training. We demonstrate that the model achieves superior performance over two real-world image-text classification datasets.
  arXiv  Detail & Related papers  (2024-06-16T19:18:06Z)
• Enhancing Information Maximization with Distance-Aware Contrastive Learning for Source-Free Cross-Domain Few-Shot Learning [55.715623885418815]
  Cross-Domain Few-Shot Learning methods require access to source domain data to train a model in the pre-training phase. Due to increasing concerns about data privacy and the desire to reduce data transmission and training costs, it is necessary to develop a CDFSL solution without accessing source data. This paper proposes an Enhanced Information Maximization with Distance-Aware Contrastive Learning method to address these challenges.
  arXiv  Detail & Related papers  (2024-03-04T12:10:24Z)
• Communication-Efficient Personalized Federated Learning for Speech-to-Text Tasks [64.02867484165476]
  To protect privacy and meet legal regulations, federated learning (FL) has gained significant attention for training speech-to-text (S2T) systems. The commonly used FL approach (i.e., textscFedAvg) in S2T tasks typically suffers from extensive communication overhead. We propose a personalized federated S2T framework that introduces textscFedLoRA, a lightweight LoRA module for client-side tuning and interaction with the server, and textscFedMem, a global model equipped with a $k$-near
  arXiv  Detail & Related papers  (2024-01-18T15:39:38Z)
• FedDiff: Diffusion Model Driven Federated Learning for Multi-Modal and Multi-Clients [32.59184269562571]
  We propose a multi-modal collaborative diffusion federated learning framework called FedDiff. Our framework establishes a dual-branch diffusion model feature extraction setup, where the two modal data are inputted into separate branches of the encoder. Considering the challenge of private and efficient communication between multiple clients, we embed the diffusion model into the federated learning communication structure.
  arXiv  Detail & Related papers  (2023-11-16T02:29:37Z)
• FACT: Federated Adversarial Cross Training [0.0]
  Federated Adrial Cross Training (FACT) uses implicit domain differences between source clients to identify domain shifts in the target domain. We empirically show that FACT outperforms state-of-the-art federated, non-federated and source-free domain adaptation models.
  arXiv  Detail & Related papers  (2023-06-01T12:25:43Z)
• Multimodal Federated Learning via Contrastive Representation Ensemble [17.08211358391482]
  Federated learning (FL) serves as a privacy-conscious alternative to centralized machine learning. Existing FL methods all rely on model aggregation on single modality level. We propose Contrastive Representation Ensemble and Aggregation for Multimodal FL (CreamFL)
  arXiv  Detail & Related papers  (2023-02-17T14:17:44Z)
• Semi-supervised Domain Adaptive Structure Learning [72.01544419893628]
  Semi-supervised domain adaptation (SSDA) is a challenging problem requiring methods to overcome both 1) overfitting towards poorly annotated data and 2) distribution shift across domains. We introduce an adaptive structure learning method to regularize the cooperation of SSL and DA.
  arXiv  Detail & Related papers  (2021-12-12T06:11:16Z)
• Exploiting Shared Representations for Personalized Federated Learning [54.65133770989836]
  We propose a novel federated learning framework and algorithm for learning a shared data representation across clients and unique local heads for each client. Our algorithm harnesses the distributed computational power across clients to perform many local-updates with respect to the low-dimensional local parameters for every update of the representation. This result is of interest beyond federated learning to a broad class of problems in which we aim to learn a shared low-dimensional representation among data distributions.
  arXiv  Detail & Related papers  (2021-02-14T05:36:25Z)
• Edge-assisted Democratized Learning Towards Federated Analytics [67.44078999945722]
  We show the hierarchical learning structure of the proposed edge-assisted democratized learning mechanism, namely Edge-DemLearn. We also validate Edge-DemLearn as a flexible model training mechanism to build a distributed control and aggregation methodology in regions.
  arXiv  Detail & Related papers  (2020-12-01T11:46:03Z)
• Federated Unsupervised Representation Learning [56.715917111878106]
  We formulate a new problem in federated learning called Federated Unsupervised Representation Learning (FURL) to learn a common representation model without supervision. FedCA is composed of two key modules: dictionary module to aggregate the representations of samples from each client and share with all clients for consistency of representation space and alignment module to align the representation of each client on a base model trained on a public data.
  arXiv  Detail & Related papers  (2020-10-18T13:28:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.