On the Effectiveness of Adaptation Strategies for VLM-Based Federated Learning in Remote Sensing

• URL: http://arxiv.org/abs/2608.04791v1
• Date: Wed, 05 Aug 2026 12:57:13 GMT
• Title: On the Effectiveness of Adaptation Strategies for VLM-Based Federated Learning in Remote Sensing
• Abstract summary: Federated learning (FL) enables collaborative training of deep learning models across decentralized image archives without requiring data centralization.<n>The presence of training data heterogeneity across clients (known as non-IID data) can impede convergence.<n>It is crucial to select an appropriate VLM adaptation strategy that balances the generalization ability with the communication and computational constraints.
• Score: 7.060465634216585
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Federated learning (FL) enables collaborative training of deep learning models across decentralized image archives without requiring data centralization. This paradigm is particularly relevant in remote sensing (RS), where legal regulations, privacy concerns, and bandwidth constraints restrict data sharing. However, the presence of training data heterogeneity across clients (known as non-IID data) can impede convergence and limit the generalization capability of the aggregated global model. To mitigate the adverse effects of training data heterogeneity, vision-language models (VLMs) can be leveraged in FL due to their transferable representations, which have demonstrated robustness under distribution shifts. However, their large parameter size may substantially increase communication overhead and local computational complexity in federated settings. Therefore, it is crucial to select an appropriate VLM adaptation strategy that balances the generalization ability with the communication and computational constraints. To address this issue, in this paper, we present the first comparative study of VLM adaptation strategies for FL in the context of RS image classification. We investigate full fine-tuning, encoder-specific fine-tuning, prompt learning, and low-rank adaptation (LoRA) tuning, and analyze them with respect to three criteria: 1) generalization capability under non-IID data, 2) communication overhead, and 3) local computational complexity. Experiments on BigEarthNet-S2, EuroSAT, RESISC45, and ImageNet reveal distinct trade-offs between task specialization, cross-domain generalization, and efficiency. Based on our findings, we derive a guideline for the selection of an appropriate VLM adaptation strategy in FL for RS image classification under different operational constraints. The code of this work is publicly available at https://git.tu-berlin.de/rsim/FL-RS-VLM.

Related papers

• FedA2L: Adaptive layer-wise learning rate adjustment in decentralized federated learning [2.252594484945365]
  Decentralized intelligence systems with heterogeneous devices increasingly rely on decentralized federated learning (DFL)<n>This work introduces FedA2L, a method that dynamically adjusts layer-wise LRs based on model divergence signals.<n>By leveraging local update intensity and network consensus constraints, FedA2L seamlessly integrates into existing DFL protocols without additional communication or coordination.
  arXiv  Detail & Related papers  (2026-08-10T07:27:21Z)
• Replacing Parameters with Preferences: Federated Alignment of Heterogeneous Vision-Language Models [63.70401095689976]
  Vision-Language Models (VLMs) have broad potential in privacy-sensitive domains such as healthcare and finance.<n>We propose MoR, a federated alignment framework that combines GRPO with Mixture-of-Rewards for heterogeneousVLMs.<n>MoR consistently outperforms federated alignment baselines in generalization and cross-client adaptability.
  arXiv  Detail & Related papers  (2026-05-05T07:02:50Z)
• Enhancing Gradient Inversion Attacks in Federated Learning via Hierarchical Feature Optimization [56.95448807869383]
  Federated Learning (FL) has emerged as a compelling paradigm for privacy-preserving distributed machine learning.<n>Recent studies find that the gradients exchanged in the FL system are also vulnerable to privacy leakage.<n>We propose textbfGradient textbfInversion over textbfFeature textbfDomains (GIFD)
  arXiv  Detail & Related papers  (2026-04-01T14:32:15Z)
• Federated Vision Transformer with Adaptive Focal Loss for Medical Image Classification [8.412980809680471]
  deep learning models like Vision Transformer (ViT) typically require large datasets.<n> Federated learning (FL) addresses this challenge by enabling global model aggregation without data exchange.<n>This study proposes a FL framework leveraging a dynamic adaptive focal loss (DAFL) and a client-aware aggregation strategy for local training.
  arXiv  Detail & Related papers  (2026-02-02T04:47:33Z)
• Replacing Parameters with Preferences: Federated Alignment of Heterogeneous Vision-Language Models [63.70401095689976]
  We argue that replacing parameters with preferences represents a more scalable and privacy-preserving future.<n>We propose MoR, a federated alignment framework based on GRPO with Mixture-of-Rewards for heterogeneous VLMs.<n>MoR consistently outperforms federated alignment baselines in generalization, robustness, and cross-client adaptability.
  arXiv  Detail & Related papers  (2026-01-31T03:11:51Z)
• Communication-Efficient Federated Learning Based on Explanation-Guided Pruning for Remote Sensing Image Classification [2.725507329935916]
  Federated learning (FL) is a decentralized machine learning paradigm in which multiple clients collaboratively train a global model.<n>Due to the large volume of model updates required to be transmitted between clients and the central server, most FL systems are associated with high transfer costs.<n>We introduce an explanation-guided pruning strategy for communication-efficient FL in the context of RS image classification.
  arXiv  Detail & Related papers  (2025-01-20T13:59:41Z)
• FedRSClip: Federated Learning for Remote Sensing Scene Classification Using Vision-Language Models [23.830133838392964]
  We propose FedRSCLIP, the first federated learning framework for remote sensing image classification based on a VLM, specifically CLIP.<n>FedRSCLIP addresses the challenges of data heterogeneity and large-scale model transmission in federated environments by introducing Prompt Learning.<n>To validate the effectiveness of our proposed model, we construct a Fed-RSIC dataset based on three existing remote sensing image classification datasets.
  arXiv  Detail & Related papers  (2025-01-05T07:10:27Z)
• Transformer-based Federated Learning for Multi-Label Remote Sensing Image Classification [2.3255040478777755]
  We investigate the capability of state-of-the-art transformer architectures to address the challenges related to non-IID training data across various clients. The considered transformer architectures increase the ability with the cost of higher local training and aggregation complexities.
  arXiv  Detail & Related papers  (2024-05-24T10:13:49Z)
• Unlocking the Potential of Prompt-Tuning in Bridging Generalized and Personalized Federated Learning [49.72857433721424]
  Vision Transformers (ViT) and Visual Prompt Tuning (VPT) achieve state-of-the-art performance with improved efficiency in various computer vision tasks. We present a novel algorithm, SGPT, that integrates Generalized FL (GFL) and Personalized FL (PFL) approaches by employing a unique combination of both shared and group-specific prompts.
  arXiv  Detail & Related papers  (2023-10-27T17:22:09Z)
• Semi-Federated Learning: Convergence Analysis and Optimization of A Hybrid Learning Framework [70.83511997272457]
  We propose a semi-federated learning (SemiFL) paradigm to leverage both the base station (BS) and devices for a hybrid implementation of centralized learning (CL) and FL. We propose a two-stage algorithm to solve this intractable problem, in which we provide the closed-form solutions to the beamformers.
  arXiv  Detail & Related papers  (2023-10-04T03:32:39Z)
• FedPD: A Federated Learning Framework with Optimal Rates and Adaptivity to Non-IID Data [59.50904660420082]
  Federated Learning (FL) has become a popular paradigm for learning from distributed data. To effectively utilize data at different devices without moving them to the cloud, algorithms such as the Federated Averaging (FedAvg) have adopted a "computation then aggregation" (CTA) model.
  arXiv  Detail & Related papers  (2020-05-22T23:07:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.