USPLIT-VQA: U-Shaped Split Learning for Visual Question Answering with Contribution-Aware Weighted Aggregation

• URL: http://arxiv.org/abs/2609.12168v1
• Date: Thu, 10 Sep 2026 19:56:27 GMT
• Title: USPLIT-VQA: U-Shaped Split Learning for Visual Question Answering with Contribution-Aware Weighted Aggregation
• Abstract summary: We propose a U-shaped split learning framework for privacy-preserving VQA.<n>CAWA reduces the attacker's influence by over 98%, while experiments at higher corruption levels identify its limitations.
• Score: 0.30938904602244355
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Visual Question Answering (VQA) systems, jointly interpreting images and natural language queries, hold significant promise across many domains, yet the privacy-sensitive nature of user data creates a fundamental barrier. Centralized training requires access to all data, while federated learning requires each client to host the full model. We propose USPLIT-VQA, a U-shaped split learning framework for privacy-preserving VQA in which each client retains the initial layers and the classification head while the server hosts the computationally heavy intermediate layers, keeping raw inputs and labels on the client device. We further introduce Contribution-Aware Weighted Aggregation (CAWA), a gradientsimilarity-based client scoring mechanism designed to reduce the influence of malicious updates. Experiments on four VQA datasets (VQA-RAD, SLAKE, PathVQA, and VizWiz) with two backbones show accuracy gains over Federated Learning for the Custom model and reduced accuracy for BiomedCLIP under the evaluated fixed split, alongside client memory reductions of up to 5.8X and communication reductions of up to 10.8X. With one malicious client, CAWA reduces the attacker's influence by over 98%, while experiments at higher corruption levels identify its limitations. Reconstruction experiments further show lower inversion quality under the evaluated attacks.

Related papers

• Harnessing Image Question Dependence for Better VLM Test-time Reinforcement Learning [66.13367818892503]
  We propose TTIQ, a test-time reinforcement learning framework that harnesses image-question dependence for better vlm adaptation.<n> TTIQ teacher-forces each sampled response under the original image-question pair and its image- and question-ablated variants.<n>It combines image and question dependence with confidence to construct a response-level reward that favors jointly grounded responses.
  arXiv  Detail & Related papers  (2026-09-09T22:54:15Z)
• Causal Unlearning in Collaborative Optimization: Exact and Approximate Influence Reversal under Adversarial Contributions [2.5565719921628443]
  Federated learning systems must support data deletion requests to comply with privacy regulations.<n>We present HF-KCU, a method that removes a client's contribution by approximating the influence through conjugate iterations in Krylov subspaces.<n>We provide convergence guarantees showing that the Krylov approximation error decreases as O((k 1/2-1)/(k1/2+1) where k is the Hessian condition number.
  arXiv  Detail & Related papers  (2026-05-19T18:00:39Z)
• FERA: Uncertainty-Aware Federated Reasoning for Large Language Models [60.52562148874846]
  We study federated reasoning, where a server improves multi-step reasoning by coordinating with heterogeneous clients holding private demonstrations.<n>The key challenge is that client reliability is query-dependent, while the server cannot inspect client data to determine which contributions are trustworthy.<n>We propose Uncertainty-Aware Federated Reasoning (FERA), a training-free framework based on iterative server-client co-refinement.
  arXiv  Detail & Related papers  (2026-05-11T07:04:51Z)
• ZORRO: Zero-Knowledge Robustness and Privacy for Split Learning (Full Version) [58.595691399741646]
  Split Learning (SL) is a distributed learning approach that enables resource-constrained clients to collaboratively train deep neural networks (DNNs)<n>This setup enables SL to leverage server capacities without sharing data, making it highly effective in resource-constrained environments dealing with sensitive data.<n>We present ZORRO, a private, verifiable, and robust SL defense scheme.
  arXiv  Detail & Related papers  (2025-09-11T18:44:09Z)
• Strategic Incentivization for Locally Differentially Private Federated Learning [12.107007441973938]
  In Federated Learning (FL), multiple clients jointly train a machine learning model by sharing gradient information, instead of raw data, with a server over multiple rounds.<n>In Local Differential Privacy (LDP), clients add a selective amount of noise to the gradients before sending the same to the server.<n>In this paper, we model this privacy-accuracy trade-off as a game, where the sever incentivizes the clients to add a lower degree of noise for achieving higher accuracy.
  arXiv  Detail & Related papers  (2025-08-10T01:57:47Z)
• Client-side Gradient Inversion Against Federated Learning from Poisoning [59.74484221875662]
  Federated Learning (FL) enables distributed participants to train a global model without sharing data directly to a central server. Recent studies have revealed that FL is vulnerable to gradient inversion attack (GIA), which aims to reconstruct the original training samples. We propose Client-side poisoning Gradient Inversion (CGI), which is a novel attack method that can be launched from clients.
  arXiv  Detail & Related papers  (2023-09-14T03:48:27Z)
• Client-specific Property Inference against Secure Aggregation in Federated Learning [52.8564467292226]
  Federated learning has become a widely used paradigm for collaboratively training a common model among different participants. Many attacks have shown that it is still possible to infer sensitive information such as membership, property, or outright reconstruction of participant data. We show that simple linear models can effectively capture client-specific properties only from the aggregated model updates.
  arXiv  Detail & Related papers  (2023-03-07T14:11:01Z)
• Subspace based Federated Unlearning [75.90552823500633]
  Federated unlearning (FL) aims to remove a specified target client's contribution in FL to satisfy the user's right to be forgotten. Most existing federated unlearning algorithms require the server to store the history of the parameter updates. We propose a simple-yet-effective subspace based federated unlearning method, dubbed SFU, that lets the global model perform gradient ascent.
  arXiv  Detail & Related papers  (2023-02-24T04:29:44Z)
• Stochastic Coded Federated Learning with Convergence and Privacy Guarantees [8.2189389638822]
  Federated learning (FL) has attracted much attention as a privacy-preserving distributed machine learning framework. This paper proposes a coded federated learning framework, namely coded federated learning (SCFL) to mitigate the straggler issue. We characterize the privacy guarantee by the mutual information differential privacy (MI-DP) and analyze the convergence performance in federated learning.
  arXiv  Detail & Related papers  (2022-01-25T04:43:29Z)
• Game of Gradients: Mitigating Irrelevant Clients in Federated Learning [3.2095659532757916]
  Federated learning (FL) deals with multiple clients participating in collaborative training of a machine learning model under the orchestration of a central server. In this setup, each client's data is private to itself and is not transferable to other clients or the server. We refer to these problems as Federated Relevant Client Selection (FRCS)
  arXiv  Detail & Related papers  (2021-10-23T16:34:42Z)
• Contrast and Classify: Training Robust VQA Models [60.80627814762071]
  We propose a novel training paradigm (ConClaT) that optimize both cross-entropy and contrastive losses. We find that optimizing both losses -- either alternately or jointly -- is key to effective training.
  arXiv  Detail & Related papers  (2020-10-13T00:23:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.