Unveiling the Depth-Performance Dilemma in Split-Federated Fine-tuning of LLMs

• URL: http://arxiv.org/abs/2608.22188v1
• Date: Sun, 23 Aug 2026 02:53:20 GMT
• Title: Unveiling the Depth-Performance Dilemma in Split-Federated Fine-tuning of LLMs
• Abstract summary: Split Federated Fine-tuning (SFF) is a promising paradigm for scaling Large Language Models (LLMs)<n>We identify and characterize the Depth-Performance Dilemma: the regime that maximizes system efficiency is precisely where fine-tuning quality collapses.
• Score: 0.8602553195689512
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Split Federated Fine-tuning (SFF) is a promising paradigm for scaling Large Language Models (LLMs) by partitioning model depth between resource-constrained clients and a centralized server. While system incentives for throughput and privacy favor deep partitions, the impact of such configurations on model utility remains poorly understood. In this work, we identify and characterize the Depth-Performance Dilemma: the regime that maximizes system efficiency is precisely where fine-tuning quality collapses. Through a comprehensive audit across four model scales (GPT-2 to Llama-3-8B) and diverse benchmarks, we demonstrate that deeper partitions provide monotonic gains in throughput and privacy at the cost of catastrophic performance plateaus. We evaluate a suite of state-of-the-art federated adapter aggregation methods including AVG, STACK, SVD, and FREEZE, revealing that while these techniques are effective in standard Federated Learning, they fail to mitigate the artifacts unique to split architectures. Finally, we provide a mechanistic diagnosis for this failure, tracing the collapse to the near-isometric topology of Transformers, which allows aggregation noise to propagate without attenuation until it triggers Attention Collapse in the server partition. Our findings challenge the prevailing assumption that partition depth is a utility-neutral tuning knob and provide a structural foundation for stable distributed LLM fine-tuning.

Related papers

• Federated Foundation Models Fine-Tuning with Heterogeneous Compressed Clients [22.724544289853558]
  FedSLM is a parameter-centric framework for federated fine-tuning with heterogeneous compressed clients.<n>We provide theoretical guarantees for adapter-level aggregation, subspace-alignment bounds for cross-group fusion, and a characterization of how the confidence loss mitigates weak-supervision noise.
  arXiv  Detail & Related papers  (2026-07-31T06:45:56Z)
• Group-wise Supervision with Focal-Dice Loss for Long-Tailed Indoor Semantic Occupancy Prediction [9.71287451024712]
  Group-UFD Occ is based on hierarchical semantic supervision and synergistic loss optimization.<n>We conduct experiments on the large-scale EmbodiedScan dataset.<n>Results demonstrate that our method yields a relative improvement of 11.38% over the baseline.
  arXiv  Detail & Related papers  (2026-07-31T01:32:45Z)
• Deep Slice Interpolation for Reducing Through-Plane Anisotropy and Noise in Head CT [0.0]
  We present a deep learning system that synthesizes intermediate CT slices from pairs of neighboring axial slices.<n>The system improves three-dimensional visualization while simultaneously producing inherently denoised outputs.
  arXiv  Detail & Related papers  (2026-06-08T10:45:51Z)
• SIGMA: Bridging Structural and Distributional Gaps for Vision Foundation Model Adaptation [5.145184780729273]
  Vision Foundation Models (VFMs) have demonstrated impressive representational capabilities.<n>textbfScale-textbfIntegrated textbfGlobal textbfModulation textbfAdapter (textbfAdapter)
  arXiv  Detail & Related papers  (2026-05-27T03:17:46Z)
• Enhancing Gradient Inversion Attacks in Federated Learning via Hierarchical Feature Optimization [56.95448807869383]
  Federated Learning (FL) has emerged as a compelling paradigm for privacy-preserving distributed machine learning.<n>Recent studies find that the gradients exchanged in the FL system are also vulnerable to privacy leakage.<n>We propose textbfGradient textbfInversion over textbfFeature textbfDomains (GIFD)
  arXiv  Detail & Related papers  (2026-04-01T14:32:15Z)
• Byzantine-Robust and Differentially Private Federated Optimization under Weaker Assumptions [89.52532304099522]
  Federated Learning (FL) enables clients to collaboratively train a shared model without centralizing their raw data, offering an inherent level of privacy.<n> gradients and model updates can still leak sensitive information, while malicious servers may mount adversarial attacks such as Byzantine manipulation.<n>These vulnerabilities highlight the need to address differential privacy (DP) and Byzantine robustness within a unified framework.<n>We propose Byz-Clip21-SGD2M, a new algorithm that integrates robust aggregation with double momentum and carefully designed clipping.
  arXiv  Detail & Related papers  (2026-03-24T17:39:09Z)
• A Secure and Private Distributed Bayesian Federated Learning Design [56.92336577799572]
  Distributed Federated Learning (DFL) enables decentralized model training across large-scale systems without a central parameter server.<n>DFL faces three critical challenges: privacy leakage from honest-but-curious neighbors, slow convergence due to the lack of central coordination, and vulnerability to Byzantine adversaries aiming to degrade model accuracy.<n>We propose a novel DFL framework that integrates Byzantine robustness, privacy preservation, and convergence acceleration.
  arXiv  Detail & Related papers  (2026-02-23T16:12:02Z)
• OmniVL-Guard: Towards Unified Vision-Language Forgery Detection and Grounding via Balanced RL [63.388513841293616]
  Existing forgery detection methods fail to handle the interleaved text, images, and videos prevalent in real-world misinformation.<n>To bridge this gap, this paper targets to develop a unified framework for omnibus vision-language forgery detection and grounding.<n>We propose textbf OmniVL-Guard, a balanced reinforcement learning framework for omnibus vision-language forgery detection and grounding.
  arXiv  Detail & Related papers  (2026-02-11T09:41:36Z)
• Why Does the LLM Stop Computing: An Empirical Study of User-Reported Failures in Open-Source LLMs [50.075587392477935]
  We conduct the first large-scale empirical study of 705 real-world failures from the open-source DeepSeek, Llama, and Qwen ecosystems.<n>Our analysis reveals a paradigm shift: white-box orchestration relocates the reliability bottleneck from model algorithmic defects to the systemic fragility of the deployment stack.
  arXiv  Detail & Related papers  (2026-01-20T06:42:56Z)
• GIFD: A Generative Gradient Inversion Method with Feature Domain Optimization [52.55628139825667]
  Federated Learning (FL) has emerged as a promising distributed machine learning framework to preserve clients' privacy. Recent studies find that an attacker can invert the shared gradients and recover sensitive data against an FL system by leveraging pre-trained generative adversarial networks (GAN) as prior knowledge. We propose textbfGradient textbfInversion over textbfFeature textbfDomains (GIFD), which disassembles the GAN model and searches the feature domains of the intermediate layers.
  arXiv  Detail & Related papers  (2023-08-09T04:34:21Z)
• Simple and Effective Prevention of Mode Collapse in Deep One-Class Classification [93.2334223970488]
  We propose two regularizers to prevent hypersphere collapse in deep SVDD. The first regularizer is based on injecting random noise via the standard cross-entropy loss. The second regularizer penalizes the minibatch variance when it becomes too small.
  arXiv  Detail & Related papers  (2020-01-24T03:44:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.