Traversal Learning Coordination For Lossless And Efficient Distributed Learning

• URL: http://arxiv.org/abs/2504.07471v1
• Date: Thu, 10 Apr 2025 05:48:57 GMT
• Title: Traversal Learning Coordination For Lossless And Efficient Distributed Learning
• Authors: Erdenebileg Batbaatar, Jeonggeol Kim, Yongcheol Kim, Young Yoon,
• Abstract summary: Traversal Learning (TL) is a novel approach designed to address the problem of decreased quality encountered in popular distributed learning (DL) paradigms.<n>TL adopts a unique strategy where the model traverses the nodes during forward propagation (FP) and performs backward propagation (BP) on the orchestrator.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this paper, we introduce Traversal Learning (TL), a novel approach designed to address the problem of decreased quality encountered in popular distributed learning (DL) paradigms such as Federated Learning (FL), Split Learning (SL), and SplitFed Learning (SFL). Traditional FL experiences from an accuracy drop during aggregation due to its averaging function, while SL and SFL face increased loss due to the independent gradient updates on each split network. TL adopts a unique strategy where the model traverses the nodes during forward propagation (FP) and performs backward propagation (BP) on the orchestrator, effectively implementing centralized learning (CL) principles within a distributed environment. The orchestrator is tasked with generating virtual batches and planning the sequential node visits of the model during FP, aligning them with the ordered index of the data within these batches. We conducted experiments on six datasets representing diverse characteristics across various domains. Our evaluation demonstrates that TL is on par with classic CL approaches in terms of accurate inference, thereby offering a viable and robust solution for DL tasks. TL outperformed other DL methods and improved accuracy by 7.85% for independent and identically distributed (IID) datasets, macro F1-score by 1.06% for non-IID datasets, accuracy by 2.60% for text classification, and AUC by 3.88% and 4.54% for medical and financial datasets, respectively. By effectively preserving data privacy while maintaining performance, TL represents a significant advancement in DL methodologies.

Related papers

• Diversity-Driven Learning: Tackling Spurious Correlations and Data Heterogeneity in Federated Models [21.672445835824053]
  Federated Learning (FL) enables decentralized training of machine learning models on distributed data. In real-world FL settings, client data is often non-identically distributed and imbalanced. We propose FedDiverse, a novel client selection algorithm in FL which is designed to manage and leverage data heterogeneity.
  arXiv  Detail & Related papers  (2025-04-15T14:20:42Z)
• Over-the-Air Fair Federated Learning via Multi-Objective Optimization [52.295563400314094]
  We propose an over-the-air fair federated learning algorithm (OTA-FFL) to train fair FL models.<n>Experiments demonstrate the superiority of OTA-FFL in achieving fairness and robust performance.
  arXiv  Detail & Related papers  (2025-01-06T21:16:51Z)
• TPFL: Tsetlin-Personalized Federated Learning with Confidence-Based Clustering [0.0]
  We propose a novel approach called Tsetlin-Personalized Federated Learning. In this way, models are grouped into clusters based on their confidence towards a specific class. Clients share only what they are confident about, resulting in the elimination of wrongful weight aggregation. Results demonstrated that TPFL performance better than baseline methods with 98.94% accuracy on MNIST, 98.52% accuracy on FashionMNIST and 91.16% accuracy on FEMNIST dataset.
  arXiv  Detail & Related papers  (2024-09-16T15:27:35Z)
• Accelerating Large Language Model Pretraining via LFR Pedagogy: Learn, Focus, and Review [50.78587571704713]
  Learn-Focus-Review (LFR) is a dynamic training approach that adapts to the model's learning progress. LFR tracks the model's learning performance across data blocks (sequences of tokens) and prioritizes revisiting challenging regions of the dataset. Compared to baseline models trained on the full datasets, LFR consistently achieved lower perplexity and higher accuracy.
  arXiv  Detail & Related papers  (2024-09-10T00:59:18Z)
• Multi-level Personalized Federated Learning on Heterogeneous and Long-Tailed Data [10.64629029156029]
  We introduce an innovative personalized Federated Learning framework, Multi-level Personalized Federated Learning (MuPFL) MuPFL integrates three pivotal modules: Biased Activation Value Dropout (BAVD), Adaptive Cluster-based Model Update (ACMU) and Prior Knowledge-assisted Fine-tuning (PKCF) Experiments on diverse real-world datasets show that MuPFL consistently outperforms state-of-the-art baselines, even under extreme non-i.i.d. and long-tail conditions.
  arXiv  Detail & Related papers  (2024-05-10T11:52:53Z)
• Decoupled Federated Learning on Long-Tailed and Non-IID data with Feature Statistics [20.781607752797445]
  We propose a two-stage Decoupled Federated learning framework using Feature Statistics (DFL-FS) In the first stage, the server estimates the client's class coverage distributions through masked local feature statistics clustering. In the second stage, DFL-FS employs federated feature regeneration based on global feature statistics to enhance the model's adaptability to long-tailed data distributions.
  arXiv  Detail & Related papers  (2024-03-13T09:24:59Z)
• FLASH: Federated Learning Across Simultaneous Heterogeneities [54.80435317208111]
  FLASH(Federated Learning Across Simultaneous Heterogeneities) is a lightweight and flexible client selection algorithm. It outperforms state-of-the-art FL frameworks under extensive sources of Heterogeneities. It achieves substantial and consistent improvements over state-of-the-art baselines.
  arXiv  Detail & Related papers  (2024-02-13T20:04:39Z)
• FedLALR: Client-Specific Adaptive Learning Rates Achieve Linear Speedup for Non-IID Data [54.81695390763957]
  Federated learning is an emerging distributed machine learning method. We propose a heterogeneous local variant of AMSGrad, named FedLALR, in which each client adjusts its learning rate. We show that our client-specified auto-tuned learning rate scheduling can converge and achieve linear speedup with respect to the number of clients.
  arXiv  Detail & Related papers  (2023-09-18T12:35:05Z)
• From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning [52.257422715393574]
  We introduce a self-guided methodology for Large Language Models (LLMs) to autonomously discern and select cherry samples from open-source datasets. Our key innovation, the Instruction-Following Difficulty (IFD) metric, emerges as a pivotal metric to identify discrepancies between a model's expected responses and its intrinsic generation capability.
  arXiv  Detail & Related papers  (2023-08-23T09:45:29Z)
• Vertical Semi-Federated Learning for Efficient Online Advertising [50.18284051956359]
  Semi-VFL (Vertical Semi-Federated Learning) is proposed to achieve a practical industry application fashion for VFL. We build an inference-efficient single-party student model applicable to the whole sample space. New representation distillation methods are designed to extract cross-party feature correlations for both the overlapped and non-overlapped data.
  arXiv  Detail & Related papers  (2022-09-30T17:59:27Z)
• Heterogeneous Federated Learning via Grouped Sequential-to-Parallel Training [60.892342868936865]
  Federated learning (FL) is a rapidly growing privacy-preserving collaborative machine learning paradigm. We propose a data heterogeneous-robust FL approach, FedGSP, to address this challenge. We show that FedGSP improves the accuracy by 3.7% on average compared with seven state-of-the-art approaches.
  arXiv  Detail & Related papers  (2022-01-31T03:15:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.