Related papers: RAPTOR: Role-Aware Private Training for Mixture-of-Experts

RAPTOR: Role-Aware Private Training for Mixture-of-Experts

URL: http://arxiv.org/abs/2609.05770v1
Date: Fri, 04 Sep 2026 23:16:12 GMT
Title: RAPTOR: Role-Aware Private Training for Mixture-of-Experts
Abstract summary: RAPTOR is a Role-Aware Private Training framework.<n>RAPTOR alternates shared and expert optimization and targets each failure directly.<n> Experiments on Switch Transformer and OLMoE fine-tuning across GLUE tasks, and on DeepSeek-VL2-Tiny, show consistent gains over standard DP baselines.
Score: 68.5626338219096
License: http://creativecommons.org/licenses/by/4.0/
Abstract: Differentially private (DP) fine-tuning methods treat sparse Mixture-of-Experts (MoE) models as a single dense block, ignoring that shared layers see all data while experts only see routed records. We identify and formally characterize three resulting failure modes: global clipping suppresses expert gradients, batch-level normalization dilutes sparse expert updates, and fixed privacy noise degrades signal-to-noise ratio on low-load experts. We introduce RAPTOR - a Role-Aware Private Training framework, which alternates shared and expert optimization and targets each failure directly, using expert-specific clipping and noise together with a public expected-owner denominator and a count-independent update schedule that avoids conditioning on private, realized expert counts. We prove the resulting mechanism satisfies $(\varepsilon,δ)$-DP: because each record is assigned to exactly one owner expert, per-expert mechanisms within a layer compose in parallel, so updating all $E$ experts costs no more, in privacy terms, than updating one, with shared and expert streams composing sequentially across training. We further derive a bias-variance decomposition of the public-denominator estimator showing its bias grows predictably with routing imbalance, yielding a privacy-free rule for selecting which layer to protect from routing entropy measured on a small public corpus. Experiments on Switch Transformer and OLMoE fine-tuning across GLUE tasks, and on DeepSeek-VL2-Tiny, show consistent gains over standard DP baselines across several privacy levels ($\varepsilon$), with the largest margins typically at the tightest budgets. Code and models are publicly available: https://github.com/leduckhai/RAPTOR

Related papers

Share First, Route What Remains: A Unified Framework for Token-Adaptive MoE Computation [49.8459545291965]
Mixture-of-experts (MoE) models have recently moved beyond routing a fixed number of complete experts.<n>We study this dependency by decomposing sparsely upcycled feed-forward experts into key-value channels.<n>We instantiate it in UniF-MoE, a unified framework for token-adaptive MoE computation.
arXiv Detail & Related papers (2026-08-11T02:40:58Z)
FedWeave: Rethinking the Unit of Specialization in Heterogeneous Federated MoE-LoRA [15.068543696107227]
Federated PEFT enables LLMs to collaboratively adapt to decentralized private data without sharing raw examples.<n>Existing methods typically specialize at client granularity, implicitly assuming task-coherent clients.<n>We propose FedWeave, a framework that adopts asymmetric aggregation, separating expert aggregation from router optimization.
arXiv Detail & Related papers (2026-07-29T08:47:58Z)
Adaptive Bayesian Online Learning via Expert Aggregation [1.8620637029128548]
We instantiate the framework in online conformal inference and Gaussian process regression.<n>Experiments show that the aggregate tracks strong experts without oracle expert selection.
arXiv Detail & Related papers (2026-07-22T14:59:09Z)
Multi-tier Differential Private Query Release [57.049084914202275]
Multiple analysts may issue the same query, while receiving answers generated under different privacy budgets.<n>Existing approaches for such multi-tier DP queries either incur excessive cumulative privacy loss or suffer from suboptimal utility.<n>We propose a framework for multi-tier DP query release that simultaneously bound the cumulative privacy loss by the maximum privacy budget among all queries.
arXiv Detail & Related papers (2026-06-14T02:10:22Z)
SHAPE: Coalition-Aware Expert Pruning for Sparse Mixture-of-Experts LLMs [4.035485260002141]
Sparse Mixture-of-Experts (MoE) models achieve strong quality with low per-token compute, yet their deployment is often limited by the memory wall.<n>We propose textbfSHAPE, a task-driven pruning framework that explicitly models emphintra-layer expert cooperation.
arXiv Detail & Related papers (2026-06-03T08:41:48Z)
When (and How) to Trust the Expert: Diagnosing Query-Time Expert-Guided Reinforcement Learning [4.913013713982676]
Continuous-control problems ship with a competent but suboptimal controller (a tuned PID, a hand-designed gait)<n>A growing family of methods uses such controllers as queryable experts during RL, but each method has been proposed in isolation, on a different benchmark, without imperfect-expert testing.<n>We harmonize the comparison on a shared SAC backbone, common HPO and evaluation protocols, 100/50 seeds per (env, method), and a degradation sweep over expert undertuning, action bias, and observation noise.
arXiv Detail & Related papers (2026-05-09T18:48:17Z)
Byzantine-Robust and Differentially Private Federated Optimization under Weaker Assumptions [89.52532304099522]
Federated Learning (FL) enables clients to collaboratively train a shared model without centralizing their raw data, offering an inherent level of privacy.<n> gradients and model updates can still leak sensitive information, while malicious servers may mount adversarial attacks such as Byzantine manipulation.<n>These vulnerabilities highlight the need to address differential privacy (DP) and Byzantine robustness within a unified framework.<n>We propose Byz-Clip21-SGD2M, a new algorithm that integrates robust aggregation with double momentum and carefully designed clipping.
arXiv Detail & Related papers (2026-03-24T17:39:09Z)
Smoothed Normalization for Efficient Distributed Private Optimization [54.197255548244705]
Federated learning enables machine learning models with privacy of participants.<n>There is no differentially private distributed method for training, non-feedback problems.<n>We introduce a new distributed algorithm $alpha$-$sf NormEC$ with provable convergence guarantees.
arXiv Detail & Related papers (2025-02-19T07:10:32Z)
Federated Online Prediction from Experts with Differential Privacy: Separations and Regret Speed-ups [6.925352969721467]
We study the problems of differentially private online prediction from experts against both adversaries and oblivious adversaries. With adversaries, we propose a Fed-DP-OPE-Stoch algorithm that achieves $sqrtm$-fold speed-up of the per-client regret. With oblivious adversaries, we establish non-trivial lower bounds indicating that collaboration among clients does not lead to regret speed-up.
arXiv Detail & Related papers (2024-09-27T18:43:24Z)
Uncertainty Voting Ensemble for Imbalanced Deep Regression [20.176217123752465]
In this paper, we introduce UVOTE, a method for learning from imbalanced data. We replace traditional regression losses with negative log-likelihood, which also predicts sample-wise aleatoric uncertainty. We show that UVOTE consistently outperforms the prior art, while at the same time producing better-calibrated uncertainty estimates.
arXiv Detail & Related papers (2023-05-24T14:12:21Z)
Understanding Clipping for Federated Learning: Convergence and Client-Level Differential Privacy [67.4471689755097]
This paper empirically demonstrates that the clipped FedAvg can perform surprisingly well even with substantial data heterogeneity. We provide the convergence analysis of a differential private (DP) FedAvg algorithm and highlight the relationship between clipping bias and the distribution of the clients' updates.
arXiv Detail & Related papers (2021-06-25T14:47:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.