Unextractable Protocol Models: Collaborative Training and Inference without Weight Materialization

• URL: http://arxiv.org/abs/2605.23464v1
• Date: Fri, 22 May 2026 10:24:57 GMT
• Title: Unextractable Protocol Models: Collaborative Training and Inference without Weight Materialization
• Abstract summary: We consider a decentralized setup in which the participants collaboratively train and serve a large neural network.<n>In this setup, we explore the possibility of unmaterializable weights, where a full weight set is never available to any one participant.<n>We introduce Unextractable Protocol Models (UPMs): a training and inference framework that leverages the sharded model setup.
• Score: 58.14514930760722
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We consider a decentralized setup in which the participants collaboratively train and serve a large neural network, and where each participant only processes a subset of the model. In this setup, we explore the possibility of unmaterializable weights, where a full weight set is never available to any one participant. We introduce Unextractable Protocol Models (UPMs): a training and inference framework that leverages the sharded model setup to ensure model shards (i.e., subsets) held by participants are incompatible at different time steps. UPMs periodically inject time-varying, random, invertible transforms at participant boundaries; preserving the overall network function yet rendering cross-time assemblies incoherent. On Qwen-2.5-0.5B and Llama-3.2-1B, 10,000 transforms leave FP32 perplexity unchanged ($Δ$PPL $< 0.01$; Jensen-Shannon drift $< 4 \times 10^{-5}$), and we show how to control growth for lower precision datatypes. Applying a transform every 30s adds 3% latency, 0.1% bandwidth, and 10% GPU-memory overhead at inference, while training overhead falls to 1.6% time and $< 1$% memory. We consider several attacks, showing that the requirements of direct attacks are impractical and easy to defend against, and that gradient-based fine-tuning of stitched partitions consumes $\geq 60$% of the tokens required to train from scratch. By enabling models to be collaboratively trained yet not extracted, UPMs make it practical to embed programmatic incentive mechanisms in community-driven decentralized training.

Related papers

• PhaseShift: Topology-Aware Data Harmonization and Model Consolidation Across Signalized Intersections [7.890686084519088]
  PhaseShift is a topology-aware framework that harmonizes heterogeneous roadside trajectories into a shared actor-centric representation.<n>We evaluate five intersections in two Florida regions on balanced field data, 100k training windows and equal-sized test sets per site.
  arXiv  Detail & Related papers  (2026-08-26T01:22:49Z)
• FEAST: Federated Shared-Space Training for Resource-Heterogeneous Clients [2.1485350418225244]
  Federated learning must serve devices with varying computational capabilities.<n>Supernet training instead learns one fixed elastic model with differently sizedteroworks, then deploys a suitable one to each device.<n>When client inference budgets differ, parameters exclusive to high-costworks are reachable by fewer clients.<n>We propose FEAST, a shared-space training framework that counters this imbalance by jointly training multipleworks within each client's limit.
  arXiv  Detail & Related papers  (2026-08-10T08:10:59Z)
• Agora: Collective and Permissionless Internet-Scale Pretraining of Large Language Models [50.733497395381164]
  Agora combines bandwidth-efficient pipeline-parallel model sharding over internet-grade links with multi-party, fault-tolerant collective operations.<n>Report culminates in the first demonstration of its kind: Pluralis-8B, an open, permissionless pretraining run of an 8.6B- parameter model on 500B tokens of FineWeb-Edu.
  arXiv  Detail & Related papers  (2026-07-14T23:32:18Z)
• q0: Primitives for Hyper-Epoch Pretraining [0.5980755233352995]
  Pretraining a single model saturates within a few passes, long before the compute budget is exhausted.<n>We introduce hyper-epoch pretraining (q0), which turns a multi-epoch budget into a population of diverse models.<n>We show that q0 matches a strong 256-epoch ensemble baseline using only 56 epochs (4.6x fewer), or 67 epochs (3.8x fewer) when matched to the baseline's ensemble size.
  arXiv  Detail & Related papers  (2026-06-02T17:27:48Z)
• MARBLE: Multi-Aspect Reward Balance for Diffusion RL [71.6241143519038]
  Reinforcement learning has become the dominant approach for aligning diffusion models with human preferences.<n>Existing practice deal with multiple rewards by training one specialist model per reward.<n>We propose MARBLE, a gradient-space optimization framework that maintains independent advantage estimators for each reward.
  arXiv  Detail & Related papers  (2026-05-07T16:20:42Z)
• Nemotron Elastic: Towards Efficient Many-in-One Reasoning LLMs [80.72350166388601]
  Nemotron Elastic is a framework for building reasoning-oriented LLMs.<n>It embeds nested submodels within a single parent model.<n>Each of these submodels shares weights with the parent model and can be extracted zero-shot during deployment.
  arXiv  Detail & Related papers  (2025-11-20T18:59:21Z)
• Diffusion Language Models are Super Data Learners [61.721441061210896]
  When unique data is limited, diffusion language models (DLMs) consistently surpass autoregressive (AR) models by training for more epochs.<n>We attribute the gains to three compounding factors: (1) any-order modeling, (2) super-dense compute from iterative bidirectional denoising, and (3) built-in Monte Carlo augmentation.
  arXiv  Detail & Related papers  (2025-11-05T08:17:42Z)
• DreamPRM-1.5: Unlocking the Potential of Each Instance for Multimodal Process Reward Model Training [28.02129783121819]
  DreamPRM-1.5 is an instance-level reweighting framework that assigns an adaptive weight to every training example via bi-level optimization.<n>It attains 84.6 accuracy on the MMMU validation set, 31.3 accuracy on R-Bench-V and, when paired with a leading backbone, achieves first-place results on public multimodal reasoning leaderboards.
  arXiv  Detail & Related papers  (2025-09-05T23:42:01Z)
• Vertical Federated Learning with Missing Features During Training and Inference [37.44022318612869]
  We propose a vertical federated learning method for efficient training and inference of neural network-based models.<n>We show that our method achieves linear convergence to a neighborhood of the optimum even in the absence of missing features.
  arXiv  Detail & Related papers  (2024-10-29T22:09:31Z)
• FINE: Factorizing Knowledge for Initialization of Variable-sized Diffusion Models [35.40065954148091]
  FINE is a method based on the Learngene framework to initializing downstream networks leveraging pre-trained models. It decomposes pre-trained knowledge into the product of matrices (i.e., $U$, $Sigma$, and $V$), where $U$ and $V$ are shared across network blocks as learngenes'' It consistently outperforms direct pre-training, particularly for smaller models, achieving state-of-the-art results across variable model sizes.
  arXiv  Detail & Related papers  (2024-09-28T08:57:17Z)
• Just One Byte (per gradient): A Note on Low-Bandwidth Decentralized Language Model Finetuning Using Shared Randomness [86.61582747039053]
  Language model training in distributed settings is limited by the communication cost of exchanges. We extend recent work using shared randomness to perform distributed fine-tuning with low bandwidth.
  arXiv  Detail & Related papers  (2023-06-16T17:59:51Z)
• SuperFedNAS: Cost-Efficient Federated Neural Architecture Search for On-Device Inference [7.860030408676541]
  Neural Architecture Search (NAS) for Federated Learning (FL) is an emerging field. It automates the design and training of Deep Neural Networks (DNNs) when data cannot be centralized due to privacy, communication costs, or regulatory restrictions. Recent federated NAS methods not only reduce manual effort but also help achieve higher accuracy than traditional FL methods like FedAvg.
  arXiv  Detail & Related papers  (2023-01-26T00:17:10Z)
• Jigsaw Clustering for Unsupervised Visual Representation Learning [68.09280490213399]
  We propose a new jigsaw clustering pretext task in this paper. Our method makes use of information from both intra- and inter-images. It is even comparable to the contrastive learning methods when only half of training batches are used.
  arXiv  Detail & Related papers  (2021-04-01T08:09:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.