Incentivised Orchestrated Training Architecture (IOTA): A Technical Primer for Release

• URL: http://arxiv.org/abs/2507.17766v1
• Date: Wed, 16 Jul 2025 15:16:21 GMT
• Title: Incentivised Orchestrated Training Architecture (IOTA): A Technical Primer for Release
• Abstract summary: IOTA transforms SN9's previously isolated competitors into a single cooperating unit.<n> Validators measure each miner's contribution and allocate token emissions proportionally.<n>We use model-bottlenecks to cut communication bandwidths of activations by up to 128x.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In August 2024, Bittensor's Subnet 9 (SN9) demonstrated that a distributed network of incentivized, permissionless actors could each pretrain large language models (LLMs) ranging from 700 million to 14 billion parameters, while surpassing established baselines. While that work validated blockchain-based decentralized pretraining as viable, it contained core issues: (i) every miner had to fit an entire model locally, and (ii) "winner-takes-all" rewards encouraged model hoarding. Here we introduce IOTA (Incentivized Orchestrated Training Architecture), an architecture that addresses these limitations by transforming SN9's previously isolated competitors into a single cooperating unit that can scale arbitrarily while still rewarding each contributor fairly. Key preliminary results: (1) Data- and Pipeline-parallel SWARM architecture - An orchestrator distributes model layers across heterogeneous miners and streams activations between them, enabling model sizes to scale with the number of participants rather than being constrained by the VRAM of a single machine; (2) Granular, continuous incentives - Validators measure each miner's contribution and allocate token emissions proportionally; (3) Activation compression - We used model-bottlenecks to cut communication bandwidths of activations by up to 128x, vastly improving training speed; (4) Butterfly All-Reduce - Miners average disjoint parameter slices in O(1) bandwidth, offering linear scalability, redundancy and built-in collusion detection; (5) CLASP (Contribution Loss Assessment via Sampling of Pathways) - A fair attribution scheme assigns credit to miners proportional to their marginal utility and detects exploits, even when contributions are interdependent across the pipeline.

Related papers

• Agora: Collective and Permissionless Internet-Scale Pretraining of Large Language Models [50.733497395381164]
  Agora combines bandwidth-efficient pipeline-parallel model sharding over internet-grade links with multi-party, fault-tolerant collective operations.<n>Report culminates in the first demonstration of its kind: Pluralis-8B, an open, permissionless pretraining run of an 8.6B- parameter model on 500B tokens of FineWeb-Edu.
  arXiv  Detail & Related papers  (2026-07-14T23:32:18Z)
• Unextractable Protocol Models: Collaborative Training and Inference without Weight Materialization [58.14514930760722]
  We consider a decentralized setup in which the participants collaboratively train and serve a large neural network.<n>In this setup, we explore the possibility of unmaterializable weights, where a full weight set is never available to any one participant.<n>We introduce Unextractable Protocol Models (UPMs): a training and inference framework that leverages the sharded model setup.
  arXiv  Detail & Related papers  (2026-05-22T10:24:57Z)
• Privacy-Aware Split Inference with Speculative Decoding for Large Language Models over Wide-Area Networks [0.0]
  We present a practical system for privacy-aware large language model (LLM) inference that splits a transformer between a trusted local GPU and a cloud GPU.<n>Our system addresses the unique challenges of autoregressive LLM decoding over high-latency wide-area networks (WANs)<n>System generalizes to larger models with only 4.9 GB local VRAM and matching 7B throughput.
  arXiv  Detail & Related papers  (2026-02-18T14:13:08Z)
• Every Step Evolves: Scaling Reinforcement Learning for Trillion-Scale Thinking Model [100.86587937568832]
  Ring-1T is the first open-source, state-of-the-art thinking model with a trillion-scale parameter.<n>It features 1 trillion total parameters and activates approximately 50 billion per token.
  arXiv  Detail & Related papers  (2025-10-21T17:46:14Z)
• On-Chain Decentralized Learning and Cost-Effective Inference for DeFi Attack Mitigation [4.37137147573556]
  Billions of dollars are lost every year in DeFi platforms by transactions exploiting business logic or accounting vulnerabilities.<n>We present the first decentralized, fully on-chain learning framework that: (i) performs gas-prohibitive computation on Layer-2 to reduce cost, (ii) propagates verified model updates to Layer-1, and (iii) enables gas-bounded low-latency inference inside smart contracts.
  arXiv  Detail & Related papers  (2025-10-15T18:58:34Z)
• Sharing is Caring: Efficient LM Post-Training with Collective RL Experience Sharing [1.5280055693761307]
  Post-training language models (LMs) with reinforcement learning (RL) can enhance their reasoning capabilities without supervised fine-tuning.<n>We present Swarm sAmpling Policy Optimization (SAPO), a fully decentralized and asynchronous RL post-training algorithm.<n>We show SAPO achieved cumulative reward gains of up to 94% in controlled experiments.
  arXiv  Detail & Related papers  (2025-09-10T16:14:20Z)
• DiLoCoX: A Low-Communication Large-Scale Training Framework for Decentralized Cluster [7.597885871452736]
  We propose DiLoCoX, a low-communication large-scale decentralized cluster training framework.<n>It combines Pipeline Parallelism with Dual-Step-Delay Overlap of Communication and Local Training, and an Adaptive Gradient Compression Scheme.<n>We show that DiLoCoX can achieve a 357x speedup in distributed training while maintaining negligible degradation in model convergence.
  arXiv  Detail & Related papers  (2025-06-26T13:45:04Z)
• Incentivizing Permissionless Distributed Learning of LLMs [7.36110927499488]
  textitGauntlet can be applied to any synchronous distributed training scheme that relies on aggregating updates or pseudo-gradients.<n>We utilize an OpenSkill rating system to track competitiveness of pseudo-gradient scores across time.<n>Our live 1.2B run, which has paid out real-valued tokens to participants based on the value of their contributions, demonstrates the utility of our incentive system.
  arXiv  Detail & Related papers  (2025-05-27T19:11:22Z)
• Reviving Any-Subset Autoregressive Models with Principled Parallel Sampling and Speculative Decoding [55.2480439325792]
  In arbitrary-order language models, it is an open question how to sample tokens in parallel from the correct joint distribution.<n>We find that a different class of models, any-subset autoregressive models (AS-ARMs), holds the solution.<n>We show that AS-ARMs achieve state-of-the-art performance among sub-200M parameter models on infilling benchmark tasks, and nearly match the performance of models 50X larger on code generation.
  arXiv  Detail & Related papers  (2025-04-29T06:33:13Z)
• Decentralized Diffusion Models [53.89995588977048]
  Large-scale AI model training divides work across thousands of GPU, then synchronizes gradients across them at each step.<n>This incurs a significant network burden that only centralized, monolithic clusters can support.<n>We propose Decentralized Diffusion Models, a scalable framework for distributing diffusion model training across independent clusters.
  arXiv  Detail & Related papers  (2025-01-09T18:59:56Z)
• FusionLLM: A Decentralized LLM Training System on Geo-distributed GPUs with Adaptive Compression [55.992528247880685]
  Decentralized training faces significant challenges regarding system design and efficiency. We present FusionLLM, a decentralized training system designed and implemented for training large deep neural networks (DNNs) We show that our system and method can achieve 1.45 - 9.39x speedup compared to baseline methods while ensuring convergence.
  arXiv  Detail & Related papers  (2024-10-16T16:13:19Z)
• Unity is Power: Semi-Asynchronous Collaborative Training of Large-Scale Models with Structured Pruning in Resource-Limited Clients [21.59433932637253]
  In this work, we study to release the potential of massive heterogeneous weak computing power to collaboratively train large-scale models on dispersed datasets. We propose a novel semi-asynchronous collaborative training framework, namely $Cotext-S2P$ with data distribution-aware structured pruning and cross-block knowledge transfer mechanism. Experiments demonstrate that $Cotext-S2P$ improves accuracy by up to 8.8% and resource utilization by up to 1.2$times$ compared to state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-10-11T02:17:50Z)
• Heterogeneity-Aware Coordination for Federated Learning via Stitching Pre-trained blocks [5.621750660969172]
  Federated learning (FL) coordinates multiple devices to collaboratively train a shared model while preserving data privacy. We propose FedStitch, a hierarchical coordination framework for heterogeneous federated learning with pre-trained blocks.
  arXiv  Detail & Related papers  (2024-09-11T11:47:50Z)
• Just One Byte (per gradient): A Note on Low-Bandwidth Decentralized Language Model Finetuning Using Shared Randomness [86.61582747039053]
  Language model training in distributed settings is limited by the communication cost of exchanges. We extend recent work using shared randomness to perform distributed fine-tuning with low bandwidth.
  arXiv  Detail & Related papers  (2023-06-16T17:59:51Z)
• Deep Model Reassembly [60.6531819328247]
  We explore a novel knowledge-transfer task, termed as Deep Model Reassembly (DeRy) The goal of DeRy is to first dissect each model into distinctive building blocks, and then selectively reassemble the derived blocks to produce customized networks. We demonstrate that on ImageNet, the best reassemble model achieves 78.6% top-1 accuracy without fine-tuning.
  arXiv  Detail & Related papers  (2022-10-24T10:16:13Z)
• Decentralized Training of Foundation Models in Heterogeneous Environments [77.47261769795992]
  Training foundation models, such as GPT-3 and PaLM, can be extremely expensive. We present the first study of training large foundation models with model parallelism in a decentralized regime over a heterogeneous network.
  arXiv  Detail & Related papers  (2022-06-02T20:19:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.