Intelligent Orchestration of Distributed Large Foundation Model Inference at the Edge

• URL: http://arxiv.org/abs/2504.03668v3
• Date: Sat, 12 Jul 2025 22:37:25 GMT
• Title: Intelligent Orchestration of Distributed Large Foundation Model Inference at the Edge
• Authors: Fernando Koch, Aladin Djuhera, Alecio Binotto,
• Abstract summary: Large Foundation Models (LFMs) promise to unlock new capabilities for next-generation Edge AI applications.<n>Current split inference strategies, which partition LFM layers across nodes, are not designed to adapt to fluctuating workloads.<n>We propose a novel adaptive split inference orchestration framework that elevates both the placement and partitioning of LFM layers to runtime-tunable variables.
• Score: 46.1232919707345
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large Foundation Models (LFMs), including multi-modal and generative models, promise to unlock new capabilities for next-generation Edge AI applications. However, performing inference with LFMs in resource-constrained and heterogeneous edge environments, such as Multi-access Edge Computing (MEC), presents significant challenges for workload orchestration due to time-varying network, compute, and storage conditions. In particular, current split inference strategies, which partition LFM layers across nodes, are not designed to adapt to fluctuating workloads, dynamic bandwidth conditions, or evolving privacy constraints in high-utilization MEC environments. In this work, we propose a novel adaptive split inference orchestration framework that elevates both the placement and partitioning of LFM layers to runtime-tunable variables. Specifically, our framework enables real-time, quality-of-service (QoS)-aware management of inference workloads by extending conventional orchestrators with three key services: (1) Capacity-aware workload distribution, which continuously profiles node resources and selects an optimal subset of MEC nodes; (2) Dynamic partition migration, which transparently relocates pre-cut LFM segments in response to changes in utilization or network conditions; (3) Real-time reconfiguration, which dynamically re-splits LFM layers to balance latency, throughput, and privacy. We formalize the joint placement-partitioning problem, outline a reference architecture and algorithmic workflow, and discuss applicability in representative smart city, V2X, and industrial edge scenarios.

Related papers

• ElasticMM: Efficient Multimodal LLMs Serving with Elastic Multimodal Parallelism [9.93378263858092]
  Multimodal large language models (MLLMs) handle images, videos, and audio by incorporating feature extractors and projection modules.<n>Current tightly coupled serving architectures struggle to distinguish between mixed request types.<n>We propose Elastic Multimodal Parallelism (EMP), a new serving paradigm that elastically adapts to resource heterogeneity.
  arXiv  Detail & Related papers  (2025-07-14T08:53:48Z)
• FindRec: Stein-Guided Entropic Flow for Multi-Modal Sequential Recommendation [50.438552588818]
  We propose textbfFindRec (textbfFlexible unified textbfinformation textbfdisentanglement for multi-modal sequential textbfRecommendation)<n>A Stein kernel-based Integrated Information Coordination Module (IICM) theoretically guarantees distribution consistency between multimodal features and ID streams.<n>A cross-modal expert routing mechanism that adaptively filters and combines multimodal features based on their contextual relevance.
  arXiv  Detail & Related papers  (2025-07-07T04:09:45Z)
• A Federated Splitting Framework for LLMs: Security, Efficiency, and Adaptability [15.194518946737801]
  We introduce FL-LLaMA, a secure, efficient, and adaptive federated split framework based on LLaMA2.<n>We employ client-batch and server-hierarchical strategies to achieve parallel training, along with attention-mask compression and KV cache mechanisms to accelerate inference.<n>Experiments on NLU, summarization and conversational QA tasks show that FL-LLaMA maintains performance comparable to centralized LLaMA2, and achieves up to 2x train speedups and 8x inference speedups.
  arXiv  Detail & Related papers  (2025-05-21T15:58:08Z)
• Token Communication-Driven Multimodal Large Models in Resource-Constrained Multiuser Networks [7.137830911253685]
  multimodal large models pose challenges for deploying intelligent applications at the wireless edge.<n>These constraints manifest as limited bandwidth, computational capacity, and stringent latency requirements.<n>We propose a token communication paradigm that facilitates decentralized proliferations across user devices and edge infrastructure.
  arXiv  Detail & Related papers  (2025-05-06T14:17:05Z)
• UniSTD: Towards Unified Spatio-Temporal Learning across Diverse Disciplines [64.84631333071728]
  We introduce bfUnistage, a unified Transformer-based framework fortemporal modeling. Our work demonstrates that a task-specific vision-text can build a generalizable model fortemporal learning. We also introduce a temporal module to incorporate temporal dynamics explicitly.
  arXiv  Detail & Related papers  (2025-03-26T17:33:23Z)
• DiffMoE: Dynamic Token Selection for Scalable Diffusion Transformers [86.5541501589166]
  DiffMoE is a batch-level global token pool that enables experts to access global token distributions during training. It achieves state-of-the-art performance among diffusion models on ImageNet benchmark. The effectiveness of our approach extends beyond class-conditional generation to more challenging tasks such as text-to-image generation.
  arXiv  Detail & Related papers  (2025-03-18T17:57:07Z)
• Multi-Objective Bayesian Optimization for Networked Black-Box Systems: A Path to Greener Profits and Smarter Designs [0.0]
  MOBONS is a novel Bayesian optimization-inspired algorithm that can efficiently optimize general function networks. We demonstrate the effectiveness of MOBONS through two case studies, including one related to sustainable process design.
  arXiv  Detail & Related papers  (2025-02-19T21:49:05Z)
• ModServe: Scalable and Resource-Efficient Large Multimodal Model Serving [19.388562622309838]
  Large multimodal models (LMMs) demonstrate impressive capabilities in understanding images, videos, and audio beyond text.<n>We present the first comprehensive systems analysis of two prominent LMM architectures, decoder-only and cross-attention, across six representative open-source models.<n>We propose ModServe, a modular LMM serving system that decouples stages for independent optimization and adaptive scaling.
  arXiv  Detail & Related papers  (2025-02-02T22:10:40Z)
• Read-ME: Refactorizing LLMs as Router-Decoupled Mixture of Experts with System Co-Design [59.00758127310582]
  We propose a novel framework Read-ME that transforms pre-trained dense LLMs into smaller MoE models. Our approach employs activation sparsity to extract experts. Read-ME outperforms other popular open-source dense models of similar scales.
  arXiv  Detail & Related papers  (2024-10-24T19:48:51Z)
• FusionLLM: A Decentralized LLM Training System on Geo-distributed GPUs with Adaptive Compression [55.992528247880685]
  Decentralized training faces significant challenges regarding system design and efficiency. We present FusionLLM, a decentralized training system designed and implemented for training large deep neural networks (DNNs) We show that our system and method can achieve 1.45 - 9.39x speedup compared to baseline methods while ensuring convergence.
  arXiv  Detail & Related papers  (2024-10-16T16:13:19Z)
• Dynamic D2D-Assisted Federated Learning over O-RAN: Performance Analysis, MAC Scheduler, and Asymmetric User Selection [13.408764465010687]
  Realworld wireless networks are susceptible to temporal variations of wireless channel capacity and users. We incorporate multi-granular dynamics (MSDs) into FL, including (M1) discrete wireless channel, by a set of events, called $mathscrD$Events, and (M2) dynamic datasets of users. We propose a hierarchical device-to-device (D2D)-assisted model training, (ii) dynamic control decisions through dedicated O-RAN MAC schedulers, and (iii) asymmetric user selection.
  arXiv  Detail & Related papers  (2024-04-09T14:03:04Z)
• Regularized Conditional Diffusion Model for Multi-Task Preference Alignment [43.86042557447689]
  Sequential decision-making is desired to align with human intents and exhibit versatility across various tasks. Previous methods formulate it as a conditional generation process, utilizing return-conditioned diffusion models to directly model trajectory distributions. In this work, we adopt multi-task preferences as a unified condition for both single- and multi-task decision-making.
  arXiv  Detail & Related papers  (2024-04-07T11:20:32Z)
• A Bayesian Framework of Deep Reinforcement Learning for Joint O-RAN/MEC Orchestration [12.914011030970814]
  Multi-access Edge Computing (MEC) can be implemented together with Open Radio Access Network (O-RAN) over commodity platforms to offer low-cost deployment. In this paper, a joint O-RAN/MEC orchestration using a Bayesian deep reinforcement learning (RL)-based framework is proposed.
  arXiv  Detail & Related papers  (2023-12-26T18:04:49Z)
• Ensemble Kalman Filtering Meets Gaussian Process SSM for Non-Mean-Field and Online Inference [47.460898983429374]
  We introduce an ensemble Kalman filter (EnKF) into the non-mean-field (NMF) variational inference framework to approximate the posterior distribution of the latent states. This novel marriage between EnKF and GPSSM not only eliminates the need for extensive parameterization in learning variational distributions, but also enables an interpretable, closed-form approximation of the evidence lower bound (ELBO) We demonstrate that the resulting EnKF-aided online algorithm embodies a principled objective function by ensuring data-fitting accuracy while incorporating model regularizations to mitigate overfitting.
  arXiv  Detail & Related papers  (2023-12-10T15:22:30Z)
• Orchestration of Emulator Assisted Mobile Edge Tuning for AI Foundation Models: A Multi-Agent Deep Reinforcement Learning Approach [10.47302625959368]
  We present a groundbreaking paradigm integrating Mobile Edge Computing with foundation models, specifically designed to enhance local task performance on user equipment (UE) Central to our approach is the innovative Emulator-Adapter architecture, segmenting the foundation model into two cohesive modules. We introduce an advanced resource allocation mechanism that is fine-tuned to the needs of the Emulator-Adapter structure in decentralized settings.
  arXiv  Detail & Related papers  (2023-10-26T15:47:51Z)
• Resource Allocation in Multicore Elastic Optical Networks: A Deep Reinforcement Learning Approach [47.187609203210705]
  A new environment is developed compatible with OpenAI's Gym. It processes the agent actions by considering the network state and physical-layer-related aspects. The best-performing agent achieves a four-times decrease in blocking probability.
  arXiv  Detail & Related papers  (2022-07-05T14:24:21Z)
• Slimmable Domain Adaptation [112.19652651687402]
  We introduce a simple framework, Slimmable Domain Adaptation, to improve cross-domain generalization with a weight-sharing model bank. Our framework surpasses other competing approaches by a very large margin on multiple benchmarks.
  arXiv  Detail & Related papers  (2022-06-14T06:28:04Z)
• Low-Latency Federated Learning over Wireless Channels with Differential Privacy [142.5983499872664]
  In federated learning (FL), model training is distributed over clients and local models are aggregated by a central server. In this paper, we aim to minimize FL training delay over wireless channels, constrained by overall training performance as well as each client's differential privacy (DP) requirement.
  arXiv  Detail & Related papers  (2021-06-20T13:51:18Z)
• Edge-assisted Democratized Learning Towards Federated Analytics [67.44078999945722]
  We show the hierarchical learning structure of the proposed edge-assisted democratized learning mechanism, namely Edge-DemLearn. We also validate Edge-DemLearn as a flexible model training mechanism to build a distributed control and aggregation methodology in regions.
  arXiv  Detail & Related papers  (2020-12-01T11:46:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.