Lifelong Learning for Fog Load Balancing: A Transfer Learning Approach
- URL: http://arxiv.org/abs/2310.05187v1
- Date: Sun, 8 Oct 2023 14:49:33 GMT
- Title: Lifelong Learning for Fog Load Balancing: A Transfer Learning Approach
- Authors: Maad Ebrahim, Abdelhakim Senhaji Hafid, Mohamed Riduan Abid
- Abstract summary: We improve the performance of privacy-aware Reinforcement Learning (RL) agents that optimize the execution delay of IoT applications by minimizing the waiting delay.
We propose a lifelong learning framework for these agents, where lightweight inference models are used during deployment to minimize action delay and only retrained in case of significant environmental changes.
- Score: 0.7366405857677226
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Fog computing emerged as a promising paradigm to address the challenges of
processing and managing data generated by the Internet of Things (IoT). Load
balancing (LB) plays a crucial role in Fog computing environments to optimize
the overall system performance. It requires efficient resource allocation to
improve resource utilization, minimize latency, and enhance the quality of
service for end-users. In this work, we improve the performance of
privacy-aware Reinforcement Learning (RL) agents that optimize the execution
delay of IoT applications by minimizing the waiting delay. To maintain privacy,
these agents optimize the waiting delay by minimizing the change in the number
of queued requests in the whole system, i.e., without explicitly observing the
actual number of requests that are queued in each Fog node nor observing the
compute resource capabilities of those nodes. Besides improving the performance
of these agents, we propose in this paper a lifelong learning framework for
these agents, where lightweight inference models are used during deployment to
minimize action delay and only retrained in case of significant environmental
changes. To improve the performance, minimize the training cost, and adapt the
agents to those changes, we explore the application of Transfer Learning (TL).
TL transfers the knowledge acquired from a source domain and applies it to a
target domain, enabling the reuse of learned policies and experiences. TL can
be also used to pre-train the agent in simulation before fine-tuning it in the
real environment; this significantly reduces failure probability compared to
learning from scratch in the real environment. To our knowledge, there are no
existing efforts in the literature that use TL to address lifelong learning for
RL-based Fog LB; this is one of the main obstacles in deploying RL LB solutions
in Fog systems.
Related papers
- Mind the Interference: Retaining Pre-trained Knowledge in Parameter Efficient Continual Learning of Vision-Language Models [79.28821338925947]
Domain-Class Incremental Learning is a realistic but challenging continual learning scenario.
To handle these diverse tasks, pre-trained Vision-Language Models (VLMs) are introduced for their strong generalizability.
This incurs a new problem: the knowledge encoded in the pre-trained VLMs may be disturbed when adapting to new tasks, compromising their inherent zero-shot ability.
Existing methods tackle it by tuning VLMs with knowledge distillation on extra datasets, which demands heavy overhead.
We propose the Distribution-aware Interference-free Knowledge Integration (DIKI) framework, retaining pre-trained knowledge of
arXiv Detail & Related papers (2024-07-07T12:19:37Z) - Fully Distributed Fog Load Balancing with Multi-Agent Reinforcement Learning [1.9643748953805935]
This paper proposes a fully distributed load-balancing solution with Multi-Agent Reinforcement Learning (MARL)
MARL agents use transfer learning for life-long self-adaptation to dynamic changes in the environment.
We analyze the impact of a realistic frequency to observe the state of the environment, unlike the unrealistic common assumption in the literature of having observations readily available in real-time for every required action.
arXiv Detail & Related papers (2024-05-15T23:44:06Z) - How Can LLM Guide RL? A Value-Based Approach [68.55316627400683]
Reinforcement learning (RL) has become the de facto standard practice for sequential decision-making problems by improving future acting policies with feedback.
Recent developments in large language models (LLMs) have showcased impressive capabilities in language understanding and generation, yet they fall short in exploration and self-improvement capabilities.
We develop an algorithm named LINVIT that incorporates LLM guidance as a regularization factor in value-based RL, leading to significant reductions in the amount of data needed for learning.
arXiv Detail & Related papers (2024-02-25T20:07:13Z) - Sparse Training for Federated Learning with Regularized Error Correction [9.852567834643292]
Federated Learning (FL) has attracted much interest due to the significant advantages it brings to training deep neural network (DNN) models.
FLARE presents a novel sparse training approach via accumulated pulling of the updated models with regularization on the embeddings in the FL process.
The performance of FLARE is validated through extensive experiments on diverse and complex models, achieving a remarkable sparsity level (10 times and more beyond the current state-of-the-art) along with significantly improved accuracy.
arXiv Detail & Related papers (2023-12-21T12:36:53Z) - Hybrid Reinforcement Learning for Optimizing Pump Sustainability in
Real-World Water Distribution Networks [55.591662978280894]
This article addresses the pump-scheduling optimization problem to enhance real-time control of real-world water distribution networks (WDNs)
Our primary objectives are to adhere to physical operational constraints while reducing energy consumption and operational costs.
Traditional optimization techniques, such as evolution-based and genetic algorithms, often fall short due to their lack of convergence guarantees.
arXiv Detail & Related papers (2023-10-13T21:26:16Z) - Prompt-Tuning Decision Transformer with Preference Ranking [83.76329715043205]
We propose the Prompt-Tuning DT algorithm to address challenges by using trajectory segments as prompts to guide RL agents in acquiring environmental information.
Our approach involves randomly sampling a Gaussian distribution to fine-tune the elements of the prompt trajectory and using preference ranking function to find the optimization direction.
Our work contributes to the advancement of prompt-tuning approaches in RL, providing a promising direction for optimizing large RL agents for specific preference tasks.
arXiv Detail & Related papers (2023-05-16T17:49:04Z) - FIRE: A Failure-Adaptive Reinforcement Learning Framework for Edge
Computing Migrations [55.131858975133085]
FIRE is a framework that adapts to rare events by training a RL policy in an edge computing digital twin environment.
We propose ImRE, an importance sampling-based Q-learning algorithm, which samples rare events proportionally to their impact on the value function.
We show that FIRE reduces costs compared to vanilla RL and the greedy baseline in the event of failures.
arXiv Detail & Related papers (2022-09-28T19:49:39Z) - Understanding and Preventing Capacity Loss in Reinforcement Learning [28.52122927103544]
We identify a mechanism by which non-stationary prediction targets can prevent learning progress in deep RL agents.
Capacity loss occurs in a range of RL agents and environments, and is particularly damaging to performance in sparse-reward tasks.
arXiv Detail & Related papers (2022-04-20T15:55:15Z) - A Distributed Deep Reinforcement Learning Technique for Application
Placement in Edge and Fog Computing Environments [31.326505188936746]
Several Deep Reinforcement Learning (DRL)-based placement techniques have been proposed in fog/edge computing environments.
We propose an actor-critic-based distributed application placement technique, working based on the IMPortance weighted Actor-Learner Architectures (IMPALA)
arXiv Detail & Related papers (2021-10-24T11:25:03Z) - Few-shot Quality-Diversity Optimization [50.337225556491774]
Quality-Diversity (QD) optimization has been shown to be effective tools in dealing with deceptive minima and sparse rewards in Reinforcement Learning.
We show that, given examples from a task distribution, information about the paths taken by optimization in parameter space can be leveraged to build a prior population, which when used to initialize QD methods in unseen environments, allows for few-shot adaptation.
Experiments carried in both sparse and dense reward settings using robotic manipulation and navigation benchmarks show that it considerably reduces the number of generations that are required for QD optimization in these environments.
arXiv Detail & Related papers (2021-09-14T17:12:20Z) - Smart Scheduling based on Deep Reinforcement Learning for Cellular
Networks [18.04856086228028]
We propose a smart scheduling scheme based on deep reinforcement learning (DRL)
We provide implementation-friend designs, i.e., a scalable neural network design for the agent and a virtual environment training framework.
We show that the DRL-based smart scheduling outperforms the conventional scheduling method and can be adopted in practical systems.
arXiv Detail & Related papers (2021-03-22T02:09:16Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.