Multi-objective Optimization of Clustering-based Scheduling for Multi-workflow On Clouds Considering Fairness

• URL: http://arxiv.org/abs/2205.11173v1
• Date: Mon, 23 May 2022 10:25:16 GMT
• Title: Multi-objective Optimization of Clustering-based Scheduling for Multi-workflow On Clouds Considering Fairness
• Authors: Feng Li, Wen Jun, Tan and Wentong, Cai
• Abstract summary: This paper defines a new multi-objective optimization model based on makespan, cost, and fairness, and then proposes a global clustering-based multi-workflow scheduling strategy for resource allocation. Experimental results show that the proposed approach performs better than the compared algorithms without significant compromise of the overall makespan and cost as well as individual fairness.
• Score: 4.021507306414546
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Distributed computing, such as cloud computing, provides promising platforms to execute multiple workflows. Workflow scheduling plays an important role in multi-workflow execution with multi-objective requirements. Although there exist many multi-objective scheduling algorithms, they focus mainly on optimizing makespan and cost for a single workflow. There is a limited research on multi-objective optimization for multi-workflow scheduling. Considering multi-workflow scheduling, there is an additional key objective to maintain the fairness of workflows using the resources. To address such issues, this paper first defines a new multi-objective optimization model based on makespan, cost, and fairness, and then proposes a global clustering-based multi-workflow scheduling strategy for resource allocation. Experimental results show that the proposed approach performs better than the compared algorithms without significant compromise of the overall makespan and cost as well as individual fairness, which can guide the simulation workflow scheduling on clouds.

Related papers

• Multi-Fidelity Bayesian Optimization With Across-Task Transferable Max-Value Entropy Search [36.14499894307206]
  This paper introduces a novel information-theoretic acquisition function that balances the need to acquire information about the current task with the goal of collecting information transferable to future tasks. Experimental results across synthetic and real-world examples reveal that the proposed provident acquisition strategy can significantly improve the optimization efficiency as soon as a sufficient number of tasks is processed.
  arXiv  Detail & Related papers  (2024-03-14T17:00:01Z)
• Scalarization for Multi-Task and Multi-Domain Learning at Scale [15.545810422759295]
  Training a single model on multiple input domains and/or output tasks allows for compressing information from multiple sources into a unified backbone. However, optimizing such networks is a challenge due to discrepancies between the different tasks or domains.
  arXiv  Detail & Related papers  (2023-10-13T07:31:04Z)
• Let the Flows Tell: Solving Graph Combinatorial Optimization Problems with GFlowNets [86.43523688236077]
  Combinatorial optimization (CO) problems are often NP-hard and out of reach for exact algorithms. GFlowNets have emerged as a powerful machinery to efficiently sample from composite unnormalized densities sequentially. In this paper, we design Markov decision processes (MDPs) for different problems and propose to train conditional GFlowNets to sample from the solution space.
  arXiv  Detail & Related papers  (2023-05-26T15:13:09Z)
• Semisoft Task Clustering for Multi-Task Learning [2.806911268410107]
  Multi-task learning (MTL) aims to improve the performance of multiple related prediction tasks by leveraging useful information from them. We propose a semisoft task clustering approach, which can simultaneously reveal the task clustering structure for both pure mixed tasks as well as select the relevant features. The experimental results based on synthetic and real-world datasets validate the effectiveness and efficiency of the proposed approach.
  arXiv  Detail & Related papers  (2022-11-28T07:23:56Z)
• Multi-Job Intelligent Scheduling with Cross-Device Federated Learning [65.69079337653994]
  Federated Learning (FL) enables collaborative global machine learning model training without sharing sensitive raw data. We propose a novel multi-job FL framework, which enables the training process of multiple jobs in parallel. We propose a novel intelligent scheduling approach based on multiple scheduling methods, including an original reinforcement learning-based scheduling method and an original Bayesian optimization-based scheduling method.
  arXiv  Detail & Related papers  (2022-11-24T06:17:40Z)
• Multi-Objective GFlowNets [59.16787189214784]
  We study the problem of generating diverse candidates in the context of Multi-Objective Optimization. In many applications of machine learning such as drug discovery and material design, the goal is to generate candidates which simultaneously optimize a set of potentially conflicting objectives. We propose Multi-Objective GFlowNets (MOGFNs), a novel method for generating diverse optimal solutions, based on GFlowNets.
  arXiv  Detail & Related papers  (2022-10-23T16:15:36Z)
• In Defense of the Unitary Scalarization for Deep Multi-Task Learning [121.76421174107463]
  We present a theoretical analysis suggesting that many specialized multi-tasks can be interpreted as forms of regularization. We show that, when coupled with standard regularization and stabilization techniques, unitary scalarization matches or improves upon the performance of complex multitasks.
  arXiv  Detail & Related papers  (2022-01-11T18:44:17Z)
• A Scalable Deep Reinforcement Learning Model for Online Scheduling Coflows of Multi-Stage Jobs for High Performance Computing [9.866286878494979]
  In multi-stage jobs, each job consists of multiple coflows and is represented by a Directed Acyclic Graph (DAG) In this paper, we propose a novel Pipelined-DAGNN to process the input and propose a novel coflow scheduling algorithm.
  arXiv  Detail & Related papers  (2021-12-21T09:36:55Z)
• Efficient Device Scheduling with Multi-Job Federated Learning [64.21733164243781]
  We propose a novel multi-job Federated Learning framework to enable the parallel training process of multiple jobs. We propose a reinforcement learning-based method and a Bayesian optimization-based method to schedule devices for multiple jobs while minimizing the cost. Our proposed approaches significantly outperform baseline approaches in terms of training time (up to 8.67 times faster) and accuracy (up to 44.6% higher)
  arXiv  Detail & Related papers  (2021-12-11T08:05:11Z)
• Resource Aware Multifidelity Active Learning for Efficient Optimization [0.8717253904965373]
  This paper introduces the Resource Aware Active Learning (RAAL) strategy to accelerate the optimization of black box functions. The RAAL strategy optimally seeds multiple points at each allowing for a major speed up of the optimization task.
  arXiv  Detail & Related papers  (2020-07-09T10:01:32Z)
• Dynamic Multi-Robot Task Allocation under Uncertainty and Temporal Constraints [52.58352707495122]
  We present a multi-robot allocation algorithm that decouples the key computational challenges of sequential decision-making under uncertainty and multi-agent coordination. We validate our results over a wide range of simulations on two distinct domains: multi-arm conveyor belt pick-and-place and multi-drone delivery dispatch in a city.
  arXiv  Detail & Related papers  (2020-05-27T01:10:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.