Multi-objective Optimization of Clustering-based Scheduling for
Multi-workflow On Clouds Considering Fairness
- URL: http://arxiv.org/abs/2205.11173v1
- Date: Mon, 23 May 2022 10:25:16 GMT
- Title: Multi-objective Optimization of Clustering-based Scheduling for
Multi-workflow On Clouds Considering Fairness
- Authors: Feng Li, Wen Jun, Tan and Wentong, Cai
- Abstract summary: This paper defines a new multi-objective optimization model based on makespan, cost, and fairness, and then proposes a global clustering-based multi-workflow scheduling strategy for resource allocation.
Experimental results show that the proposed approach performs better than the compared algorithms without significant compromise of the overall makespan and cost as well as individual fairness.
- Score: 4.021507306414546
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Distributed computing, such as cloud computing, provides promising platforms
to execute multiple workflows. Workflow scheduling plays an important role in
multi-workflow execution with multi-objective requirements. Although there
exist many multi-objective scheduling algorithms, they focus mainly on
optimizing makespan and cost for a single workflow. There is a limited research
on multi-objective optimization for multi-workflow scheduling. Considering
multi-workflow scheduling, there is an additional key objective to maintain the
fairness of workflows using the resources. To address such issues, this paper
first defines a new multi-objective optimization model based on makespan, cost,
and fairness, and then proposes a global clustering-based multi-workflow
scheduling strategy for resource allocation. Experimental results show that the
proposed approach performs better than the compared algorithms without
significant compromise of the overall makespan and cost as well as individual
fairness, which can guide the simulation workflow scheduling on clouds.
Related papers
- Benchmarking Agentic Workflow Generation [80.74757493266057]
We introduce WorFBench, a unified workflow generation benchmark with multi-faceted scenarios and intricate graph workflow structures.
We also present WorFEval, a systemic evaluation protocol utilizing subsequence and subgraph matching algorithms.
We observe that the generated can enhance downstream tasks, enabling them to achieve superior performance with less time during inference.
arXiv Detail & Related papers (2024-10-10T12:41:19Z) - QPO: Query-dependent Prompt Optimization via Multi-Loop Offline Reinforcement Learning [58.767866109043055]
We introduce Query-dependent Prompt Optimization (QPO), which iteratively fine-tune a small pretrained language model to generate optimal prompts tailored to the input queries.
We derive insights from offline prompting demonstration data, which already exists in large quantities as a by-product of benchmarking diverse prompts on open-sourced tasks.
Experiments on various LLM scales and diverse NLP and math tasks demonstrate the efficacy and cost-efficiency of our method in both zero-shot and few-shot scenarios.
arXiv Detail & Related papers (2024-08-20T03:06:48Z) - Scalarization for Multi-Task and Multi-Domain Learning at Scale [15.545810422759295]
Training a single model on multiple input domains and/or output tasks allows for compressing information from multiple sources into a unified backbone.
However, optimizing such networks is a challenge due to discrepancies between the different tasks or domains.
arXiv Detail & Related papers (2023-10-13T07:31:04Z) - Semisoft Task Clustering for Multi-Task Learning [2.806911268410107]
Multi-task learning (MTL) aims to improve the performance of multiple related prediction tasks by leveraging useful information from them.
We propose a semisoft task clustering approach, which can simultaneously reveal the task clustering structure for both pure mixed tasks as well as select the relevant features.
The experimental results based on synthetic and real-world datasets validate the effectiveness and efficiency of the proposed approach.
arXiv Detail & Related papers (2022-11-28T07:23:56Z) - Multi-Job Intelligent Scheduling with Cross-Device Federated Learning [65.69079337653994]
Federated Learning (FL) enables collaborative global machine learning model training without sharing sensitive raw data.
We propose a novel multi-job FL framework, which enables the training process of multiple jobs in parallel.
We propose a novel intelligent scheduling approach based on multiple scheduling methods, including an original reinforcement learning-based scheduling method and an original Bayesian optimization-based scheduling method.
arXiv Detail & Related papers (2022-11-24T06:17:40Z) - Multi-Objective GFlowNets [59.16787189214784]
We study the problem of generating diverse candidates in the context of Multi-Objective Optimization.
In many applications of machine learning such as drug discovery and material design, the goal is to generate candidates which simultaneously optimize a set of potentially conflicting objectives.
We propose Multi-Objective GFlowNets (MOGFNs), a novel method for generating diverse optimal solutions, based on GFlowNets.
arXiv Detail & Related papers (2022-10-23T16:15:36Z) - A Scalable Deep Reinforcement Learning Model for Online Scheduling
Coflows of Multi-Stage Jobs for High Performance Computing [9.866286878494979]
In multi-stage jobs, each job consists of multiple coflows and is represented by a Directed Acyclic Graph (DAG)
In this paper, we propose a novel Pipelined-DAGNN to process the input and propose a novel coflow scheduling algorithm.
arXiv Detail & Related papers (2021-12-21T09:36:55Z) - Efficient Device Scheduling with Multi-Job Federated Learning [64.21733164243781]
We propose a novel multi-job Federated Learning framework to enable the parallel training process of multiple jobs.
We propose a reinforcement learning-based method and a Bayesian optimization-based method to schedule devices for multiple jobs while minimizing the cost.
Our proposed approaches significantly outperform baseline approaches in terms of training time (up to 8.67 times faster) and accuracy (up to 44.6% higher)
arXiv Detail & Related papers (2021-12-11T08:05:11Z) - Resource Aware Multifidelity Active Learning for Efficient Optimization [0.8717253904965373]
This paper introduces the Resource Aware Active Learning (RAAL) strategy to accelerate the optimization of black box functions.
The RAAL strategy optimally seeds multiple points at each allowing for a major speed up of the optimization task.
arXiv Detail & Related papers (2020-07-09T10:01:32Z) - Dynamic Multi-Robot Task Allocation under Uncertainty and Temporal
Constraints [52.58352707495122]
We present a multi-robot allocation algorithm that decouples the key computational challenges of sequential decision-making under uncertainty and multi-agent coordination.
We validate our results over a wide range of simulations on two distinct domains: multi-arm conveyor belt pick-and-place and multi-drone delivery dispatch in a city.
arXiv Detail & Related papers (2020-05-27T01:10:41Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.