SeBS-Flow: Benchmarking Serverless Cloud Function Workflows

• URL: http://arxiv.org/abs/2410.03480v2
• Date: Mon, 7 Oct 2024 16:28:39 GMT
• Title: SeBS-Flow: Benchmarking Serverless Cloud Function Workflows
• Authors: Larissa Schmid, Marcin Copik, Alexandru Calotoiu, Laurin Brandner, Anne Koziolek, Torsten Hoefler,
• Abstract summary: We propose the first serverless workflow benchmarking suite SeBS-Flow. SeBS-Flow includes six real-world application benchmarks and four microbenchmarks representing different computational patterns. We conduct comprehensive evaluations on three major cloud platforms, assessing performance, cost, scalability, and runtime deviations.
• Score: 51.4200085836966
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Serverless computing has emerged as a prominent paradigm, with a significant adoption rate among cloud customers. While this model offers advantages such as abstraction from the deployment and resource scheduling, it also poses limitations in handling complex use cases due to the restricted nature of individual functions. Serverless workflows address this limitation by orchestrating multiple functions into a cohesive application. However, existing serverless workflow platforms exhibit significant differences in their programming models and infrastructure, making fair and consistent performance evaluations difficult in practice. To address this gap, we propose the first serverless workflow benchmarking suite SeBS-Flow, providing a platform-agnostic workflow model that enables consistent benchmarking across various platforms. SeBS-Flow includes six real-world application benchmarks and four microbenchmarks representing different computational patterns. We conduct comprehensive evaluations on three major cloud platforms, assessing performance, cost, scalability, and runtime deviations. We make our benchmark suite open-source, enabling rigorous and comparable evaluations of serverless workflows over time.

Related papers

• On the Cost of Model-Serving Frameworks: An Experimental Evaluation [2.6232657671486983]
  Serving strategies are crucial for deploying and managing models in production environments effectively. These strategies ensure that models are available, scalable, reliable, and performant for real-world applications. We show that DL-specific frameworks (TensorFlow Serving and TorchServe) display significantly lower latencies than the three general-purpose ML frameworks.
  arXiv  Detail & Related papers  (2024-11-15T16:36:21Z)
• Benchmarking Agentic Workflow Generation [80.74757493266057]
  We introduce WorFBench, a unified workflow generation benchmark with multi-faceted scenarios and intricate graph workflow structures. We also present WorFEval, a systemic evaluation protocol utilizing subsequence and subgraph matching algorithms. We observe that the generated can enhance downstream tasks, enabling them to achieve superior performance with less time during inference.
  arXiv  Detail & Related papers  (2024-10-10T12:41:19Z)
• ShuffleBench: A Benchmark for Large-Scale Data Shuffling Operations with Distributed Stream Processing Frameworks [1.4374467687356276]
  This paper introduces ShuffleBench, a novel benchmark to evaluate the performance of modern stream processing frameworks. ShuffleBench is inspired by requirements for near real-time analytics of a large cloud observability platform. Our results show that Flink achieves the highest throughput while Hazelcast processes data streams with the lowest latency.
  arXiv  Detail & Related papers  (2024-03-07T15:06:24Z)
• Green AI: A Preliminary Empirical Study on Energy Consumption in DL Models Across Different Runtime Infrastructures [56.200335252600354]
  It is common practice to deploy pre-trained models on environments distinct from their native development settings. This led to the introduction of interchange formats such as ONNX, which includes its infrastructure, and ONNX, which work as standard formats.
  arXiv  Detail & Related papers  (2024-02-21T09:18:44Z)
• FuzzyFlow: Leveraging Dataflow To Find and Squash Program Optimization Bugs [92.47146416628965]
  FuzzyFlow is a fault localization and test case extraction framework designed to test program optimizations. We leverage dataflow program representations to capture a fully reproducible system state and area-of-effect for optimizations. To reduce testing time, we design an algorithm for minimizing test inputs, trading off memory for recomputation.
  arXiv  Detail & Related papers  (2023-06-28T13:00:17Z)
• Cheaply Evaluating Inference Efficiency Metrics for Autoregressive Transformer APIs [66.30706841821123]
  Large language models (LLMs) power many state-of-the-art systems in natural language processing. LLMs are extremely computationally expensive, even at inference time. We propose a new metric for comparing inference efficiency across models.
  arXiv  Detail & Related papers  (2023-05-03T21:51:42Z)
• Performance Modeling of Metric-Based Serverless Computing Platforms [5.089110111757978]
  The proposed performance model can help developers and providers predict the performance and cost of deployments with different configurations. We validate the applicability and accuracy of the proposed performance model by extensive real-world experimentation on Knative.
  arXiv  Detail & Related papers  (2022-02-23T00:39:01Z)
• SOLIS -- The MLOps journey from data acquisition to actionable insights [62.997667081978825]
  In this paper we present a unified deployment pipeline and freedom-to-operate approach that supports all requirements while using basic cross-platform tensor framework and script language engines. This approach however does not supply the needed procedures and pipelines for the actual deployment of machine learning capabilities in real production grade systems.
  arXiv  Detail & Related papers  (2021-12-22T14:45:37Z)
• Reproducible and Portable Big Data Analytics in the Cloud [4.948702463455218]
  There are two main difficulties in reproducing big data applications in the cloud. The first is how to automate end-to-end execution of big data analytics in the cloud. The second is an application developed for one cloud, such as AWS or Azure, is difficult to reproduce in another cloud.
  arXiv  Detail & Related papers  (2021-12-17T20:52:03Z)
• Benchmarking and Performance Modelling of MapReduce Communication Pattern [0.0]
  Models can be used to infer the performance of unseen applications and approximate their performance when an arbitrary dataset is used as input. Our approach is validated by running empirical experiments in two setups.
  arXiv  Detail & Related papers  (2020-05-23T21:52:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.