Bauplan: zero-copy, scale-up FaaS for data pipelines

• URL: http://arxiv.org/abs/2410.17465v1
• Date: Tue, 22 Oct 2024 22:49:01 GMT
• Title: Bauplan: zero-copy, scale-up FaaS for data pipelines
• Authors: Jacopo Tagliabue, Tyler Caraza-Harter, Ciro Greco,
• Abstract summary: bauplan is a novel F programming model and serverless runtime designed for data practitioners. bauplan enables users to declaratively define functional Directed Acyclic Graphs (DAGs) along with their runtime environments. We show that bauplan both better performance and a superior developer experience for data workloads by making trade-off of reducing generality in favor of data-awareness.
• Score: 4.6797109107617105
• License:
• Abstract: Chaining functions for longer workloads is a key use case for FaaS platforms in data applications. However, modern data pipelines differ significantly from typical serverless use cases (e.g., webhooks and microservices); this makes it difficult to retrofit existing pipeline frameworks due to structural constraints. In this paper, we describe these limitations in detail and introduce bauplan, a novel FaaS programming model and serverless runtime designed for data practitioners. bauplan enables users to declaratively define functional Directed Acyclic Graphs (DAGs) along with their runtime environments, which are then efficiently executed on cloud-based workers. We show that bauplan achieves both better performance and a superior developer experience for data workloads by making the trade-off of reducing generality in favor of data-awareness

Related papers

• SeBS-Flow: Benchmarking Serverless Cloud Function Workflows [51.4200085836966]
  We propose the first serverless workflow benchmarking suite SeBS-Flow. SeBS-Flow includes six real-world application benchmarks and four microbenchmarks representing different computational patterns. We conduct comprehensive evaluations on three major cloud platforms, assessing performance, cost, scalability, and runtime deviations.
  arXiv  Detail & Related papers  (2024-10-04T14:52:18Z)
• ToolACE: Winning the Points of LLM Function Calling [139.07157814653638]
  ToolACE is an automatic agentic pipeline designed to generate accurate, complex, and diverse tool-learning data. We demonstrate that models trained on our synthesized data, even with only 8B parameters, achieve state-of-the-art performance on the Berkeley Function-Calling Leaderboard.
  arXiv  Detail & Related papers  (2024-09-02T03:19:56Z)
• Reproducible data science over data lakes: replayable data pipelines with Bauplan and Nessie [5.259526087073711]
  We introduce a system designed to decouple compute from data management, by leveraging a cloud runtime alongside Nessie. We demonstrate its ability to offer time-travel and branching semantics on top of object storage, and offer full pipeline with a few CLI commands.
  arXiv  Detail & Related papers  (2024-04-21T14:53:33Z)
• In-depth Analysis On Parallel Processing Patterns for High-Performance Dataframes [0.0]
  We present a set of parallel processing patterns for distributed dataframe operators and the reference runtime implementation, Cylon. In this paper, we are expanding on the initial concept by introducing a cost model for evaluating the said patterns. We evaluate the performance of Cylon on the ORNL Summit supercomputer.
  arXiv  Detail & Related papers  (2023-07-03T23:11:03Z)
• Data-Copilot: Bridging Billions of Data and Humans with Autonomous Workflow [49.724842920942024]
  Industries such as finance, meteorology, and energy generate vast amounts of data daily. We propose Data-Copilot, a data analysis agent that autonomously performs querying, processing, and visualization of massive data tailored to diverse human requests.
  arXiv  Detail & Related papers  (2023-06-12T16:12:56Z)
• PDSketch: Integrated Planning Domain Programming and Learning [86.07442931141637]
  We present a new domain definition language, named PDSketch. It allows users to flexibly define high-level structures in the transition models. Details of the transition model will be filled in by trainable neural networks.
  arXiv  Detail & Related papers  (2023-03-09T18:54:12Z)
• Desbordante: from benchmarking suite to high-performance science-intensive data profiler (preprint) [36.537985747809245]
  Desbordante is a high-performance science-intensive data profiler with open source code. Unlike similar systems, it is built with emphasis on industrial application in a multi-user environment. It is efficient, resilient to crashes, and scalable.
  arXiv  Detail & Related papers  (2023-01-14T19:14:51Z)
• Object as a Service (OaaS): Enabling Object Abstraction in Serverless Clouds [2.0575037267955305]
  We propose a new abstraction level atop the function abstraction, known as Object as a Service (O) programming. O encapsulates the application data and function into the object abstraction and relieves the developers from resource and data management burdens. It also unlocks opportunities for built-in optimization features, such as software reusability, data locality, and caching.
  arXiv  Detail & Related papers  (2022-06-10T21:31:22Z)
• Pushing the Limits of Simple Pipelines for Few-Shot Learning: External Data and Fine-Tuning Make a Difference [74.80730361332711]
  Few-shot learning is an important and topical problem in computer vision. We show that a simple transformer-based pipeline yields surprisingly good performance on standard benchmarks.
  arXiv  Detail & Related papers  (2022-04-15T02:55:58Z)
• SOLIS -- The MLOps journey from data acquisition to actionable insights [62.997667081978825]
  In this paper we present a unified deployment pipeline and freedom-to-operate approach that supports all requirements while using basic cross-platform tensor framework and script language engines. This approach however does not supply the needed procedures and pipelines for the actual deployment of machine learning capabilities in real production grade systems.
  arXiv  Detail & Related papers  (2021-12-22T14:45:37Z)
• tf.data: A Machine Learning Data Processing Framework [0.4588028371034406]
  Training machine learning models requires feeding input data for models to ingest. We present tf.data, a framework for building and executing efficient input pipelines for machine learning jobs. We demonstrate that input pipeline performance is critical to the end-to-end training time of state-of-the-art machine learning models.
  arXiv  Detail & Related papers  (2021-01-28T17:16:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.