CodeReef: an open platform for portable MLOps, reusable automation actions and reproducible benchmarking

• URL: http://arxiv.org/abs/2001.07935v2
• Date: Mon, 27 Jan 2020 11:09:34 GMT
• Title: CodeReef: an open platform for portable MLOps, reusable automation actions and reproducible benchmarking
• Authors: Grigori Fursin, Herve Guillou and Nicolas Essayan
• Abstract summary: We present CodeReef - an open platform to share all the components necessary to enable cross-platform MLOps (MLSysOps) We also introduce the CodeReef solution - a way to package and share models as non-virtualized, portable, customizable archive files.
• Score: 0.2148535041822524
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present CodeReef - an open platform to share all the components necessary to enable cross-platform MLOps (MLSysOps), i.e. automating the deployment of ML models across diverse systems in the most efficient way. We also introduce the CodeReef solution - a way to package and share models as non-virtualized, portable, customizable and reproducible archive files. Such ML packages include JSON meta description of models with all dependencies, Python APIs, CLI actions and portable workflows necessary to automatically build, benchmark, test and customize models across diverse platforms, AI frameworks, libraries, compilers and datasets. We demonstrate several CodeReef solutions to automatically build, run and measure object detection based on SSD-Mobilenets, TensorFlow and COCO dataset from the latest MLPerf inference benchmark across a wide range of platforms from Raspberry Pi, Android phones and IoT devices to data centers. Our long-term goal is to help researchers share their new techniques as production-ready packages along with research papers to participate in collaborative and reproducible benchmarking, compare the different ML/software/hardware stacks and select the most efficient ones on a Pareto frontier using online CodeReef dashboards.

Related papers

• MALPOLON: A Framework for Deep Species Distribution Modeling [3.1457219084519004]
  MALPOLON aims to facilitate training and inferences of deep species distribution models (deep-SDM) It is written in Python and built upon the PyTorch library. The framework is open-sourced on GitHub and PyPi.
  arXiv  Detail & Related papers  (2024-09-26T17:45:10Z)
• CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agents [49.68117560675367]
  Crab is the first benchmark framework designed to support cross-environment tasks. Our framework supports multiple devices and can be easily extended to any environment with a Python interface. The experimental results demonstrate that the single agent with GPT-4o achieves the best completion ratio of 38.01%.
  arXiv  Detail & Related papers  (2024-07-01T17:55:04Z)
• UltraEval: A Lightweight Platform for Flexible and Comprehensive Evaluation for LLMs [74.1976921342982]
  This paper introduces UltraEval, a user-friendly evaluation framework characterized by its lightweight nature, comprehensiveness, modularity, and efficiency. The resulting composability allows for the free combination of different models, tasks, prompts, benchmarks, and metrics within a unified evaluation workflow.
  arXiv  Detail & Related papers  (2024-04-11T09:17:12Z)
• CLAID: Closing the Loop on AI & Data Collection -- A Cross-Platform Transparent Computing Middleware Framework for Smart Edge-Cloud and Digital Biomarker Applications [2.953239144917]
  We present CLAID, an open-source framework based on transparent computing compatible with Android, iOS, WearOS, Linux, and Windows. We provide modules for data collection from various sensors as well as for the deployment of machine-learning models. We propose a novel methodology, "ML-Model in the Loop," for verifying deployed machine learning models.
  arXiv  Detail & Related papers  (2023-10-09T11:56:51Z)
• Model Share AI: An Integrated Toolkit for Collaborative Machine Learning Model Development, Provenance Tracking, and Deployment in Python [0.0]
  We introduce Model Share AI (AIMS), an easy-to-use MLOps platform designed to streamline collaborative model development, model provenance tracking, and model deployment. AIMS features collaborative project spaces and a standardized model evaluation process that ranks model submissions based on their performance on unseen evaluation data. AIMS allows users to deploy ML models built in Scikit-Learn, Keras, PyTorch, and ONNX into live REST APIs and automatically generated web apps.
  arXiv  Detail & Related papers  (2023-09-27T15:24:39Z)
• Learning Cross-Modal Affinity for Referring Video Object Segmentation Targeting Limited Samples [61.66967790884943]
  Referring video object segmentation (RVOS) relies on sufficient data for a given scene. In more realistic scenarios, only minimal annotations are available for a new scene. We propose a model with a newly designed cross-modal affinity (CMA) module based on a Transformer architecture. CMA module builds multimodal affinity with a few samples, thus quickly learning new semantic information, and enabling the model to adapt to different scenarios.
  arXiv  Detail & Related papers  (2023-09-05T08:34:23Z)
• In Situ Framework for Coupling Simulation and Machine Learning with Application to CFD [51.04126395480625]
  Recent years have seen many successful applications of machine learning (ML) to facilitate fluid dynamic computations. As simulations grow, generating new training datasets for traditional offline learning creates I/O and storage bottlenecks. This work offers a solution by simplifying this coupling and enabling in situ training and inference on heterogeneous clusters.
  arXiv  Detail & Related papers  (2023-06-22T14:07:54Z)
• SOLIS -- The MLOps journey from data acquisition to actionable insights [62.997667081978825]
  In this paper we present a unified deployment pipeline and freedom-to-operate approach that supports all requirements while using basic cross-platform tensor framework and script language engines. This approach however does not supply the needed procedures and pipelines for the actual deployment of machine learning capabilities in real production grade systems.
  arXiv  Detail & Related papers  (2021-12-22T14:45:37Z)
• VEGA: Towards an End-to-End Configurable AutoML Pipeline [101.07003005736719]
  VEGA is an efficient and comprehensive AutoML framework that is compatible and optimized for multiple hardware platforms. VEGA can improve the existing AutoML algorithms and discover new high-performance models against SOTA methods.
  arXiv  Detail & Related papers  (2020-11-03T06:53:53Z)
• The Collective Knowledge project: making ML models more portable and reproducible with open APIs, reusable best practices and MLOps [0.2538209532048866]
  This article provides an overview of the Collective Knowledge technology (CK or cKnowledge CK) CK attempts to make it easier to reproduce ML&systems research, deploy ML models in production, and adapt them to changing data sets, models, research techniques, software, and hardware.
  arXiv  Detail & Related papers  (2020-06-12T13:18:52Z)
• MLModelCI: An Automatic Cloud Platform for Efficient MLaaS [15.029094196394862]
  We release the platform as an open-source project on GitHub under Apache 2.0 license. Our system bridges the gap between current ML training and serving systems and thus free developers from manual and tedious work often associated with service deployment.
  arXiv  Detail & Related papers  (2020-06-09T07:48:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.