Context-aware Execution Migration Tool for Data Science Jupyter Notebooks on Hybrid Clouds

• URL: http://arxiv.org/abs/2107.00187v1
• Date: Thu, 1 Jul 2021 02:33:18 GMT
• Title: Context-aware Execution Migration Tool for Data Science Jupyter Notebooks on Hybrid Clouds
• Authors: Renato L. F. Cunha, Lucas V. Real, Renan Souza, Bruno Silva, Marco A. S. Netto
• Abstract summary: This paper presents a solution developed as a Jupyter extension that automatically selects which cells, as well as in which scenarios, such cells should be migrated to a more suitable platform for execution. Using notebooks from Earth science (remote sensing), image recognition, and hand written digit identification (machine learning), our experiments show notebook state reductions of up to 55x and migration decisions leading to performance gains of up to 3.25x when the user interactivity with the notebook is taken into consideration.
• Score: 0.22908242575265025
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Interactive computing notebooks, such as Jupyter notebooks, have become a popular tool for developing and improving data-driven models. Such notebooks tend to be executed either in the user's own machine or in a cloud environment, having drawbacks and benefits in both approaches. This paper presents a solution developed as a Jupyter extension that automatically selects which cells, as well as in which scenarios, such cells should be migrated to a more suitable platform for execution. We describe how we reduce the execution state of the notebook to decrease migration time and we explore the knowledge of user interactivity patterns with the notebook to determine which blocks of cells should be migrated. Using notebooks from Earth science (remote sensing), image recognition, and hand written digit identification (machine learning), our experiments show notebook state reductions of up to 55x and migration decisions leading to performance gains of up to 3.25x when the user interactivity with the notebook is taken into consideration.

Related papers

• X-MIC: Cross-Modal Instance Conditioning for Egocentric Action Generalization [56.75782714530429]
  We propose a cross-modal adaptation framework, which we call X-MIC. Our pipeline learns to align frozen text embeddings to each egocentric video directly in the shared embedding space. This results in an enhanced alignment of text embeddings to each egocentric video, leading to a significant improvement in cross-dataset generalization.
  arXiv  Detail & Related papers  (2024-03-28T19:45:35Z)
• A Flexible Cell Classification for ML Projects in Jupyter Notebooks [0.21485350418225244]
  This paper presents a more flexible approach to cell classification based on a hybrid classification approach that combines a rule-based and a decision tree classifier. We implemented the new flexible cell classification approach in a tool called JupyLabel.
  arXiv  Detail & Related papers  (2024-03-12T11:50:47Z)
• Online Adaptation of Language Models with a Memory of Amortized Contexts [82.02369596879817]
  Memory of Amortized Contexts (MAC) is an efficient and effective online adaptation framework for large language models. We show how MAC can be combined with and improve the performance of popular alternatives such as retrieval augmented generations.
  arXiv  Detail & Related papers  (2024-03-07T08:34:57Z)
• MemGPT: Towards LLMs as Operating Systems [50.02623936965231]
  Large language models (LLMs) have revolutionized AI, but are constrained by limited context windows. We propose virtual context management, a technique drawing inspiration from hierarchical memory systems in traditional operating systems. We release MemGPT code and data for our experiments at https://memgpt.ai.
  arXiv  Detail & Related papers  (2023-10-12T17:51:32Z)
• MOON: Assisting Students in Completing Educational Notebook Scenarios [0.0]
  Notebooks come with many attractive features, such as the ability to combine textual explanations, multimedia content, and executable code. This execution model can quickly become an issue when students do not follow the intended execution order of the teacher. We present a novel approach, MOON, designed to remedy this problem.
  arXiv  Detail & Related papers  (2023-09-28T06:49:30Z)
• Notably Inaccessible -- Data Driven Understanding of Data Science Notebook (In)Accessibility [13.428631054625797]
  We perform a large scale systematic analysis of 100000 Jupyter notebooks to identify various accessibility challenges. We make recommendations to improve accessibility of the artifacts of a notebook, suggest authoring practices, and propose changes to infrastructure to make notebooks accessible.
  arXiv  Detail & Related papers  (2023-08-07T01:33:32Z)
• SuperNOVA: Design Strategies and Opportunities for Interactive Visualization in Computational Notebooks [34.04783941358773]
  We analyze 163 interactive visualization tools for notebooks. We identify key design implications and trade-offs. We develop SuperNOVA, an open-source interactive browser to help researchers explore existing notebook visualization tools.
  arXiv  Detail & Related papers  (2023-05-04T17:57:54Z)
• MetaNetwork: A Task-agnostic Network Parameters Generation Framework for Improving Device Model Generalization [65.02542875281233]
  We propose a novel task-agnostic framework, named MetaNetwork, for generating adaptive device model parameters from cloud without on-device training. The MetaGenerator is designed to learn a mapping function from samples to model parameters, and it can generate and deliver the adaptive parameters to the device based on samples uploaded from the device to the cloud. The MetaStabilizer aims to reduce the oscillation of the MetaGenerator, accelerate the convergence and improve the model performance during both training and inference.
  arXiv  Detail & Related papers  (2022-09-12T13:26:26Z)
• X2T: Training an X-to-Text Typing Interface with Online Learning from User Feedback [83.95599156217945]
  We focus on assistive typing applications in which a user cannot operate a keyboard, but can supply other inputs. Standard methods train a model on a fixed dataset of user inputs, then deploy a static interface that does not learn from its mistakes. We investigate a simple idea that would enable such interfaces to improve over time, with minimal additional effort from the user.
  arXiv  Detail & Related papers  (2022-03-04T00:07:20Z)
• StickyLand: Breaking the Linear Presentation of Computational Notebooks [5.1175396458764855]
  StickyLand is a notebook extension for empowering users to freely organize their code in non-linear ways. With sticky cells that are always shown on the screen, users can quickly access their notes, instantly observe experiment results, and easily build interactive dashboards.
  arXiv  Detail & Related papers  (2022-02-22T18:25:54Z)
• Realistic simulation of users for IT systems in cyber ranges [63.20765930558542]
  We instrument each machine by means of an external agent to generate user activity. This agent combines both deterministic and deep learning based methods to adapt to different environment. We also propose conditional text generation models to facilitate the creation of conversations and documents.
  arXiv  Detail & Related papers  (2021-11-23T10:53:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.