Challenges in Migrating Imperative Deep Learning Programs to Graph Execution: An Empirical Study

• URL: http://arxiv.org/abs/2201.09953v1
• Date: Mon, 24 Jan 2022 21:12:38 GMT
• Title: Challenges in Migrating Imperative Deep Learning Programs to Graph Execution: An Empirical Study
• Authors: Tatiana Castro V\'elez, Raffi Khatchadourian, Mehdi Bagherzadeh, Anita Raja
• Abstract summary: We conduct a data-driven analysis of challenges -- and resultant bugs -- involved in writing reliable yet performant imperative DL code. We put forth several recommendations, best practices, and anti-patterns for effectively hybridizing imperative DL code.
• Score: 4.415977307120617
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Efficiency is essential to support responsiveness w.r.t. ever-growing datasets, especially for Deep Learning (DL) systems. DL frameworks have traditionally embraced deferred execution-style DL code that supports symbolic, graph-based Deep Neural Network (DNN) computation. While scalable, such development tends to produce DL code that is error-prone, non-intuitive, and difficult to debug. Consequently, more natural, less error-prone imperative DL frameworks encouraging eager execution have emerged but at the expense of run-time performance. While hybrid approaches aim for the "best of both worlds," the challenges in applying them in the real world are largely unknown. We conduct a data-driven analysis of challenges -- and resultant bugs -- involved in writing reliable yet performant imperative DL code by studying 250 open-source projects, consisting of 19.7 MLOC, along with 470 and 446 manually examined code patches and bug reports, respectively. The results indicate that hybridization: (i) is prone to API misuse, (ii) can result in performance degradation -- the opposite of its intention, and (iii) has limited application due to execution mode incompatibility. We put forth several recommendations, best practices, and anti-patterns for effectively hybridizing imperative DL code, potentially benefiting DL practitioners, API designers, tool developers, and educators.

Related papers

• Safe Automated Refactoring for Efficient Migration of Imperative Deep Learning Programs to Graph Execution [4.461099699060121]
  We present an automated approach to determine when it is safe and potentially advantageous to migrate imperative DL code to graph execution. The approach is implemented as a PyDev Eclipse plug-in that integrates the WALA Ariadne analysis framework.
  arXiv  Detail & Related papers  (2025-04-07T18:48:43Z)
• Deep-Bench: Deep Learning Benchmark Dataset for Code Generation [2.897621520197328]
  DeepBench is a novel benchmark dataset for function-level Deep learning code generation. GPT-4o -- the state-of-the-art LLM -- achieved 31% accuracy on DeepBench, significantly lower than its 60% on DS-1000. DeepBench offers valuable insights into the LLMs' performance and areas for potential improvement in the DL domain.
  arXiv  Detail & Related papers  (2025-02-26T00:43:50Z)
• SIaM: Self-Improving Code-Assisted Mathematical Reasoning of Large Language Models [54.78329741186446]
  We propose a novel paradigm that uses a code-based critic model to guide steps including question-code data construction, quality control, and complementary evaluation. Experiments across both in-domain and out-of-domain benchmarks in English and Chinese demonstrate the effectiveness of the proposed paradigm.
  arXiv  Detail & Related papers  (2024-08-28T06:33:03Z)
• What's Wrong with Your Code Generated by Large Language Models? An Extensive Study [80.18342600996601]
  Large language models (LLMs) produce code that is shorter yet more complicated as compared to canonical solutions. We develop a taxonomy of bugs for incorrect codes that includes three categories and 12 sub-categories, and analyze the root cause for common bug types. We propose a novel training-free iterative method that introduces self-critique, enabling LLMs to critique and correct their generated code based on bug types and compiler feedback.
  arXiv  Detail & Related papers  (2024-07-08T17:27:17Z)
• Science-Informed Deep Learning (ScIDL) With Applications to Wireless Communications [11.472232944923558]
  This article provides a tutorial on science-informed deep learning (ScIDL) ScIDL aims to integrate existing scientific knowledge with DL techniques to develop more powerful algorithms. We discuss both recent applications of ScIDL and potential future research directions in the field of wireless communications.
  arXiv  Detail & Related papers  (2024-06-29T02:35:39Z)
• On the Worst Prompt Performance of Large Language Models [93.13542053835542]
  Performance of large language models (LLMs) is acutely sensitive to the phrasing of prompts. We introduce RobustAlpacaEval, a new benchmark that consists of semantically equivalent case-level queries. Experiments on RobustAlpacaEval with ChatGPT and six open-source LLMs from the Llama, Mistral, and Gemma families uncover substantial variability in model performance.
  arXiv  Detail & Related papers  (2024-06-08T13:40:38Z)
• DLAP: A Deep Learning Augmented Large Language Model Prompting Framework for Software Vulnerability Detection [12.686480870065827]
  This paper contributes textbfDLAP, a framework that combines the best of both deep learning (DL) models and Large Language Models (LLMs) to achieve exceptional vulnerability detection performance. Experiment results confirm that DLAP outperforms state-of-the-art prompting frameworks, including role-based prompts, auxiliary information prompts, chain-of-thought prompts, and in-context learning prompts.
  arXiv  Detail & Related papers  (2024-05-02T11:44:52Z)
• Towards Safe Automated Refactoring of Imperative Deep Learning Programs to Graph Execution [4.786072763033669]
  More natural, less error-prone imperative DL frameworks encouraging eager execution have emerged at the expense of run-time performance. We present our ongoing work on an automated approach that assists developers in specifying whether and how their otherwise imperative DL code could be reliably and efficiently executed as graphs. The approach is being implemented as a PyDev Eclipse plug-in and uses the WALA Ariadne analysis framework.
  arXiv  Detail & Related papers  (2023-08-22T20:50:19Z)
• NAPG: Non-Autoregressive Program Generation for Hybrid Tabular-Textual Question Answering [52.10214317661547]
  Current numerical reasoning methods autoregressively decode program sequences. The accuracy of program generation drops sharply as the decoding steps unfold due to error propagation. In this paper, we propose a non-autoregressive program generation framework.
  arXiv  Detail & Related papers  (2022-11-07T11:25:21Z)
• DeepFD: Automated Fault Diagnosis and Localization for Deep Learning Programs [15.081278640511998]
  DeepFD is a learning-based fault diagnosis and localization framework. It maps the fault localization task to a learning problem. It correctly diagnoses 52% faulty DL programs, compared with around half (27%) achieved by the best state-of-the-art works.
  arXiv  Detail & Related papers  (2022-05-04T08:15:56Z)
• Automatic Fault Detection for Deep Learning Programs Using Graph Transformations [13.572917264310119]
  We propose NeuraLint, a model-based fault detection approach for Deep Learning programs. NeuraLint effectively detects faults and design issues in both synthesized and real-world examples with a recall of 70.5 % and a precision of 100 %. Although the proposed meta-model is designed for feedforward neural networks, it can be extended to support other neural network architectures.
  arXiv  Detail & Related papers  (2021-05-17T18:06:11Z)
• CogDL: A Comprehensive Library for Graph Deep Learning [55.694091294633054]
  We present CogDL, a library for graph deep learning that allows researchers and practitioners to conduct experiments, compare methods, and build applications with ease and efficiency. In CogDL, we propose a unified design for the training and evaluation of GNN models for various graph tasks, making it unique among existing graph learning libraries. We develop efficient sparse operators for CogDL, enabling it to become the most competitive graph library for efficiency.
  arXiv  Detail & Related papers  (2021-03-01T12:35:16Z)
• PolyDL: Polyhedral Optimizations for Creation of High Performance DL primitives [55.79741270235602]
  We present compiler algorithms to automatically generate high performance implementations of Deep Learning primitives. We develop novel data reuse analysis algorithms using the polyhedral model. We also show that such a hybrid compiler plus a minimal library-use approach results in state-of-the-art performance.
  arXiv  Detail & Related papers  (2020-06-02T06:44:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.